Rebuild against current mozilla-central SpiderMonkey (157a1), enable SharedArrayBuffer/Atomics - #509
dan-distributive wants to merge 2 commits into
Conversation
…SharedArrayBuffer/Atomics
pythonmonkey previously embedded mozjs-136a1.dll, a ~19-month-old Firefox
Nightly Alpha build that predates Firefox 136's stable release and its
out-of-band security patch (136.0.4) for an actively-exploited sandbox-escape
CVE. This rebuilds against a current mozilla-central snapshot (commit
1704651e7d6c706fcb753adab577e0954d61cee0) instead.
Ten distinct SpiderMonkey embedder-API breaks between 136a1 and current
trunk are fixed, ranging from mechanical (asm.js removed, mozilla::Unused
removed) to a real architecture change: JS::JobQueue's per-job
enqueuePromiseJob push callback was removed entirely in favor of an
engine-internal queue the embedder must explicitly drain via js::RunJobs(),
which required adding checkpoints at three call sites (top-level script
execution, and the two places PromiseType.cc attaches/resolves promise
reactions across the Python/JS boundary) to avoid a real hang this surfaced
during testing.
Also enables SharedArrayBuffer/Atomics (JS::RealmCreationOptions::
setSharedMemoryAndAtomicsEnabled), off by default in this embedding but
required for Pyodide's threaded WASM build to link at all
("LinkError: shared memory is disabled" otherwise) - unrelated to the
version bump itself but bundled here since both were verified together.
See SPIDERMONKEY_VERSION_BUMP.md for the full change-by-change rationale,
explicit risk/review flags on the two changes needing real SpiderMonkey-
internals judgment (the JobQueue redesign and a GC-safety adaptation in
BufferType.cc), and what was and wasn't verified. Verified against basic
eval, SharedArrayBuffer/Atomics, async/await across the Python/JS boundary,
and both real localExec() test suites (dcp_local_job_test.py,
pycomod_localexec_test.py) end-to-end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
localExec() never leaves the process; exec() submits to the real DCP scheduler and depends on a funded wallet and live workers on the target compute group. Verified separately and documented since it's a materially different code path from everything else already tested. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Note on authorship: this PR was largely produced by Claude (Anthropic's Claude Code), working under my direction and review, not hand-written by me. The SpiderMonkey engine rebuild, the ten API-break fixes, and the testing described in Flagging this explicitly because none of this has had a second human reviewer yet, and reviewers should know to apply the same scrutiny here they would to any AI-assisted change — especially the two sections the doc itself marks NEEDS REVIEW (the |
Summary
mozjs-157a1, commit1704651e7d6c706fcb753adab577e0954d61cee0), replacing the ~19-month-oldmozjs-136a1Nightly Alpha build it previously shipped — which predates Firefox 136's stable release and its out-of-band patch for an actively-exploited sandbox-escape CVE.136a1and current trunk, most mechanical (asm.jsremoved,mozilla::Unusedremoved, off-threadDispatchabledispatch API redesigned) but one a real architecture change:JS::JobQueue's per-jobenqueuePromiseJobpush callback was removed entirely in favor of an engine-internal queue the embedder must explicitly drain viajs::RunJobs().SharedArrayBuffer/Atomics(JS::RealmCreationOptions::setSharedMemoryAndAtomicsEnabled), off by default in this embedding but required for Pyodide's threaded WASM build to link at all.Full change-by-change rationale, diffs, and explicit risk/review flags are in
SPIDERMONKEY_VERSION_BUMP.md(added by this PR) — please read it before merging. It calls out two changes needing real SpiderMonkey-internals judgment rather than a mechanical port (a GC-safety adaptation inBufferType.cc, and theJobQueueredesign), and is explicit about what was and wasn't independently verified.Test plan
All of the following were run against the rebuilt engine and passed:
pm.eval()(arithmetic, JSON)SharedArrayBuffer/Atomicsround-trip (the original motivating fix)await, an ordered two-hopawaitchain inside a JS async function, and asetTimeout-based promisedcp_local_job_test.py— fulllocalExec()pipeline, correct outputpycomod_localexec_test.py— heavier stress test: filesystem shipping, extra Pyodide modules (pandas), cloudpickle round-trip of nested numpy arrays, correct numeric outputjob.exec()(notlocalExec()) against the live DCP network on thedemo/dcpcompute group — full real scheduler lifecycle, real worker-produced results, correct outputNot yet independently verified (see
SPIDERMONKEY_VERSION_BUMP.md's closing section for the full list):Atomics.waitAsyncwith a real timeout, Debugger-API-facing JobQueue paths, and theBufferType.ccGC-safety reasoning under a compacting/moving-GC stress configuration. None of this has had a second reviewer yet — recommend focused review on the sections flagged NEEDS REVIEW in the doc before merging.🤖 Generated with Claude Code