Skip to content

Rebuild against current mozilla-central SpiderMonkey (157a1), enable SharedArrayBuffer/Atomics - #509

Open
dan-distributive wants to merge 2 commits into
mainfrom
spidermonkey-157a1-rebuild
Open

dan-distributive wants to merge 2 commits into
mainfrom
spidermonkey-157a1-rebuild

Conversation

@dan-distributive

Copy link
Copy Markdown
Contributor

Summary

  • Rebuilds pythonmonkey against a current mozilla-central SpiderMonkey snapshot (mozjs-157a1, commit 1704651e7d6c706fcb753adab577e0954d61cee0), replacing the ~19-month-old mozjs-136a1 Nightly Alpha build it previously shipped — which predates Firefox 136's stable release and its out-of-band patch for an actively-exploited sandbox-escape CVE.
  • Fixes ten distinct SpiderMonkey embedder-API breaks between 136a1 and current trunk, most mechanical (asm.js removed, mozilla::Unused removed, off-thread Dispatchable dispatch API redesigned) but one a real architecture change: JS::JobQueue's per-job enqueuePromiseJob push callback was removed entirely in favor of an engine-internal queue the embedder must explicitly drain via js::RunJobs().
  • Also enables SharedArrayBuffer/Atomics (JS::RealmCreationOptions::setSharedMemoryAndAtomicsEnabled), off by default in this embedding but required for Pyodide's threaded WASM build to link at all.

Full change-by-change rationale, diffs, and explicit risk/review flags are in SPIDERMONKEY_VERSION_BUMP.md (added by this PR) — please read it before merging. It calls out two changes needing real SpiderMonkey-internals judgment rather than a mechanical port (a GC-safety adaptation in BufferType.cc, and the JobQueue redesign), and is explicit about what was and wasn't independently verified.

Test plan

All of the following were run against the rebuilt engine and passed:

  • Basic pm.eval() (arithmetic, JSON)
  • SharedArrayBuffer/Atomics round-trip (the original motivating fix)
  • Async/await across the Python/JS boundary: single await, an ordered two-hop await chain inside a JS async function, and a setTimeout-based promise
  • dcp_local_job_test.py — full localExec() pipeline, correct output
  • pycomod_localexec_test.py — heavier stress test: filesystem shipping, extra Pyodide modules (pandas), cloudpickle round-trip of nested numpy arrays, correct numeric output
  • Real job.exec() (not localExec()) against the live DCP network on the demo/dcp compute group — full real scheduler lifecycle, real worker-produced results, correct output

Not yet independently verified (see SPIDERMONKEY_VERSION_BUMP.md's closing section for the full list): Atomics.waitAsync with a real timeout, Debugger-API-facing JobQueue paths, and the BufferType.cc GC-safety reasoning under a compacting/moving-GC stress configuration. None of this has had a second reviewer yet — recommend focused review on the sections flagged NEEDS REVIEW in the doc before merging.

🤖 Generated with Claude Code

dan-distributive and others added 2 commits September 14, 2026 14:48
…SharedArrayBuffer/Atomics

pythonmonkey previously embedded mozjs-136a1.dll, a ~19-month-old Firefox
Nightly Alpha build that predates Firefox 136's stable release and its
out-of-band security patch (136.0.4) for an actively-exploited sandbox-escape
CVE. This rebuilds against a current mozilla-central snapshot (commit
1704651e7d6c706fcb753adab577e0954d61cee0) instead.

Ten distinct SpiderMonkey embedder-API breaks between 136a1 and current
trunk are fixed, ranging from mechanical (asm.js removed, mozilla::Unused
removed) to a real architecture change: JS::JobQueue's per-job
enqueuePromiseJob push callback was removed entirely in favor of an
engine-internal queue the embedder must explicitly drain via js::RunJobs(),
which required adding checkpoints at three call sites (top-level script
execution, and the two places PromiseType.cc attaches/resolves promise
reactions across the Python/JS boundary) to avoid a real hang this surfaced
during testing.

Also enables SharedArrayBuffer/Atomics (JS::RealmCreationOptions::
setSharedMemoryAndAtomicsEnabled), off by default in this embedding but
required for Pyodide's threaded WASM build to link at all
("LinkError: shared memory is disabled" otherwise) - unrelated to the
version bump itself but bundled here since both were verified together.

See SPIDERMONKEY_VERSION_BUMP.md for the full change-by-change rationale,
explicit risk/review flags on the two changes needing real SpiderMonkey-
internals judgment (the JobQueue redesign and a GC-safety adaptation in
BufferType.cc), and what was and wasn't verified. Verified against basic
eval, SharedArrayBuffer/Atomics, async/await across the Python/JS boundary,
and both real localExec() test suites (dcp_local_job_test.py,
pycomod_localexec_test.py) end-to-end.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
localExec() never leaves the process; exec() submits to the real DCP
scheduler and depends on a funded wallet and live workers on the target
compute group. Verified separately and documented since it's a materially
different code path from everything else already tested.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dan-distributive

Copy link
Copy Markdown
Contributor Author

Note on authorship: this PR was largely produced by Claude (Anthropic's Claude Code), working under my direction and review, not hand-written by me. The SpiderMonkey engine rebuild, the ten API-break fixes, and the testing described in SPIDERMONKEY_VERSION_BUMP.md were all done by Claude in an agentic session — I supervised, made the judgment calls it flagged back to me (e.g. which SpiderMonkey commit to target), and ran/confirmed the real-network exec() test myself.

Flagging this explicitly because none of this has had a second human reviewer yet, and reviewers should know to apply the same scrutiny here they would to any AI-assisted change — especially the two sections the doc itself marks NEEDS REVIEW (the BufferType.cc GC-safety adaptation and the JobQueue redesign). Happy to answer questions or dig deeper into any part of it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant