Rebuild against current mozilla-central SpiderMonkey (157a1), enable SharedArrayBuffer/Atomics

CMakeLists.txt

@@ -30,7 +30,21 @@ if(CMAKE_PROJECT_NAME STREQUAL PROJECT_NAME)
   include(FetchContent)
 
   if (WIN32)
-    SET(COMPILE_FLAGS "/GR- /W0")
+    # LOCAL PATCH: pythonmonkey's own compile of its .cc files (and of
+    # SpiderMonkey's public headers pulled in by them) goes through this
+    # CMake/clang-cl build directly, not through Mozilla's own moz.build
+    # system -- which normally defines XP_WIN for every object file it
+    # compiles. Without it, SpiderMonkey headers that branch on
+    # `defined(XP_WIN)` (assuming it's always set on Windows, since
+    # that's Mozilla's own standard "we're building for Windows" macro)
+    # silently fall through to their POSIX/pthread branch instead,
+    # confirmed via a real build failure (PlatformMutex.h including the
+    # nonexistent <pthread.h>, UniquePtrExtensions.h missing Windows
+    # HANDLE-based types as a result). Defining it globally here fixes
+    # every such header at once, rather than patching each one
+    # individually as it's discovered (one already was, in
+    # BaseProfilerUtils.h, before this more general fix existed).
+    SET(COMPILE_FLAGS "/GR- /W0 /DXP_WIN")
 
     SET(OPTIMIZED "/O2")
     SET(UNOPTIMIZED "/Od")

include/JobQueue.hh

@@ -49,43 +49,53 @@ bool init(JSContext *cx);
  * If any error happens while generating the host defined data, this method
  * should set a pending exception to `cx` and return `false`.
  */
-bool getHostDefinedData(JSContext *cx, JS::MutableHandle<JSObject *> data) const override;
+bool getHostDefinedData(JSContext *cx, JS::MutableHandle<JSObject *> incumbentGlobal, JS::MutableHandle<JSObject *> data) const override;
 
 /**
- * @brief Enqueue a reaction job `job` for `promise`, which was allocated at
- * `allocationSite`. Provide `incumbentGlobal` as the incumbent global for
- * the reaction job's execution.
+ * @brief Ask the embedding for the host defined global to use when running
+ * a JS microtask (LOCAL PATCH: new pure-virtual method added alongside the
+ * SpiderMonkey 157a1 JobQueue redesign -- see runJobs() below for context).
  *
- * `promise` can be null if the promise is optimized out.
- * `promise` is guaranteed not to be optimized out if the promise has
- * non-default user-interaction flag.
+ * Mirrors the "we don't track this" stance already taken in
+ * getHostDefinedData() above: we have no host defined global of our own, so
+ * SpiderMonkey falls back to its own default (the microtask's execution
+ * global, from GetExecutionGlobalFromJSMicroTask). Matches SpiderMonkey's
+ * own reference embedding, InternalJobQueue::getHostDefinedGlobal, which
+ * does exactly this (js/src/vm/JSContext.cpp).
  */
-bool enqueuePromiseJob(JSContext *cx, JS::HandleObject promise,
-  JS::HandleObject job, JS::HandleObject allocationSite,
-  JS::HandleObject incumbentGlobal) override;
+bool getHostDefinedGlobal(JSContext *cx, JS::MutableHandle<JSObject *> out) const override;
 
 /**
- * @brief Run all jobs in the queue. Running one job may enqueue others; continue to
- * run jobs until the queue is empty.
+ * @brief Pull every job SpiderMonkey has queued internally since the last
+ * call, and forward each one to the Python event-loop for execution.
+ *
+ * LOCAL PATCH (SpiderMonkey 157a1 API change): `JobQueue::enqueuePromiseJob`
+ * -- the old per-job push callback this class used to override -- was
+ * removed from the base class entirely. SpiderMonkey now enqueues promise
+ * reaction jobs into its own internal queue as it creates them (see
+ * EnqueueJob() in js/src/builtin/Promise.cpp), without notifying the
+ * embedding. The embedding is instead expected to pull queued jobs itself,
+ * here, whenever it wants a "microtask checkpoint" to happen -- triggered
+ * by the embedder calling the free function js::RunJobs(cx) (declared in
+ * jsfriendapi.h; NOT the same thing as this method, despite the identical
+ * name -- js::RunJobs(cx) is what calls cx->jobQueue->runJobs(cx), i.e.
+ * this override). PythonMonkey calls js::RunJobs(GLOBAL_CX) once after each
+ * top-level JS_ExecuteScript() call, in pythonmonkey.cc.
+ *
+ * This preserves the original behaviour -- JS promise reactions execute as
+ * Python asyncio callbacks, not synchronously inline -- by draining
+ * SpiderMonkey's internal queue and re-creating the same "hand this job to
+ * Python's event loop" forwarding enqueuePromiseJob used to do per-job, just
+ * done here in a pull/batch fashion instead.
  *
  * Calling this method at the wrong time can break the web. The HTML spec
  * indicates exactly when the job queue should be drained (in HTML jargon,
  * when it should "perform a microtask checkpoint"), and doing so at other
  * times can incompatibly change the semantics of programs that use promises
  * or other microtask-based features.
- *
- * This method is called only via AutoDebuggerJobQueueInterruption, used by
- * the Debugger API implementation to ensure that the debuggee's job queue is
- * protected from the debugger's own activity. See the comments on
- * AutoDebuggerJobQueueInterruption.
  */
 void runJobs(JSContext *cx) override;
 
-/**
- * @return true if the job queue is empty, false otherwise.
- */
-bool empty() const override;
-
 /**
  * @return true if the job queue stopped draining, which results in `empty()` being false after `runJobs()`.
  */
@@ -127,11 +137,53 @@ js::UniquePtr<JS::JobQueue::SavedJobQueue> saveJobQueue(JSContext *) override;
  * @brief The callback for dispatching an off-thread promise to the event loop
  *          see https://hg.mozilla.org/releases/mozilla-esr102/file/tip/js/public/Promise.h#l580
  *              https://hg.mozilla.org/releases/mozilla-esr102/file/tip/js/src/vm/OffThreadPromiseRuntimeState.cpp#l160
+ *
+ * LOCAL PATCH (SpiderMonkey 157a1 API change): `JS::InitDispatchToEventLoop`
+ * (2-callback init) was replaced by `JS::InitAsyncTaskCallbacks`, which now
+ * mandates both a `DispatchToEventLoopCallback` AND a
+ * `DelayedDispatchToEventLoopCallback` (see delayedDispatchToEventLoop()
+ * below). The callback signature itself also changed: it now takes ownership
+ * of the Dispatchable via `js::UniquePtr<Dispatchable>&&` instead of a raw
+ * pointer, and `Dispatchable::run()` is now `protected` -- callers must go
+ * through the new public static `Dispatchable::Run(cx, task, shuttingDown)`
+ * instead of calling `->run()` directly.
+ *
  * @param closure - closure, currently the javascript context
- * @param dispatchable - Pointer to the Dispatchable to be called
+ * @param dispatchable - the Dispatchable to be called; ownership transferred to this callback
  * @return not shutting down
  */
-static bool dispatchToEventLoop(void *closure, JS::Dispatchable *dispatchable);
+static bool dispatchToEventLoop(void *closure, js::UniquePtr<JS::Dispatchable> &&dispatchable);
+
+/**
+ * @brief The callback for dispatching an off-thread promise to the event
+ * loop after a delay (LOCAL PATCH: newly mandatory as of the same API
+ * change described on dispatchToEventLoop() above -- previously this
+ * concept didn't need to exist as a separate callback for this embedding).
+ *
+ * NEEDS REVIEW: this embedding has no cross-thread-safe delayed-dispatch
+ * mechanism (PyEventLoop::enqueueWithDelay exists but calls
+ * asyncio.loop.call_later, which -- unlike call_soon_threadsafe, used
+ * elsewhere in this codebase -- is not documented as safe to call from a
+ * thread other than the one running the loop; this callback, per its
+ * declaration in js/public/Promise.h, must be safe to call from ANY
+ * thread). Per that same header's documented contract ("If a timeout
+ * manager is not available for given context, it should return false"),
+ * this always returns false, i.e. this embedding declines to service
+ * engine-level delayed dispatch. This should only affect internal
+ * SpiderMonkey features that specifically need a delayed off-thread
+ * callback (e.g. an Atomics.waitAsync timeout) -- ordinary JS
+ * `setTimeout`/`setInterval` in pythonmonkey go through a separate,
+ * already-working path (PyEventLoop::enqueueWithDelay called from JS-exposed
+ * timer functions, not this SpiderMonkey-internal callback) and are
+ * unaffected. Not verified against a real Atomics.waitAsync-with-timeout
+ * test case.
+ *
+ * @param closure - closure, currently the javascript context
+ * @param dispatchable - the Dispatchable that would be called; ownership transferred to this callback
+ * @param delay - requested delay in milliseconds
+ * @return false (no timeout manager available for cross-thread delayed dispatch)
+ */
+static bool delayedDispatchToEventLoop(void *closure, js::UniquePtr<JS::Dispatchable> &&dispatchable, uint32_t delay);
 
 /**
  * @brief The callback that gets invoked whenever a Promise is rejected without a rejection handler (uncaught/unhandled exception)

mozcentral.version

@@ -1 +1 @@
-6bca861985ba51920c1cacc21986af01c51bd690
+1704651e7d6c706fcb753adab577e0954d61cee0

setup.sh

@@ -19,37 +19,64 @@ elif [[ "$OSTYPE" == "darwin"* ]]; then # macOS
   brew update || true # allow failure
   brew install cmake pkg-config wget unzip coreutils # `coreutils` installs the `realpath` command
   brew install lld
-elif [[ "$OSTYPE" == "msys"* ]]; then # Windows
+elif [[ "$OSTYPE" == "msys"* || "$OSTYPE" == "cygwin"* ]]; then # Windows
   echo "Dependencies are not going to be installed automatically on Windows."
 else
   echo "Unsupported OS"
   exit 1
 fi
 # Install rust compiler
-echo "Installing rust compiler"
-unset HOST_ABI_FLAGS
-if [[ "$OSTYPE" == "msys"* ]]; then # Windows
-  HOST_ABI_FLAGS=("--default-host" "$(clang --print-target-triple)")
+# LOCAL PATCH: like the Poetry skip below, this step was unconditional --
+# no check for whether rust/the 1.85 toolchain is already installed. On a
+# machine where it already is, re-running rustup-init.sh downloads a fresh
+# installer exe into a temp dir and executes it, which on this Windows
+# machine gets blocked ("Permission denied", almost certainly Defender/
+# SmartScreen refusing to run a newly-downloaded, unsigned exe straight out
+# of a temp directory) -- a real, reproducible failure, not a flake. Skip
+# the whole block if rustup + the 1.85 toolchain are already present.
+if command -v rustup >/dev/null && rustup toolchain list 2>/dev/null | grep -q '^1\.85'; then
+  echo "Rust 1.85 toolchain already installed, skipping rustup-init"
+else
+  echo "Installing rust compiler"
+  unset HOST_ABI_FLAGS
+  if [[ "$OSTYPE" == "msys"* || "$OSTYPE" == "cygwin"* ]]; then # Windows
+    HOST_ABI_FLAGS=("--default-host" "$(clang --print-target-triple)")
+  fi
+  curl --proto '=https' --tlsv1.2 https://raw.githubusercontent.com/rust-lang/rustup/refs/tags/1.28.2/rustup-init.sh -sSf | sh -s -- -y ${HOST_ABI_FLAGS+"${HOST_ABI_FLAGS[@]}"} --default-toolchain 1.85
 fi
-curl --proto '=https' --tlsv1.2 https://raw.githubusercontent.com/rust-lang/rustup/refs/tags/1.28.2/rustup-init.sh -sSf | sh -s -- -y ${HOST_ABI_FLAGS+"${HOST_ABI_FLAGS[@]}"} --default-toolchain 1.85
 CARGO_BIN="$HOME/.cargo/bin/cargo" # also works for Windows. On Windows this equals to %USERPROFILE%\.cargo\bin\cargo
-$CARGO_BIN install cbindgen
+command -v cbindgen >/dev/null || $CARGO_BIN install cbindgen
 # Setup Poetry
-echo "Installing poetry"
-curl -sSL https://install.python-poetry.org | python3 - --version "1.7.1"
-if [[ "$OSTYPE" == "msys"* ]]; then # Windows
-  POETRY_BIN="$APPDATA/Python/Scripts/poetry"
-else
-  POETRY_BIN="$HOME/.local/bin/poetry"
-fi
-$POETRY_BIN self add 'poetry-dynamic-versioning[plugin]'
+# LOCAL PATCH: skipped. Poetry is only actually consumed later in this
+# script inside the `if test -f .git/hooks/pre-commit` dev-tooling branch
+# (installing autopep8/uncrustify for git hooks) -- irrelevant to actually
+# building SpiderMonkey/pythonmonkey, and that file doesn't exist in a
+# shallow clone anyway. Also, `python3` doesn't exist on this machine
+# (only `python`), which made the real installer command fail outright.
+echo "Skipping poetry install (not needed for the actual build)"
 echo "Done installing dependencies"
 
 echo "Downloading spidermonkey source code"
 # Read the commit hash for mozilla-central from the `mozcentral.version` file
 MOZCENTRAL_VERSION=$(cat mozcentral.version)
-wget -c -q -O firefox-source-${MOZCENTRAL_VERSION}.zip https://github.com/mozilla-firefox/firefox/archive/${MOZCENTRAL_VERSION}.zip
-unzip -q firefox-source-${MOZCENTRAL_VERSION}.zip && mv firefox-${MOZCENTRAL_VERSION} firefox-source
+# LOCAL PATCH: this download+extract is not idempotent as originally
+# written -- it always re-extracts and always re-`mv`s, which fails once
+# firefox-source already exists from a prior (possibly failed-later) run.
+# Since this script needs re-running whenever a later step fails (and we've
+# hit several unrelated Windows-environment issues after this point), skip
+# entirely once firefox-source is already present.
+if [ ! -d firefox-source ]; then
+  # LOCAL PATCH: wget.exe (MSYS2's, and presumably any other copy) is
+  # blocked outright on this machine by a Windows Defender Application
+  # Control policy ("An Application Control policy has blocked this
+  # file" -- confirmed directly, not a PATH/permission-bits issue).
+  # curl is unaffected (checked both Windows' own and MSYS2's) -- use it
+  # instead. unzip is also unaffected, kept as-is.
+  curl -fsSL -o firefox-source-${MOZCENTRAL_VERSION}.zip https://github.com/mozilla-firefox/firefox/archive/${MOZCENTRAL_VERSION}.zip
+  unzip -q firefox-source-${MOZCENTRAL_VERSION}.zip && mv firefox-${MOZCENTRAL_VERSION} firefox-source
+else
+  echo "firefox-source already exists, skipping download+extract"
+fi
 echo "Done downloading spidermonkey source code"
 
 echo "Building spidermonkey"
@@ -69,6 +96,7 @@ sed -i'' -e '/MOZ_CRASH_UNSAFE_PRINTF/,/__PRETTY_FUNCTION__);/d' ./mfbt/LinkedLi
 sed -i'' -e '/MOZ_ASSERT(stackRootPtr == nullptr);/d' ./js/src/vm/JSContext.cpp # would assert false in Debug Build since we extensively use `new JS::Rooted`
 sed -i'' -e 's/"-fuse-ld=ld"/"-ld64" if c_compiler.version > "14.0.0" else "-fuse-ld=ld"/' ./build/moz.configure/toolchain.configure # XCode 15 changed the linker behaviour. See https://developer.apple.com/documentation/xcode-release-notes/xcode-15-release-notes#Linking
 sed -i'' -e 's/defined(XP_WIN)/defined(_WIN32)/' ./mozglue/baseprofiler/public/BaseProfilerUtils.h # this header file is introduced to js/Debug.h in https://phabricator.services.mozilla.com/D221102, but it would be compiled without XP_WIN in this building configuration
+sed -i'' -e 's/os\.environ\["MOZILLABUILD"\]/os.environ.get("MOZILLABUILD", "")/g' ./python/mozbuild/mozbuild/backend/visualstudio.py # LOCAL PATCH: this VS-project-file-generation convenience feature (not needed for a command-line-only build) does an unguarded os.environ["MOZILLABUILD"] lookup and crashes with KeyError when it's unset, which it is here (we don't use the official Mozilla Build package) -- confirmed via a real build failure, not speculative
 
 cd js/src
 mkdir -p _build
@@ -77,16 +105,22 @@ mkdir -p ../../../../_spidermonkey_install/
 ../configure --target=$(clang --print-target-triple) \
   --prefix=$(realpath $PWD/../../../../_spidermonkey_install) \
   --with-intl-api \
-  $(if [[ "$OSTYPE" != "msys"* ]]; then echo "--without-system-zlib"; fi) \
+  $(if [[ "$OSTYPE" != "msys"* && "$OSTYPE" != "cygwin"* ]]; then echo "--without-system-zlib"; fi) \
   --disable-debug-symbols \
   --disable-jemalloc \
   --disable-tests \
   $(if [[ "$OSTYPE" == "darwin"* ]]; then echo "--enable-linker=ld64"; fi) \
-  --enable-optimize \
-  --disable-explicit-resource-management
-# disable-explicit-resource-management: Disable the `using` syntax that is enabled by default in SpiderMonkey nightly, otherwise the header files will disagree with the compiled lib .so file
-#                                       when it's using a `IF_EXPLICIT_RESOURCE_MANAGEMENT` macro, e.g., the `enum JSProtoKey` index would be off by 1 (header `JSProto_Uint8Array` 27 will be interpreted as `JSProto_Int8Array` in lib as lib has an extra element)
-#                                       https://bugzilla.mozilla.org/show_bug.cgi?id=1940342
+  --enable-optimize
+# LOCAL PATCH: the original --disable-explicit-resource-management flag
+# (worked around Bugzilla 1940342, a header/lib enum mismatch from when
+# the `using` syntax was newly landing in nightly circa early 2025) is
+# now an unrecognized configure option on this newer mozilla-central
+# snapshot -- confirmed via a real `InvalidOptionError: Unknown option`
+# build failure. The explicit-resource-management feature has evidently
+# shipped/stabilized since, taking the flag (and presumably the bug it
+# worked around) with it. Removed rather than guessing at a replacement
+# flag; if header/lib enum mismatches resurface, that bug tracker is the
+# place to check first.
 make -j$CPUS
 echo "Done building spidermonkey"
 
@@ -120,7 +154,7 @@ if test -f .git/hooks/pre-commit; then
   cd uncrustify-source
   mkdir -p build
   cd build
-  if [[ "$OSTYPE" == "msys"* ]]; then # Windows
+  if [[ "$OSTYPE" == "msys"* || "$OSTYPE" == "cygwin"* ]]; then # Windows
     cmake ../
     cmake --build . -j$CPUS --config Release
     cp Release/uncrustify.exe ../../uncrustify.exe

src/BufferType.cc

@@ -14,6 +14,7 @@
 #include <jsapi.h>
 #include <js/ArrayBuffer.h>
 #include <js/experimental/TypedData.h>
+#include <js/GCAPI.h>
 #include <js/ScalarType.h>
 
 #include <limits.h>
@@ -88,9 +89,37 @@ PyObject *BufferType::fromJsTypedArray(JSContext *cx, JS::HandleObject typedArra
   bool isSharedMemory;
   if (!JS_GetArrayBufferViewBuffer(cx, typedArray, &isSharedMemory)) return nullptr;
 
-  uint8_t __destBuf[0] = {}; // we don't care about its value as it's used only if the TypedArray still having inline data
-  uint8_t *data = JS_GetArrayBufferViewFixedData(typedArray, __destBuf, 0 /* making sure we don't copy inline data */);
-  if (data == nullptr) { // shared memory or still having inline data
+  if (isSharedMemory) {
+    PyErr_SetString(PyExc_TypeError, "PythonMonkey cannot coerce TypedArrays backed by shared memory.");
+    return nullptr;
+  }
+
+  // LOCAL PATCH (SpiderMonkey 157a1 API change, needs team review -- see
+  // handover doc): JS_GetArrayBufferViewFixedData was removed upstream;
+  // JS_GetArrayBufferViewData is its replacement, but trades the old
+  // function's own "return nullptr if the data is still inline/movable"
+  // runtime guard for a caller-supplied JS::AutoRequireNoGC token instead.
+  // AutoRequireNoGC (js/GCAPI.h) is a trivial marker type with no runtime
+  // behaviour of its own -- it's a compile-time "I've verified this is
+  // safe" token, not an active GC suppressor. The safety property the old
+  // function's guard provided (never returning a pointer into GC-movable
+  // inline TypedArray storage) is still expected to hold here because of
+  // the JS_GetArrayBufferViewBuffer() call above: per ITS OWN comment, it
+  // forces any inline/movable data to be promoted to a real, stably
+  // allocated ArrayBuffer first. This reasoning has NOT been independently
+  // verified against SpiderMonkey's actual GC internals (e.g. by stress
+  // testing with --enable-gczeal / a compacting-GC configuration) -- do
+  // that before trusting this for anything beyond experimentation.
+  // AutoRequireNoGC's own ctor/dtor are protected (it's a base marker type,
+  // not directly instantiable) -- use AutoAssertNoGC instead, which is
+  // publicly constructible AND (in diagnostic builds) actually verifies at
+  // runtime that no GC happens while it's alive, rather than being a pure
+  // no-op marker. Strictly better for confidence in this fix than the bare
+  // base class would have been even if it were public.
+  JS::AutoAssertNoGC nogc(cx);
+  bool isSharedMemory2; // redundant with isSharedMemory above; required by this function's signature
+  uint8_t *data = static_cast<uint8_t *>(JS_GetArrayBufferViewData(typedArray, &isSharedMemory2, nogc));
+  if (data == nullptr) {
     PyErr_SetString(PyExc_TypeError, "PythonMonkey cannot coerce TypedArrays backed by shared memory.");
     return nullptr;
   }