Repository navigation
Improvements to Emscripten build script #145219
Description
@freakboy3742's desired improvements to the Emscripten build script:
cleanshould be a little more selective. It currently destroys all ofcross-build; it should only be purging the parts of cross-build that are emscripten specific (so that Emscripten, iOS and Android builds can co-exist). This is admittedly a "me" problem... but it's a problem I have :-)- The
cross-builddirectory name should be configurable. Again, this is somewhat a "me" problem... but I currently need to have cross builds for 3.10-3.15, and they can't currently co-exist. - We shouldn't be building dependencies (libFFI and mpdec) on every build. A similar caching strategy to the one described for EMSDK should be possible - look to this directory; if the cache has a build for the right emscripten version, use it rather than rebuilding.
- Add install-emscripten subcommand
Linked PRs
- gh-145219: Emscripten bld configurable directory & only clean Emscripten #145581
- [3.14] gh-145219: Add Emscripten cross-build and clean configurability (GH-145581) #145654
- gh-145219: Cache Emscripten libffi and mpdec builds, add install-emscripten cmd #145664
- [3.14] gh-145219: Cache Emscripten build dependencies, add install-emscripten (GH-145664) #145790
Activity
hoodmane commentedon Feb 25, 2026
@freakboy3742 I can make these changes but I want to wait until #145177 is merged to avoid conflicts.
hoodmane commentedon Feb 25, 2026
I pushed fixes for 1 and 2 on https://github.com/hoodmane/cpython/tree/configurable-cross-build-dir on top of #145177.
pythongh-145219: Emscripten bld configurable directory & only clean E…
pythongh-145219: Emscripten bld configurable directory & only clean E…
pythongh-145219: Emscripten bld configurable directory & only clean E…
hoodmane commentedon Mar 6, 2026
@freakboy3742 for caching libraries, I'm thinking maybe we do the following:
emsdkfrom<emsdk-cache>/<version>/emsdk_env.shto<emsdk-cache>/<version>/emsdk/emsdk_env.sh--emsdk-cacheis provided, put the prefix at<emsdk-cache>/<version>/prefix.Sound reasonable to you?
freakboy3742 commentedon Mar 7, 2026
Yes, I think that makes sense.
An additional suggestion: For consistency with other builds, we should simplify the
<cross-build>/wasm32-emscripten/pythondirectory to just be the Python build (i.e., build Python in<cross-build>/wasm32-emscripten), use<emsdk-cache>/<version>/mpdecimal-x.y.zas the location for a build for mpdecimal if a cache is provided, or<cross-build>/dependencies/wasm32-emscripten/mpdecimal-x.y.zif it isn't, and shortcut dependency builds if that directory already exists.That would allow build artefacts for mpdecimal and libffi to persist between builds, increases consistency between Android, iOS and Emscripten build structures, and in particular makes the cleaning strategy consistent.
(I guess strictly we should be looking for one of the the installed products in
<prefix>rather than the build directory as the shortcut trigger; I'm not sure if it will make that much difference in practice)gh-145219: Add Emscripten cross-build and clean configurability (#145581
pythongh-145219: Add Emscripten cross-build and clean configurability (…
[3.14] gh-145219: Add Emscripten cross-build and clean configurability (
pythongh-145219: Cache Emscripten libffi and mpdec builds, add instal…
hoodmane commentedon Mar 9, 2026
I'm confused about this: it looks to me like both android and iOS have their prefix under
<cross-build>/<host>/prefixand don't make a dependencies directory at all? I do see that they build Python directly at<cross-build>/<host>but that seems weird given thatprefixis underneath it and has nothing directly to do with the Python build.Seems to me that cleaning is simplest if everything about
wasm32-emscriptenstays undercross-build/wasm32-emscripten?hoodmane commentedon Mar 9, 2026
Anyways, I'm okay with whatever file structure but I don't understand what you intend here. If you want you can directly commit whatever changes you want to the structure to #145664 or do it as a followup PR?
freakboy3742 commentedon Mar 9, 2026
The difference is that Android and iOS don't have to build their dependencies as part of the CPython build. The build of mpdecimal-1.0.0 can be shared across all builds of all versions of CPython; there's no analog of the EMSDK version dependency. As a result, they both use downloads; the
prefixis just the unpacked version of the downloads; and the downloads can be cached outside the build tree.What I'm trying to optimize here is needing to do a full rebuilt of mpdecimal, libffi et al on every rebuild of wasm32-emscripten.
As long as what we clean up doesn't mean re-building the same version of mpdecimal with the same verision of EMSDK.
I guess we could do the build for mpdecimal somewhere into the emsdk directory, install it into a standalone prefix, then zip up that artefact and cache it in the downloads folder; then, for the actual build, you look for the download, unpack it into the "live" prefix, and proceed as normal.
hoodmane commentedon Mar 10, 2026
What do you think of the approach in #145664 then?
freakboy3742 commentedon Mar 10, 2026
Just reviewed the PR - that approach looks good to me. I've flagged one more cleanup that I'd like to make, but it requires a parallel build in the buildbot, so I suspect that will be easier to make when we get to the "move to Platforms" PR.
gh-145219: Cache Emscripten build dependencies, add install-emscripten (
pythongh-145219: Cache Emscripten build dependencies, add install-ems…
[3.14] gh-145219: Cache Emscripten build dependencies, add install-em…
pythongh-145219: Add Emscripten cross-build and clean configurability (…
pythongh-145219: Cache Emscripten build dependencies, add install-ems…