Repository navigation
multiprocessing forkserver main contains dead main_path= handling code #126631
Description
Activity
It did silently break something:
__main__can no longer be pre-loaded.import multiprocessing import os print(f"importing {__name__} ({__file__}) pid={os.getpid()}") def runner(): print(f"running in {os.getpid()}") if __name__ == '__main__': multiprocessing.set_start_method('forkserver') multiprocessing.set_forkserver_preload(['__main__']) N=4 procs = [multiprocessing.Process(target=runner) for _ in range(N)] for p in procs: p.start() for p in procs: p.join()
Currently:
importing __main__ (/home/duaneg/src/cpython/data/126631/repro.py) pid=946571 importing __mp_main__ (/home/duaneg/src/cpython/data/126631/repro.py) pid=946574 running in 946574 importing __mp_main__ (/home/duaneg/src/cpython/data/126631/repro.py) pid=946575 running in 946575 importing __mp_main__ (/home/duaneg/src/cpython/data/126631/repro.py) pid=946576 running in 946576 importing __mp_main__ (/home/duaneg/src/cpython/data/126631/repro.py) pid=946577 running in 946577If we fix it (and presumably before 3.4, although I haven't tested):
importing __main__ (/home/duaneg/src/cpython/data/126631/repro.py) pid=946458 importing __mp_main__ (/home/duaneg/src/cpython/data/126631/repro.py) pid=946460 running in 946461 running in 946462 running in 946463 running in 946464Having said that, no-one noticing for eleven years suggests it isn't actually required. Perhaps it would be better to simplify things and remove the code? I'll post a quick and dirty fix, for reference.
- added a commit that references this issue
on Jun 9, 2025 FWIW one place where
__main__preloading is explicitly used "to speed up tests" are the fork server tests themselves. The fixed version does indeed use a little less CPU (a quick & crude check shows ~2% improvement), however the actual elapsed time is not significantly affected as these tests spend most of their time waiting and sleeping.In fact, the fork server's default preload list is
['__main__'], so this bug affects all uses of fork server that don't explicitly set preload to not include__main__. Given that, I am more inclined to think we should fix this: CPU/energy savings may be small for each individual use, but will quickly add up across all fork server use globally. However, it also raises the risk, as it will affect all multiprocessing users who don't override the default settings.In trying to reshape the repro script into a regression test, I ran into some truly bizarre behaviour: it was working when run directly but produced spurious output when run as a unit test, even with the exact same command line and environment! It turns out the fork server is not flushing
stdoutbefore forking, so if any of the modules it preloads writes to a bufferedstdout/stderrbut doesn't flush, it will be inherited by the child process(es). This is partially masked by this bug, since it is most likely to be an issue for the__main__module. I'll open a separate issue for this.- added a commit that references this issue
on Jun 18, 2025 - marked forkserver slow to start because of __main__ preloading issue #137996 as a duplicate of this issue
on Sep 7, 2025 14 remaining items
- moved this from In Progress to Done in Multiprocessing issues
on Nov 22, 2025
Metadata
Metadata
Assignees
Labels
Projects
- Status
Bug report
Bug description:
The
multiprocessingmodule's forkserver start methodmultiprocessing.forkserver.mainhas two arguments that are values passed from the parent process to the forkserver when first launching it.main_path=andsys_path=. Via code inspection while fixing and testing #126538:There is no possible way for
main_path=to be set on the multiprocessing's forkserver.main() call since 2013's 9a76735 for #64145 as shipped in 3.4 (which also introduced the forkserver feature).the
forkserver.main(...)call is constructed in Lib/multiprocessing/forkserver.py getting its two possible allowed named args fromspawn.get_preparation_data()inLib/multiprocessing/spawn.pywhich was changed to set"init_main_from_name"in the kwargs dict it returns instead of"main_path"in the above change. Effectively making themain_path=handling code dead inforkserver.main.We should either remove the dead code, or determine if it was intended to support something that has been silently broken when using the forkserver start method for the past 11 years, and if so, add an explicit test for that and adapt it to use
"init_main_from_name"as the spawn code does.CPython versions tested on:
3.12, 3.13, 3.14, CPython main branch
Operating systems tested on:
Linux
Linked PRs
__main__(GH-135295) #138607__main__(GH-135295) #138609