multiprocessing forkserver does not flush output before fork (was: preloading '__main__' with forkserver has been broken for a long time)

Bug report

The forkserver start method provides the ability to call set_forkserver_preload on the multiprocessing context to load modules into and configure the forkserver process. By doing this carefully, you can avoid having to do module loading and other work each time the forkserver process is forked to create a new process. Without doing such work, forkserver can be way slower than the traditional fork start method

You can specify the module '__main__' in the set_forkserver_preload list, and the forkserver source has special code when you do this. It ensures that the main file path does not have to be configured/loaded after each fork. To do this, in multiprocessing.forkserver.ensure_running, it calls multiprocessing.spawn.get_preparation_data and then uses the main_path entry that may be in the returned dict.

Unfortunately, 3 months after it was introduced, this functionality was broken in commit 9a76735. That commit renamed the main_path dictionary entry returned in get_preparation_data to init_main_from_path, but didn't update the use in multiprocessing.fork_server

Not having the ability to load and configure main on the forkserver ends up being unusually painful for my recent scenario, which led to tracking this down. I have a python program on a share that spawns short-lived processes at a high rate. Then multiple machines run this program from the share. Huge slowdown ensues as smbd processes on the server go crazy responding to every new process on every client reading the file and stat-ing the directory the file is contained in.

A simple fix in multiprocessing.forkserver accounting for the changed name rectifies the problem. I'll work on putting that PR together. Any thoughts on a workaround that doesn't require modifying the python source are welcome, as I imagine it will be a while until I'm on a python with the fix.

Here's a simple repro.

import time
import multiprocessing
print('hi from forkserver_repro')
def _silly(i):
    time.sleep(0.2)
    return i%2

def run_subprocesses():
    process_list=[]
    for i in range(10):
        process_list.append(multiprocessing.Process(target=_silly,args=[i]))
    for process in process_list:
        process.start()
    for process in process_list:
        process.join()
    return 0

if __name__ == '__main__':
    multiprocessing.set_start_method('forkserver')
    ctx = multiprocessing.get_context('forkserver')
    ctx.set_forkserver_preload(['__main__',])
    print('Only one more "hi from forkserver_repro" should print! More means a bug!')
    run_subprocesses()

Your environment

Ubuntu 20.04.4 LTS, CPython 3.8
Python source code examination indicates this bug is still present in the current version of CPython.

PR: [3.10] gh-98552: Fix preloading '__main__' with forkserver being broken #98924

PR: gh-98552: Fix preloading '__main__' with forkserver being broken for a #99515

PR: gh-98552: flush std streams in the multiprocessing forkserver before fork #141849

PR: [3.14] gh-98552: flush std streams in the multiprocessing forkserver before fork (GH-141849) #141851

PR: [3.13] gh-98552: flush std streams in the multiprocessing forkserver before fork (GH-141849) #141852

PR: gh-141860: Add on_error= keyword arg to multiprocessing.set_forkserver_preload #141859

PR: gh-98552: Revert (unneeded, already done elsewhere) "flush std streams in the multiprocessing forkserver before fork (#141849)" #141871

PR: [3.14] gh-98552: Revert (unneeded, already done elsewhere) "flush std streams in the multiprocessing forkserver before fork (GH-141849)" (GH-141871) #141872

PR: [3.13] gh-98552: Revert (unneeded, already done elsewhere) "flush std streams in the multiprocessing forkserver before fork (GH-141849)" (GH-141871) #141873

A PR fixing this and adding unittest coverage would be great.

BTW as you are using forkserver, be aware of the recently disclosed #97514 if you ever run on 3.9 or later. (that issue can be worked around)

Thanks for the pointer on the CVE.

The simple fix for this is pretty straightforward and I'll try to get it together quickly (first cpython PR so may take a little longer). It is just to deal with the renamed dict entry.

However, I've discovered some other "quirks" with forkserver, and was wondering if they are known or being worked on, if they could use improvement, or are expected and it's just my newness in looking at it. (I'm on linux but trying to use multiprocessing in programs that also use threading, and have gotten burned a few times by deadlocks because of it, leading me to move to using forkserver. But maybe I should be considering another way forward?)

Anyways, with the break fixed by using the right dict key to get main_path into main, preloading __main__ works, but imports in the main module can fail, because sys.path is not set in forkserver's main before importing main. (The correct sys_path is actually fed into forkserver.main, but not used?! It's been that way since forkserver was added 9 years ago...)

Not having sys.path set leads to other weirdness even when not importing main. The modules specified to preload will succeed/fail depending on the cwd of the process, so python a/foo.py behaves different then cd a; python foo.py . And I'm pretty sure I've seen the same module imported twice, once at the name given as preload and once as the name used by another module (e.g., a.my_module and my_module).

The try/catch around each preloaded module hides all the failures too. In my case, I'd much prefer to get the exception, because if the preload fails, then every process created by the forkserver is going to have to do expensive module loading itself, and I don't realize it until I notice the performance problem.

When spawn is used, we set/clear _inheriting and call prepare with the result of get_preparation_data in spawn._main. My completely uninformed inclination is to do the same in forkserver.main, instead of what's currently happening. Is that a reasonable thing to look into?

I did some experiments. Using spawn.prepare in forkserver.py to consume the dict from get_prepare_data seems to work well. It fixes sys.path to be correct so imports work regardless of cwd, fixes preloading __main__ in a less brittle way, and does a bunch of other stuff that I'm guessing should be happening too. But I don't know enough about forkserver and spawn to be certain.

I'll put together a PR for this approach too, to solicit input. forkserver.main gets its parameters from the string created in forkserver.ensure_running, so the dict from get_prepare_data has to be turned into a string. A quick way to do this and not have to worry about escaping quotes/etc is to remove the authkey from the dict, then json.dumps to utf-8 to b64encode it. Then main does the reverse to get it back. (Thought about pickle but don't know enough to know the security implications here.)

The downside (for me) of this more general solution is that it alters code on the new process side, versus the not as complete fix that is on the original process side. This means the latter can be done with monkey-patching, but the former cannot, as far as I understand.

Updating the tests for these changes led to some comical confusion. Turns out there was another bug waiting... forkserver does not flush its stdout/stderr before each fork, and that makes things really confusing.

I was using print statements in files to make sure preload was working and they were only imported in the forkserver and not in the new forked processes. But the output said otherwise, even though everything was actually working...

pythongh-98552: Fix preloading '__main__' with forkserver being broke…

I think this one is fixed now: #99515 (comment).

@aggieNick02

I think this one is fixed now: #99515 (comment).

Ah, apologies @aggieNick02 for not finding this issue and your fix when I looked at #126631. You even discovered and fixed the same stdout buffering issue that so confused me!

preloading main got fixed as noted above, while this one was overlooked as having had a fix for it already in the shuffle. thanks for tying things together. does the output buffering issue still remain?

[3.14] gh-98552: flush std streams in the multiprocessing forkserver …

[3.13] gh-98552: flush std streams in the multiprocessing forkserver …

Revert (unneeded, already done elsewhere) "pythongh-98552: flush std …

the remaining purpose i had for keeping this open wound up a dupe of #135335 which was already done elsewhere in the code that I hadn't noticed. reverting those. closing this out. a new issue #141860 tracks the continuation of your feature to be able to see the errors from preload module imports in the forkserver process.

gh-98552: Revert (unneeded, already done elsewhere) "flush std stream…

pythongh-98552: Revert (unneeded, already done elsewhere) "flush std …

[3.13] gh-98552: Revert (unneeded, already done elsewhere) "flush std…

[3.14] gh-98552: Revert (unneeded, already done elsewhere) "flush std…

preloading main got fixed as noted above, while this one was overlooked as having had a fix for it already in the shuffle. thanks for tying things together. does the output buffering issue still remain?

As you noted, this was fixed under #135335. There is still an open PR, #138686, which simplifies the unit test added for #126631. With the buffering bug fixed we can print from __main__, which makes things much easier. It would be lovely if that could be merged, assuming it looks good ofc 🙂

pythongh-98552: flush std streams in the multiprocessing forkserver b…

pythongh-98552: Revert (unneeded, already done elsewhere) "flush std …

pythongh-98552: flush std streams in the multiprocessing forkserver b…

pythongh-98552: Revert (unneeded, already done elsewhere) "flush std …

Navigation Menu

Repository navigation

multiprocessing forkserver does not flush output before fork (was: preloading 'main' with forkserver has been broken for a long time) #98552

Bug report

Your environment

3 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

multiprocessing forkserver does not flush output before fork (was: preloading '__main__' with forkserver has been broken for a long time) #98552

Description

Bug report

Your environment

Activity

gpshead commented on Oct 23, 2022

aggieNick02 commented on Oct 24, 2022

aggieNick02 commented on Oct 24, 2022

aggieNick02 commented on Oct 26, 2022

thesamesam commented on Oct 9, 2025

duaneg commented on Nov 22, 2025

gpshead commented on Nov 22, 2025

3 remaining items

gpshead commented on Nov 23, 2025

duaneg commented on Nov 23, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

multiprocessing forkserver does not flush output before fork (was: preloading 'main' with forkserver has been broken for a long time) #98552