Skip to content

forkserver slow to start because of __main__ preloading issue #137996

Closed as duplicate
Closed as duplicate
forkserver slow to start because of __main__ preloading issue#137996
Assignees
gpshead
Labels
stdlibStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or error

Description

@emaxx-google
emaxx-google
opened on Aug 20, 2025

Bug report

Bug description:

When switching a Python project from the default multiprocessing fork mode to forkserver, I've noticed the processes in the pool got significantly slower to start. Specifying preload modules didn't help.

Debugging internals of multiprocessing showed that most of the time (50 ms in each process in my test) is spent in _fixup_main_from_path(data['init_main_from_path']):

cpython/Lib/multiprocessing/spawn.py

Line 246 in eae9d7d

_fixup_main_from_path(data['init_main_from_path'])

This seemed surprising because __main__ was mentioned in the preload parameter, however I've noticed that forkserver.py tries to populate the main_path parameter from spawn.get_preparation_data():

cpython/Lib/multiprocessing/forkserver.py

Lines 149 to 151 in eae9d7d

desired_keys = {'main_path', 'sys_path'}
data = spawn.get_preparation_data('ignore')
main_kws = {x: y for x, y in data.items() if x in desired_keys}

However, the latter only writes the path to the init_main_from_path parameter:

cpython/Lib/multiprocessing/spawn.py

Line 202 in eae9d7d

d['init_main_from_path'] = os.path.normpath(main_path)

The end effect is that the __main__ module wasn't preloaded in practice, and every child process had to re-run the main script. Unless I'm missing something, the logic in forkserver.py needs to get main_path from the value of init_main_from_path?

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

Activity

  1. emaxx-google
    added
    type-bugAn unexpected behavior, bug, or error
    on Aug 20, 2025
  2. ZeroIntensity
    added
    stdlibStandard Library Python modules in the Lib/ directory
    topic-multiprocessing
    on Aug 20, 2025

  3. XChaitanyaX commented on Aug 22, 2025

    @XChaitanyaX
    XChaitanyaX
    on Aug 22, 2025
    Contributor

    I am new here, is this a good first issue for me to work on?
    @emaxx-google can you please share the testcase or the benchmark script.

  4. picnixz commented on Aug 22, 2025

    @picnixz
    picnixz
    on Aug 22, 2025
    Member

    I am new here, is this a good first issue for me to work on?

    I don't think so. If you want a good first issue, check for the ones with the easy label.

  5. emaxx-google
    added a commit that references this issue on Aug 22, 2025

    pythongh-137996: Fix path to __main__ in forkserver preload logic

    7dcf6e1

  6. emaxx-google commented on Aug 22, 2025

    @emaxx-google
    emaxx-google
    on Aug 22, 2025
    Author

    I have a PR already, but as I'm not deeply familiar with the internals I'm not 100% sure about it.

  7. gpshead
    self-assigned this
    on Aug 23, 2025
  8. emaxx-google
    added a commit that references this issue on Aug 28, 2025

    Use forkserver multiprocessing mode (#305)

    0316a25

  9. duaneg commented on Sep 1, 2025

    @duaneg
    duaneg
    on Sep 1, 2025
    Contributor

    See also issue #126631 and comments, and the associated pull request #135295.

  10. emaxx-google commented on Sep 1, 2025

    @emaxx-google
    emaxx-google
    on Sep 1, 2025
    Author

    See also issue #126631 and comments, and the associated pull request #135295.

    Thanks for pointing this out! The implementation change looks almost identical, and you have a regression test already; do you know what's blocking that change from landing?

  11. duaneg commented on Sep 1, 2025

    @duaneg
    duaneg
    on Sep 1, 2025
    Contributor

    Thanks for pointing this out! The implementation change looks almost identical, and you have a regression test already; do you know what's blocking that change from landing?

    No worries at all, and yeah, our independent fixes are basically the same, which is nice 🙂

    I don't think anything is blocking the change as such, but AIUI the maintainers and core devs in general are overworked and have to prioritise where they spend their time and attention. Before you raised this issue it was a fairly academic fix: it had been broken for many years, seemingly without anyone noticing. It is useful to know it has caused a noticeable performance regression in real-world use!

  12. gpshead
    closed this as a duplicateof multiprocessing forkserver main contains dead main_path= handling code #126631 on Sep 7, 2025
  13. gpshead
    added a commit that references this issue on Sep 7, 2025

    gh-126631: gh-137996: fix pre-loading of `__main__` (GH-135295)

    0912b3a
  14. miss-islington
    added a commit that references this issue on Sep 7, 2025

    pythongh-126631: pythongh-137996: fix pre-loading of `__main__` (pyth…

    92c663b
  15. gpshead
    added 2 commits that reference this issue on Sep 7, 2025

    [3.13] pythongh-126631: pythongh-137996: fix pre-loading of `__main__` (

    c376dea

    [3.13] gh-126631: gh-137996: fix pre-loading of `__main__` (GH-135295) (

    9a6137a
  16. lkollar
    added a commit that references this issue on Sep 9, 2025

    pythongh-126631: pythongh-137996: fix pre-loading of `__main__` (pyth…

    2ff72e8
  17. gpshead
    added a commit that references this issue on Oct 8, 2025

    [3.14] gh-126631: gh-137996: fix pre-loading of `__main__` (GH-135295) (

    64f0e2d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

stdlibStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions