Repository navigation
Not able to start an instantiated Thread in a child process through multiprocessing.Process in python 3.13 #134381
Description
Activity
Hi, I am a beginner contributor to the CPython project, I scan your issue and I believe this related to following function,
Modules/_threadmodule.cvoid _PyThread_AfterFork(struct _pythread_runtime_state *state) { // gh-115035: We mark ThreadHandles as not joinable early in the child's // after-fork handler. We do this before calling any Python code to ensure // that it happens before any ThreadHandles are deallocated, such as by a // GC cycle. PyThread_ident_t current = PyThread_get_thread_ident_ex(); struct llist_node *node; llist_for_each_safe(node, &state->handles) { ThreadHandle *handle = llist_data(node, ThreadHandle, node); if (handle->ident == current) { continue; } // Mark all threads as done. Any attempts to join or detach the // underlying OS thread (if any) could crash. We are the only thread; // it's safe to set this non-atomically. /* Issue Here, it will set `handle->state` to THREAD_HANDLE_DONE */ // handle->state = THREAD_HANDLE_DONE; handle->once = (_PyOnceFlag){_Py_ONCE_INITIALIZED}; handle->mutex = (PyMutex){_Py_UNLOCKED}; _PyEvent_Notify(&handle->thread_is_exiting); llist_remove(node); remove_from_shutdown_handles(handle); } }
There is a related PR to this topic: gh-114271
If I've made any mistakes, please point them out and I'll continue to explore the issue.
Reacted by adrian manriqueThe problem is that
_PyThread_AfterForkassumes allThreadobjects have started so it can remove them. This line is the problem:cpython/Modules/_threadmodule.c
Line 302 in e6dde10
We need to ignore threads that haven't started. That said, doing this sort of thing is incredibly weird. What's the use-case for creating a
Threadin a parent process, but then starting it in a child process?In my particular scenario, I'm running tests in client/server project. Test code for the client executes server code in a Process instance, as a precondition. Some parts of the server are implemented as Thread subclasses.
Reacted by Jiucheng(Oliver)Hi everyone, I was thinking about this today. Based on an idea from @ZeroIntensity. (It's probably a very basic implement), since this is a thread that hasn't been started yet, I can't think of any harm in keeping it. (The only trouble could be we will need another GC cycle to cleanup the dead thread)
void _PyThread_AfterFork(struct _pythread_runtime_state *state) { // gh-115035: We mark ThreadHandles as not joinable early in the child's // after-fork handler. We do this before calling any Python code to ensure // that it happens before any ThreadHandles are deallocated, such as by a // GC cycle. PyThread_ident_t current = PyThread_get_thread_ident_ex(); struct llist_node *node; llist_for_each_safe(node, &state->handles) { ThreadHandle *handle = llist_data(node, ThreadHandle, node); if (handle->ident == current) { continue; } + // keep not_start handles – they are safe to start in the child + if (handle->state == THREAD_HANDLE_NOT_STARTED){ + continue; + } // Mark all threads as done. Any attempts to join or detach the // underlying OS thread (if any) could crash. We are the only thread; // it's safe to set this non-atomically. handle->state = THREAD_HANDLE_DONE; handle->once = (_PyOnceFlag){_Py_ONCE_INITIALIZED}; handle->mutex = (PyMutex){_Py_UNLOCKED}; _PyEvent_Notify(&handle->thread_is_exiting); llist_remove(node); remove_from_shutdown_handles(handle); } }Sounds reasonable to me.
Reacted by Jiucheng(Oliver)Sounds reasonable to me.
Hi there, I wish to start draft a PR for this issue. Should I target to branch
mainor3.13, I did more test on 3.14 / main branch. It seems another issue pop up.TypeError: cannot pickle '_thread._ThreadHandle' object.This only fix the issue on 3.13 now, but I will also keep observe more detail about the new issue on
3.14andmain.Thanks for your help @ZeroIntensity @anadrianmanrique
Reacted by adrian manriquePut up a PR for the main branch. If we decide to fix this for older versions, we can automatically backport the fix.
Reacted by Jiucheng(Oliver) and adrian manriqueDone, thanks so much :)
Thank you all for the quick response/fix
This pattern will only work with "fork", not "spawn" because you can't pickle a
Thread. In3.133.14, the default start method switched from "fork" to "spawn", so this test case won't work as is. You'd need something likemultiprocessing.set_start_method("fork")in theif __name__ == '__main__'in addition to the changes proposed by @zangjiucheng. It also will not work on Windows, which doesn't support "fork".Reacted by Jiucheng(Oliver)@colesbury Thanks a bunch! I was also wondering why this crashed. I learned something new, thanks!
- added a commit that references this issue
on May 23, 2025 - added a commit that references this issue
on May 23, 2025 - added a commit that references this issue
on May 23, 2025 - added a commit that references this issue
on May 23, 2025
Bug report
Bug description:
I'm experiencing an issue specifically in python 3.13 regarding Threads in the context of multiprocessing.
The following code is working in python < 3.13
After executing the above in python 3.13 I get this output:
I managed to workaround this by delaying the Thread instance initialization, moving the Thread.init() call to start() method in the derived class, so everything gets executed in the context of the child process.
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Linked PRs