Repository navigation
Multiprocessing: bug with Native ID for threading.mainthread() #82888
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-11-19.21:35:38.239>
created_at = <Date 2019-11-05.22:39:45.970>
labels = ['3.8', 'type-bug', 'library', '3.9']
title = 'Multiprocessing: bug with Native ID for threading.mainthread()'
updated_at = <Date 2019-11-19.22:27:08.361>
user = 'https://github.com/jaketesler'bugs.python.org fields:
activity = <Date 2019-11-19.22:27:08.361>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-11-19.21:35:38.239>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2019-11-05.22:39:45.970>
creator = 'jaketesler'
dependencies = []
files = []
hgrepos = []
issue_num = 38707
keywords = ['patch']
message_count = 8.0
messages = ['356070', '356202', '356218', '356543', '356987', '356989', '356992', '356998']
nosy_count = 5.0
nosy_names = ['pitrou', 'vstinner', 'davin', 'miss-islington', 'jaketesler']
pr_nums = ['17088', '17261']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38707'
versions = ['Python 3.8', 'Python 3.9']
Activity
jaketesler commentedon Nov 5, 2019
I have encountered a minor bug with the new
threading.get_native_id()featureset in Python 3.8. The bug occurs when creating a new multiprocessing.Process object on Unix (or on any platform where the multiprocessing start_method is 'fork' or 'forkserver').When creating a new process via fork, the Native ID in the new MainThread is incorrect. The new forked process' threading.MainThread object inherits the Native ID from the parent process' MainThread instead of capturing/updating its own (new) Native ID.
See the following snippet:
As you can see, the new Process object's MainThread.native_id attribute matches that of the MainThread of its parent process.
Unfortunately, I'm not too familiar with the underlying mechanisms that Multiprocessing uses to create forked processes.
I believe this behavior occurs because (AFAIK) a forked multiprocessing.Process copies the MainThread object from its parent process, rather than reinitializing a new one. Looking further into the multiprocessing code, it appears the right spot to fix this would be in the multiprocessing.Process.bootstrap() function.
I've created a branch containing a working fix - I'm also open to suggestions of how a fix might otherwise be implemented.
If it looks correct I'll create a PR against the CPython 3.8 branch.
See the branch here: https://github.com/jaketesler/cpython/tree/fix-mp-native-id
Thanks all!
-Jake
vstinner commentedon Nov 7, 2019
Can you please create a PR?
jaketesler commentedon Nov 8, 2019
@vstinner PR created :)
#17088
jaketesler commentedon Nov 13, 2019
PR was updated with tests and is ready for core developer review and then the merge to cpython:master. After that (if I understand correctly) a backport will automatically get picked into the 3.8 branch if there aren't any conflicts.
miss-islington commentedon Nov 19, 2019
New changeset c6b20be by Miss Islington (bot) (Jake Tesler) in branch 'master':
bpo-38707: Fix for multiprocessing.Process MainThread.native_id (GH-17088)
c6b20be
miss-islington commentedon Nov 19, 2019
New changeset 829593a by Miss Islington (bot) in branch '3.8':
bpo-38707: Fix for multiprocessing.Process MainThread.native_id (GH-17088)
829593a
pitrou commentedon Nov 19, 2019
Thank you Jake for the report and PR!
vstinner commentedon Nov 19, 2019
Thanks for the fix. That was an interesting bug ;-) I like the simplicity of the fix.