Repository navigation
[multiprocessing] Multiprocessing in spawn mode doesn't work when the target is a method in a unittest.TestCase subclass, when run either with unittest or with pytest #78065
Description
| BPO | 33884 |
|---|---|
| Nosy | @pitrou, @applio |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2018-06-17.10:45:32.688>
labels = ['3.7', '3.8', 'type-bug', 'library', 'invalid']
title = "[multiprocessing] Multiprocessing in spawn mode doesn't work when the target is a method in a unittest.TestCase subclass, when run either with unittest or with pytest"
updated_at = <Date 2018-06-17.18:51:59.760>
user = 'https://bugs.python.org/YoniRozenshein'bugs.python.org fields:
activity = <Date 2018-06-17.18:51:59.760>
actor = 'pitrou'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-06-17.10:45:32.688>
creator = 'Yoni Rozenshein'
dependencies = []
files = ['47645']
hgrepos = []
issue_num = 33884
keywords = []
message_count = 2.0
messages = ['319811', '319826']
nosy_count = 3.0
nosy_names = ['pitrou', 'davin', 'Yoni Rozenshein']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'pending'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33884'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']
Activity
YoniRozenshein commentedon Jun 17, 2018
multiprocessing will attempt to pickle things using ForkingPickler when starting a new process in spawn mode (in Windows this is the only mode, in Linux this is a non-default but settable mode).
When run within the context of a unit test, if it has to pickle a TestCase subclass, it encounters objects that can't be pickled. The attached file shows a minimum working example (uncomment the two commented lines under __main__ to run with pytest).
When run with unittest.main(), it raises:
TypeError: cannot serialize '_io.TextIOWrapper' object
When run with pytest.main(), it raises:
AttributeError: Can't pickle local object 'ArgumentParser.__init__.<locals>.identity'
Note that changing the _child_process in my example to a classmethod/staticmethod or moving it to a top-level function outside the class works around this issue (both with unittest and with pytest).
pitrou commentedon Jun 17, 2018
While this is limitation of the spawn and forkserver modes, I don't see how this is a bug. Also, as you point out, making the method a classmethod or staticmethod works around the limitation.
gpshead commentedon Dec 13, 2022
not a bug. many things will intentionally not be serializable.