Skip to content

test_multiprocessing_spawn.test_manager: _TestCondition hung (20 min timeout) on AMD64 RHEL8 3.x #110206

Closed
Closed
test_multiprocessing_spawn.test_manager: _TestCondition hung (20 min timeout) on AMD64 RHEL8 3.x#110206
Labels
testsTests in the Lib/test dirtopic-multiprocessing

Description

@vstinner
vstinner
opened on Oct 2, 2023
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/test/_test_multiprocessing.py", line 1426, in f
    woken.release()

This frame comes from _TestCondition.

AMD64 RHEL8 3.x:

0:04:43 load avg: 7.07 [464/467] test_zipfile passed -- running (2): test.test_multiprocessing_spawn.test_manager (1 min 12 sec), test_math (1 min 22 sec)
0:04:52 load avg: 6.14 [465/467] test_xmlrpc passed -- running (2): test.test_multiprocessing_spawn.test_manager (1 min 21 sec), test_math (1 min 30 sec)
0:05:19 load avg: 4.38 [466/467] test_math passed (1 min 57 sec) -- running (1): test.test_multiprocessing_spawn.test_manager (1 min 48 sec)
0:05:49 load avg: 2.66 running (1): test.test_multiprocessing_spawn.test_manager (2 min 18 sec)
0:06:19 load avg: 1.61 running (1): test.test_multiprocessing_spawn.test_manager (2 min 48 sec)
0:06:49 load avg: 0.97 running (1): test.test_multiprocessing_spawn.test_manager (3 min 18 sec)
(...)
0:22:19 load avg: 0.00 running (1): test.test_multiprocessing_spawn.test_manager (18 min 48 sec)
0:22:49 load avg: 0.00 running (1): test.test_multiprocessing_spawn.test_manager (19 min 18 sec)
0:23:19 load avg: 0.00 running (1): test.test_multiprocessing_spawn.test_manager (19 min 48 sec)
0:23:31 load avg: 0.00 [467/467/1] test.test_multiprocessing_spawn.test_manager worker non-zero exit code (Exit code 1)
Process Process-44:
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/test/_test_multiprocessing.py", line 1426, in f
    woken.release()
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/multiprocessing/managers.py", line 1059, in release
    return self._callmethod('release')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/multiprocessing/managers.py", line 840, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/multiprocessing/managers.py", line 263, in serve_client
    self.id_to_local_proxy_obj[ident]
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: '7f528ad142e0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/multiprocessing/managers.py", line 265, in serve_client
    raise ke
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/multiprocessing/managers.py", line 259, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
                              ~~~~~~~~~^^^^^^^
KeyError: '7f528ad142e0'
---------------------------------------------------------------------------
Timeout (0:20:00)!
Thread 0x00007f3635005740 (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/multiprocessing/popen_fork.py", line 27 in poll
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/multiprocessing/popen_fork.py", line 43 in wait
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/multiprocessing/process.py", line 149 in join
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/unittest/case.py", line 597 in _callCleanup
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/unittest/case.py", line 673 in doCleanups
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/unittest/case.py", line 640 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/unittest/case.py", line 692 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/unittest/runner.py", line 240 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64/build/Lib/test/support/__init__.py", line 1155 in _run_suite

test_multiprocessing_spawn.test_manager passed when re-run.

build: https://buildbot.python.org/all/#/builders/185/builds/5160

Linked PRs

Activity

  1. vstinner
    added
    testsTests in the Lib/test dir
    topic-multiprocessing
    on Oct 2, 2023

  2. vstinner commented on Oct 5, 2023

    @vstinner
    vstinner
    on Oct 5, 2023
    MemberAuthor

    Similar error on PPC64LE RHEL8 Refleaks 3.x: https://buildbot.python.org/all/#/builders/384/builds/892

  3. vstinner commented on Oct 5, 2023

    @vstinner
    vstinner
    on Oct 5, 2023
    MemberAuthor

    I failed to reproduce the issue on my Fedora 38 by stressing my laptop (12 logical CPUs) with:

    ./python -m test test_multiprocessing_spawn.test_manager -m WithManagerTestCondition -F -j70 -W

    I ran the test for 3 min 30 sec.

  4. vstinner commented on Nov 8, 2023

    @vstinner
    vstinner
    on Nov 8, 2023
    MemberAuthor

    I didn't see this failure recently, I close the issue.

  5. vstinner
    closed this as completedon Nov 8, 2023

  6. colesbury commented on Mar 4, 2025

    @colesbury
    colesbury
    on Mar 4, 2025
    Contributor

    I've seen this a few times recently. For example on the GH ubuntu-24.04-arm runner: https://github.com/python/cpython/actions/runs/13642688178/job/38135745891?pr=130811

    test_notify_all (test.test_multiprocessing_spawn.test_manager.WithManagerTestCondition.test_notify_all) ... Process Process-43:
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/_test_multiprocessing.py", line 1621, in f
    woken.release()
    ~~~~~~~~~~~~~^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/multiprocessing/managers.py", line 1067, in release
    return self._callmethod('release')
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/multiprocessing/managers.py", line 848, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/multiprocessing/managers.py", line 264, in serve_client
    self.id_to_local_proxy_obj[ident]
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 'ff241[728](https://github.com/python/cpython/actions/runs/13642688178/job/38135745891?pr=130811#step:22:729)daa0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/multiprocessing/managers.py", line 266, in serve_client
    raise ke
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/multiprocessing/managers.py", line 260, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
                              ~~~~~~~~~^^^^^^^
KeyError: 'ff241728daa0'
---------------------------------------------------------------------------
Timeout (0:10:00)!
  7. colesbury
    reopened this on Mar 4, 2025
  8. colesbury
    added 2 commits that reference this issue on Mar 6, 2025

    pythongh-110206: Fix multiprocessing test_notify_all

    c06d0f1

    gh-110206: Fix multiprocessing test_notify_all (#130933)

    c476410
  9. miss-islington
    added 2 commits that reference this issue on Mar 7, 2025

    pythongh-110206: Fix multiprocessing test_notify_all (pythonGH-130933)

    5113e1e

    pythongh-110206: Fix multiprocessing test_notify_all (pythonGH-130933)

    7f85038
  10. colesbury
    added 2 commits that reference this issue on Mar 7, 2025

    [3.12] gh-110206: Fix multiprocessing test_notify_all (GH-130933) (#1…

    4230697

    [3.13] gh-110206: Fix multiprocessing test_notify_all (GH-130933) (#1…

    94b94d0

  11. colesbury commented on Mar 7, 2025

    @colesbury
    colesbury
    on Mar 7, 2025
    Contributor

    test_notify_all should be fixed now, but see #130954 for a similar issue with test_notify_n.

  12. colesbury
    closed this as completedon Mar 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dirtopic-multiprocessing

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions