Skip to content

[subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression) #90228

Closed
Closed
[subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)#90228
Labels
3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifeinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-regextopic-subinterpreterstype-crashA hard crash of the interpreter, possibly with a core dump

Description

@graysky
graysky
mannequin
opened on Dec 14, 2021
BPO 46070
Nosy @vstinner, @encukou, @ericsnowcurrently, @ndjensen, @hroncok, @uckelman, @corona10, @miss-islington, @shihai1991, @erlend-aasland, @bsteffensmeier, @M-Reimer, @prahal
PRs
  • bpo-46070: Fix asyncio initialisation guard #30423
  • [3.10] bpo-46070: Fix asyncio initialisation guard (GH-30423) #30453
  • [3.9] bpo-46070: Fix asyncio initialisation guard (GH-30423) #30454
  • bpo-46070: Revert "bpo-36854: Move _PyRuntimeState.gc to PyInterpreterState (GH-17287)" #30564
  • [3.10] bpo-46070: Revert "bpo-36854: Move _PyRuntimeState.gc to PyInterpreterState (GH-17287)" #30565
  • [3.9] bpo-46070: Revert "bpo-36854: Move _PyRuntimeState.gc to PyInterpreterState (GH-17287)" #30566
  • bpo-46070: _PyGC_Fini() untracks objects #30577
  • [3.10] bpo-46070: _PyGC_Fini() untracks objects (GH-30577) #30578
  • [3.9] bpo-46070: _PyGC_Fini() untracks objects (GH-30577) #30579
  • [3.9] bpo-46070: _PyGC_Fini() untracks objects (GH-30577) #30580
    		•
    Files
  • win_py399_crash_reproducer.py: Reproducer that intermittently crashes on windows
  • bug.py
  • pyobject_ob_interp.patch
  • sqlite3_crash.py: Reproducer that crashes 90% of the time
  • bug.py_asyncio_cpustressed-crash.log: asyncio bug.py crash while running stress -c nproc --all
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2021-12-14.10:57:50.149>
labels = ['interpreter-core', 'type-bug', '3.9', '3.10', '3.11', 'expert-subinterpreters']
title = '[subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)'
updated_at = <Date 2022-03-24.21:16:20.771>
user = 'https://bugs.python.org/graysky'

    bugs.python.org fields:

    activity = <Date 2022-03-24.21:16:20.771>
actor = 'prahal'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Subinterpreters']
creation = <Date 2021-12-14.10:57:50.149>
creator = 'graysky'
dependencies = []
files = ['50494', '50497', '50560', '50699', '50700']
hgrepos = []
issue_num = 46070
keywords = ['patch']
message_count = 54.0
messages = ['408520', '408539', '408540', '408541', '408558', '408568', '408633', '408662', '408664', '408665', '409245', '409255', '409461', '409573', '409686', '409772', '409778', '409780', '409795', '409798', '409802', '409962', '409965', '409966', '409970', '409980', '409983', '409984', '409991', '410010', '410014', '410442', '410444', '410446', '410447', '410493', '410497', '410498', '410500', '410505', '410507', '410509', '410510', '410513', '410517', '410518', '410520', '415954', '415955', '415963', '415964', '415970', '415975', '415977']
nosy_count = 15.0
nosy_names = ['vstinner', 'petr.viktorin', 'eric.snow', 'ndjensen', 'hroncok', 'uckelman', 'corona10', 'miss-islington', 'shihai1991', 'erlendaasland', 'graysky', 'bsteffensmeier', 'M-Reimer', 'jokot3', 'prahal']
pr_nums = ['30423', '30453', '30454', '30564', '30565', '30566', '30577', '30578', '30579', '30580']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46070'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    Activity

    1. graysky commented on Dec 14, 2021

      @graysky
      grayskymannequin
      on Dec 14, 2021
      MannequinAuthor

      Seems as though cpython is broken when working with subinterpreters. The problematic change could be (d0d2965) affecting import.c.[1] Reverting this commit and rebuilding python fixes the issues on my system with some scripts that import sqlite, for example, the Kodi plugin YouTube[2] and the IMDB Trailers plugin. Others have reported similar breakage with other python code[3].

      Example output when the bug manifests:

      ERROR <general>: Traceback (most recent call last):
ERROR <general>:   File "/usr/lib/python3.10/sqlite3/dbapi2.py", line 83, in register_adapters_and_converters
ERROR <general>:     
ERROR <general>: register_converter("timestamp", convert_timestamp)
ERROR <general>: 
ERROR <general>: KeyError
ERROR <general>: : 
ERROR <general>: 'timepart_full'
ERROR <general>:                                              
ERROR <general>: Exception ignored deletion of interned string failed
ERROR <general>: :

      References:

      1. d0d2965
      2. Plugin incompatible with python 3.10.1 anxdpanic/plugin.video.youtube#255
      3. https://bbs.archlinux.org/viewtopic.php?id=272121
    2. graysky
      added
      3.10 (EOL)end of life
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      on Dec 14, 2021

    3. ericsnowcurrently commented on Dec 14, 2021

      @ericsnowcurrently
      ericsnowcurrently
      on Dec 14, 2021
      Member

      (related: bpo-44059)

      Presumably the problem relates to global state used in different interpreters leading to an inconsistent state in the crashing extension (or its dependencies).

      @graysky, do you know if this was a problem before Python 3.8?

    4. ericsnowcurrently
      added
      topic-subinterpreters
      3.9 (EOL)end of life
      3.11only security fixes
      type-bugAn unexpected behavior, bug, or error
      on Dec 14, 2021

    5. encukou commented on Dec 14, 2021

      @encukou
      encukou
      on Dec 14, 2021
      Member

      Interned strings were broken in #64257, see bpo-46006. Maybe that's also the issue here?

    6. 58 remaining items

    7. ezio-melotti
      added this to Subinterpreterson Apr 15, 2022
    8. ezio-melotti
      moved this to Todo in Subinterpreterson Apr 15, 2022
    9. AlexWaygood
      added
      topic-regex
      on Apr 19, 2022

    10. vstinner commented on Apr 28, 2022

      @vstinner
      vstinner
      on Apr 28, 2022
      Member

      So far, only released Python v3.11.0a4 include my "_PyGC_Fini() untracks objects" fix. The 3.9 and 3.10 branch include my fix, but there is no release including it yet.

      Scheduled releases:

      I managed to reproduce the bug.py crash with main branch up to commit 12c0012 .

      Oh really? This version include my fix. I cannot reproduce it on the current main branch (commit ea2f5bc) nor at commit 12c0012. Which commands did you type to build Python and reproduce the crash? What is your OS and C compiler?

    11. vstinner commented on Apr 28, 2022

      @vstinner
      vstinner
      on Apr 28, 2022
      Member

      The original issue is fixed. Please open a new issue for sqlite3_crash.py.

    12. vstinner commented on Apr 28, 2022

      @vstinner
      vstinner
      on Apr 28, 2022
      Member

      I created #92036 for sqlite3_crash.py bug.

    13. erlend-aasland commented on Apr 28, 2022

      @erlend-aasland
      erlend-aasland
      on Apr 28, 2022
      Contributor

      Should this be closed then?

    14. vstinner commented on Apr 28, 2022

      @vstinner
      vstinner
      on Apr 28, 2022
      Member

      I kept this issue open for this reason:

      It would be nice to add some tests.

    15. vstinner commented on Apr 28, 2022

      @vstinner
      vstinner
      on Apr 28, 2022
      Member

      My notes about this bug: https://twitter.com/VictorStinner/status/1481783100191428611

    16. heitbaum
      added a commit that references this issue on May 17, 2022

      Python3: update to 3.9.13

      5c655c7
    17. serhiy-storchaka
      added
      type-crashA hard crash of the interpreter, possibly with a core dump
      and removed
      type-bugAn unexpected behavior, bug, or error
      on Jul 3, 2026

    18. serhiy-storchaka commented on Jul 3, 2026

      @serhiy-storchaka
      serhiy-storchaka
      on Jul 3, 2026
      Member

      _sre was converted to multi-phase init with per-interpreter state, which removed the root cause. Importing _sre in 160 subinterpreters across 8 parallel threads runs cleanly on a debug build of main. Closing as fixed.

    19. serhiy-storchaka
      closed this as completedon Jul 3, 2026
    20. github-project-automation
      moved this from Todo to Done in Subinterpreterson Jul 3, 2026
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifeinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-regextopic-subinterpreterstype-crashA hard crash of the interpreter, possibly with a core dump

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions