Repository navigation
Runaway recursion on 3.13 and higher for _PY_FRAME_GENERAL #132744
Description
Activity
Can this be closed? I think we still need the manual backport.
I think Mark intends to backport it but I'm ambivalent
@markshannon, looks like this needs a manual backport.
I marked this easy to welcome contributors to backport this to 3.13. You don't need to implement anything new, please instead see the already merged PR and follow the backport procedure here https://devguide.python.org/contrib/core-team/committing/#backporting-changes-to-an-older-version. the backport has to be manual.
Hello everyone! Can I work on this issue?
@skv0zsneg yes please.
Reacted by Kliment LamonovFor anyone wanting to contribute. These are the steps:
- Copy the changes in bytecodes.c and the test to the 3.13 branch from my original PR.
- Run
make regen-caseson Linux orbuild.bat --regenon Windows. - Open PR.
I'm hesitant to backport the fix to 3.13, as it might break existing code.
3.13.8 commit ebccd1de88d indeed broke
ruamelin a hyper-specific test case (only with pytest-xdist) and in a totally inexplicable way.In that specific test configuration, a default integer function parameter becomes a
ruamel.yaml.reader.Readerobject!!?! Hilarity ensues. This makes absolutely no sense.While the specific test configuration that reproduces requires a bit of a setup effort, once it's set up then
git bisectis 100% deterministic and lands exactly on ebccd1de88d. 0% repro without that commit, 100% repro with it.See zephyrproject-rtos/west#908 (comment) for complete details.
@marc-hb anything that this breaks should be buggy. You should up the sys recursion limit for that test.
You should up the sys recursion limit for that test.
Interesting suggestion, thanks. I tried
sys.setrecursionlimit(100)andsys.setrecursionlimit(10000)and they only make things faster or slower but they cause absolutely no functional change whatsoever. All the failing configurations still fail and all the passing configurations still pass.anything that this breaks should be buggy.
So I bisected
ruameland the git bisect unambiguously landed on some 0.16.3 commit that merely moves someyaml_versionfield from one class to another which I can't possibly relate to anything that is happening...To make sure, I reverted that ruamel 0.16.3 commit on top of the latest 0.18.6 release, solved a few minor conflicts and that revert does make the test pass again!!
Revert "move YAML directive info to scanner for TAG parsing of 1.2 URI"
Reverts old 0.16.3 commit
https://sourceforge.net/p/ruamel-yaml/code/ci/f57c3e16091a6dec8cde656d22afba305f03bf77
on top of latest release 0.18.6.
(= git commit marc-hb/ruamel-yaml@0df0bd6 in git mirror
https://github.com/pycontribs/ruamel-yaml)This is crazy, I've never seen anything like it. This type of elusive bug usually involves non-determinism but not in this case! In this case git bisects are never ambiguous and all configurations that fail, fail exactly the same.
Any idea?
Interesting suggestion, thanks. I tried sys.setrecursionlimit(100) and sys.setrecursionlimit(10000) and they only make things faster or slower but they cause absolutely no functional change whatsoever. All the failing configurations still fail and all the passing configurations still pass.
I think I just got a breakthrough and I need to take that back. Apologies for using this issue for https://en.wikipedia.org/wiki/Rubber_duck_debugging
Instead of making "exponential" variations, I just tried these tiny ones (all other things being equal)
sys.setrecursionlimit(99) # PASS
sys.setrecursionlimit(100) # FAIL
sys.setrecursionlimit(101) # PASSSo now I suspect some subtle
try/catchbug in ruamel is being triggered by this change.I can see this happening with other libraries!
Link to tentative fix just for the record:
Data point from production (Windows 11, CPython 3.13.5): a test double declared as
def fake(path, **kw)that recursed into itself never raised RecursionError
(sys.getrecursionlimit() == 1000). It reached a depth of 5.27 million frames in 2.4 s
while allocating ~400-800 MB/s, and exhausted 128 GB of RAM plus the page file three
times in one day (two resource-exhaustion shutdowns and one 0xEF bugcheck).
The same function without**kwraises RecursionError at depth 998 as expected, so the
_PY_FRAME_GENERALpath is the trigger here too (not only default arguments).Minimal reproduction:
n = 0 def f(path, **kw): global n n += 1 if n >= 200_000: return return f(path) f("x") # returns after 200,000 frames on 3.13.5; RecursionError expected
Hoping this helps prioritize the 3.13 backport (gh-138032). Thank you.
Hoping this helps prioritize the 3.13 backport (#138032). Thank you.
This backport was merged more than 1 year ago and released in 3.13.8...
Note it also had a 3.13-specific regression, see above and in the summary I just posted in 138032.
That regression was fixed by PR #145015 merged 6+ months ago and released in 3.13.13Data point from production (Windows 11, CPython 3.13.5)
3.13.5 ??
Bug report
Bug description:
Reported here https://discuss.python.org/t/infinite-recursion/88900
The problem is that normal Calls check the recursion limit remaining, but
_PY_FRAME_GENERALdoes not.I'm hesitant to backport the fix to 3.13, as it might break existing code.
CPython versions tested on:
CPython main branch, 3.13
Operating systems tested on:
No response
Linked PRs