`test_json` case `test_highly_nested_objects_decoding` fails under PGO on Clang 20

[zanieb]

Building 3.15.0a1 on aarch64 macOS with Clang 20 in python-build-standalone failed during the PGO test run in test_highly_nested_objects_decoding:

cpython-3.15> FAIL: test_highly_nested_objects_decoding (test.test_json.test_recursion.TestCRecursion.test_highly_nested_objects_decoding)
cpython-3.15> ----------------------------------------------------------------------
cpython-3.15> Traceback (most recent call last):
cpython-3.15>   File "/private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmp7fxaapg7/Python-3.15.0a1/Lib/test/test_json/test_recursion.py", line 77, in test_highly_nested_objects_decoding
cpython-3.15>     with self.assertRaises(RecursionError):
cpython-3.15>          ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
cpython-3.15> AssertionError: RecursionError not raised

This failure is specific to macOS, and also occurs on x86-64. I haven't reproduced it outside of python-build-standalone yet. Jump to #140125 (comment) for the root cause.

Linked PRs

• gh-140125: Increase object recursion depth for test_json from 200k to 500k #142226
• [3.14] gh-140125: Increase object recursion depth for test_json from 200k to 500k (GH-142226) #142416

[vstinner]

I cannot reproduce this issue on x86-64 Linux with clang 20. I used these commands:

./configure --enable-optimizations CC=clang LD=clang
time make
./python -m test -v test_json

My clang version: clang version 20.1.8 (Fedora 20.1.8-4.fc42).

[zanieb]

Thanks for trying to reproduce!

Just to be clear, my failure occurred while running the profile task, e.g., here, rather than during the test run after a successful build — so I'd expect it to fail during make.

We're specifically using clang 20.1.4 over in python-build-standalone. I tried to reproduce this on HEAD (2ca3c85) on my macOS machine with LLVM clang 20.1.3 from homebrew and could not — the build succeeded. I suspect there may be an interaction with another facet of our build system.

I've opened astral-sh/python-build-standalone#822 to reproduce in our CI matrix and determine if this is specific to aarch64 macOS (I encountered this locally) or occurs on other platforms.

[zanieb]

astral-sh/python-build-standalone#822 confirms that this is only an issue on aarch64 and x86-64 macOS.

I tried to reproduce in cpython with build flags that more closely resemble python-build-standalone's

export CFLAGS='-arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC  -Werror=unguarded-availability-new'
export CPPFLAGS='-arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC  -Werror=unguarded-availability-new'
export CFLAGS_JIT='-arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC  -Werror=unguarded-availability-new'
export LDFLAGS='-arch arm64 -mmacosx-version-min=11.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -LModules/_hacl -Wl,-headerpad,40'
./configure --build=aarch64-apple-darwin --host=aarch64-apple-darwin --prefix=/install --with-system-expat --with-system-libmpdec --without-ensurepip --with-tail-call-interp --enable-shared --with-mimalloc --enable-optimizations --enable-experimental-jit=yes-off ac_cv_lib_intl_textdomain=no ac_cv_func_ptsname_r=no ac_cv_func_explicit_bzero=no --with-dbmliborder=ndbm

which requires the following patch to allow LLVM 20 in the JIT (we are currently using this in 3.14 without problem)

diff --git a/Tools/jit/_llvm.py b/Tools/jit/_llvm.py
index bc3b50ffe61..c90abad38da 100644
--- a/Tools/jit/_llvm.py
+++ b/Tools/jit/_llvm.py
@@ -11,7 +11,7 @@
 import _targets
 
 
-_LLVM_VERSION = "19"
+_LLVM_VERSION = "20"
 _EXTERNALS_LLVM_TAG = "llvm-19.1.7.0"
 
 _P = typing.ParamSpec("_P")

Unfortunately, this failed with a segfault

./Programs/_freeze_module getpath ./Modules/getpath.py Python/frozen_modules/getpath.h
make[2]: *** [Python/frozen_modules/getpath.h] Segmentation fault: 11
make[2]: *** Deleting file `Python/frozen_modules/getpath.h'
make[1]: *** [profile-gen-stamp] Error 2
make: *** [profile-run-stamp] Error 2

If I use --enable-experimental-jit=no instead, the build succeeds. Unfortunately, that means our configure flags are not the sole cause of the test_json failure. I suspect the segfault is a JIT interaction with LLVM 20. I'm not sure why I get this segfault in cpython but not in python-build-standalone. It's worth noting my clang versions do not match exactly.

However, the test_json failure cannot be caused by the JIT because it is occurring on our x86-64 macOS and free-threaded builds, which have the JIT disabled.

[zanieb]

I cannot reproduce the segfault after running make clean, so I think we can disregard that for now.

[zanieb]

Looking at the test case... the failure is at

cpython/Lib/test/test_json/test_recursion.py

Lines 74 to 79 in 77eade3

	very_deep = 200000
	# test that loading highly-nested objects doesn't segfault when C
	# accelerations are used. See #12017
	with self.assertRaises(RecursionError):
	with support.infinite_recursion():
	self.loads('{"a":' * very_deep + '1' + '}' * very_deep)

which is checking that there are not segfaults when parsing a deep object.

The recursion limit should be set to 20,000 (+ current depth)

cpython/Lib/test/support/__init__.py

Lines 2398 to 2409 in 41712c4

	def infinite_recursion(max_depth=None):
	if max_depth is None:
	# Pick a number large enough to cause problems
	# but not take too long for code that can handle
	# very deep recursion.
	max_depth = 20_000
	elif max_depth < 3:
	raise ValueError(f"max_depth must be at least 3, got {max_depth}")
	depth = get_recursion_depth()
	depth = max(depth - 1, 1) # Ignore infinite_recursion() frame.
	limit = depth + max_depth
	return set_recursion_limit(limit)

The object is an order of magnitude deeper, so we should be hitting the recursion limit.

Specifically, we're failing in the cjson test, so it's just the C-accelerated JSON implementation that has different behavior

cpython/Lib/test/test_json/__init__.py

Lines 23 to 29 in 872eafd

	@unittest.skipUnless(cjson, 'requires _json')
	class CTest(unittest.TestCase):
	if cjson is not None:
	json = cjson
	loads = staticmethod(cjson.loads)
	dumps = staticmethod(cjson.dumps)
	JSONDecodeError = staticmethod(cjson.JSONDecodeError)

[zanieb]

Applying the following patch to revert 7016044 from #139232 (cc @rokm and @markshannon) resolves the issue

diff --git a/Python/ceval.c b/Python/ceval.c
index f48f412fab8..3bf1c098003 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -448,12 +448,6 @@ hardware_stack_limits(uintptr_t *top, uintptr_t *base)
     ULONG guarantee = 0;
     SetThreadStackGuarantee(&guarantee);
     *base = (uintptr_t)low + guarantee;
-#elif defined(__APPLE__)
-    pthread_t this_thread = pthread_self();
-    void *stack_addr = pthread_get_stackaddr_np(this_thread); // top of the stack
-    size_t stack_size = pthread_get_stacksize_np(this_thread);
-    *top = (uintptr_t)stack_addr;
-    *base = ((uintptr_t)stack_addr) - stack_size;
 #else
     /// XXX musl supports HAVE_PTHRED_GETATTR_NP, but the resulting stack size
     /// (on alpine at least) is much smaller than expected and imposes undue limits

You can see the CI pass in astral-sh/python-build-standalone#823

[rokm]

@zanieb Do you perhaps have a problematic build with PGO+LTO available for testing? I imagine you would need to disable the cjson test that triggers segfault to get past the profiling stage for PGO...

But it would be interesting to see if a) such a build also causes segfault in my test case from #139231, b) if it does, what the stack depth increments were between recursions. Could be that LTO causes a problem similar to #137573, and the cost of (the last) recursion step ends up exceeding the margin.

[zanieb]

Indeed! I've disabled this test in order to release 3.15.0a1 in python-build-standalone so uv python install 3.15.0a1 will get you what you've requested. I'm not sure if the failure will occur when run outside of the profiling step, I haven't checked yet.

edit: I tried the questionable invocation

PYTHONPATH=Lib uvx python@3.15 -m test test_json

and the tests passed

[rokm]

edit: I tried the questionable invocation

Can you try the example from #139231? (The recorded stack depth increments between recursion steps there are what is most interesting - i.e., whether they are suspiciously large).

Although if the test_json passes, it may be that the final PGO+LTO build is OK, and it is the intermediate (presumably just LTO-enabled) build that is problematic.

[zanieb]

❯ uvx python@3.13 test.py | tail
Recursion level: 4990, stack pointer: 0x16F3272B0, depth: 4451.33 kB
Recursion level: 4991, stack pointer: 0x16F326F20, depth: 4452.22 kB
Recursion level: 4992, stack pointer: 0x16F326B90, depth: 4453.11 kB
Recursion level: 4993, stack pointer: 0x16F326800, depth: 4454.00 kB
Recursion level: 4994, stack pointer: 0x16F326470, depth: 4454.89 kB
Recursion level: 4995, stack pointer: 0x16F3260E0, depth: 4455.78 kB
Recursion level: 4996, stack pointer: 0x16F325D50, depth: 4456.67 kB
Recursion level: 4997, stack pointer: 0x16F3259C0, depth: 4457.56 kB
Recursion level: 4998, stack pointer: 0x16F325630, depth: 4458.45 kB
Recursion Limit ok (reached level 4999)
❯ uvx python@3.15 test.py | tail
Recursion level: 10007, stack pointer: 0x16C9D19F0, depth: 8137.52 kB
Recursion level: 10008, stack pointer: 0x16C9D16B0, depth: 8138.33 kB
Recursion level: 10009, stack pointer: 0x16C9D1370, depth: 8139.14 kB
Recursion level: 10010, stack pointer: 0x16C9D1030, depth: 8139.95 kB
Recursion level: 10011, stack pointer: 0x16C9D0CF0, depth: 8140.77 kB
Recursion level: 10012, stack pointer: 0x16C9D09B0, depth: 8141.58 kB
Recursion level: 10013, stack pointer: 0x16C9D0670, depth: 8142.39 kB
Recursion level: 10014, stack pointer: 0x16C9D0330, depth: 8143.20 kB
Recursion level: 10015, stack pointer: 0x16C9CFFF0, depth: 8144.02 kB
Recursion Limit ok (reached level 10016)

(note I compiled the stackpointer.dylib with gcc as suggested there)

It's also plausible that the instrumentation from PGO is causing the problem.