Free-threaded build: memory grows with every foreign thread that calls an extern "Python" callback #286

Activity

1. gudoshnikovn

   added a commit that references this issue on Oct 13, 2026

   fix: batch callbacks enter cffi one at a time on free-threaded CPython

   ...

   3bef529
2. gudoshnikovn commented on Oct 14, 2026

   

   gudoshnikovn

   on Oct 14, 2026

   Author

   More actions

   The same mechanism can also crash the process, not only leak. On a free-threaded build, a foreign thread that enters an extern "Python" callback segfaults inside PyThreadState_Clear(). That call is thread_canary_free_zombies() clearing the state of another thread that has already exited.

   Reproducer

   build.py:

   import cffi
   
   ffibuilder = cffi.FFI()
   ffibuilder.cdef("""
       extern "Python" void callback(int);
       void run_threads(int n, int calls);
   """)
   ffibuilder.set_source("_repro", r"""
   #include <pthread.h>
   #include <unistd.h>
   
   static void callback(int);
   
   static void *worker(void *calls) {
       for (int i = 0; i < (int)(long)calls; i++) {
           usleep(50);  /* some work between the calls */
           callback(i);
       }
       return NULL;
   }
   
   /* Start n new threads, each calling a Python callback `calls` times, and join them. */
   static void run_threads(int n, int calls) {
       pthread_t threads[64];
       for (int i = 0; i < n; i++)
           pthread_create(&threads[i], NULL, worker, (void *)(long)calls);
       for (int i = 0; i < n; i++)
           pthread_join(threads[i], NULL);
   }
   """)
   
   if __name__ == "__main__":
       ffibuilder.compile()

   repro.py:

   from _repro import ffi, lib
   
   shared = []
   
   
   @ffi.def_extern()
   def callback(i):
       shared.append(i)  # a list that every thread grows
   
   
   for _ in range(2000):
       lib.run_threads(2, 30)
   print("ok")

   python build.py && python -X faulthandler repro.py
   

   It crashes within a few seconds, usually in under one:

   Fatal Python error: Segmentation fault
   
   <Cannot show all threads while the GIL is disabled>
   Stack (most recent call first):
     <no Python frame>
   
   Current thread's C stack trace (most recent call first):
     python3.14t, at _PyMem_AbandonDelayed+0x114
     python3.14t, at _PyMem_AbandonDelayed+0x114
     python3.14t, at PyThreadState_Clear+0x3b0
     _cffi_backend.cpython-314t-darwin.so, at gil_ensure+0xa4
     _cffi_backend.cpython-314t-darwin.so, at cffi_call_python+0x50
     _repro.cpython-314t-darwin.so, at worker+0x5c
     libsystem_pthread.dylib, at _pthread_start+0x88
   

   Under lldb, the faulting instruction is in mimalloc. _PyMem_AbandonDelayed() frees the blocks queued for delayed freeing, and the PyMem_Free() it calls ends in _mi_page_retire():

   * thread #3, stop reason = EXC_BAD_ACCESS (code=1, address=0x5e8)
     * frame #0: python`_mi_page_retire + 140
       frame #1: python`_PyMem_AbandonDelayed + 276
       frame #2: python`PyThreadState_Clear + 944
       frame #3: _cffi_backend.cpython-314t-darwin.so`gil_ensure + 164
       frame #4: _cffi_backend.cpython-314t-darwin.so`cffi_call_python + 80
       frame #5: _repro.cpython-314t-darwin.so`callback(a0=0) [inlined]
       frame #6: _repro.cpython-314t-darwin.so`worker(calls=0x1e)
   

   Results

   Each row is 20 runs of the script above, or of a variant of it.

   macOS arm64, CPython 3.14.7:

   	crashed
   cffi 2.1.1, free-threaded	19 / 20
   cffi main (ed994e0), free-threaded	17 / 20
   cffi main (ed994e0) with the patch below, free-threaded	0 / 20
   cffi 2.1.1, free-threaded, PYTHON_GIL=1	0 / 20
   cffi 2.1.1, free-threaded, every callback(i) under one pthread_mutex_t in C	0 / 20
   cffi 2.1.1, free-threaded, callback appends to a new local list	0 / 20
   cffi 2.1.1, GIL	0 / 20
   ctypes, free-threaded	0 / 20
   ctypes, GIL	0 / 20

   The ctypes rows use the same thread code, compiled as a plain shared library and called with a ctypes.CFUNCTYPE(None, c_int) callback.

   Three things are needed for the crash:

   • The callback appends to a list that several threads share. When a shared list grows, the free-threaded build frees its old item array later, through QSBR. That queues a block in the worker's mem_free_queue. With a fresh local list nothing is queued, and nothing crashes.
   • New threads keep coming. Each exited thread leaves its state on the zombie list. The next thread that enters a callback clears it from its own thread.
   • The timing has to be right. With usleep(10) to usleep(100) between the calls, the script crashes. Without the sleep it ran 20 out of 20 times without a crash.

   What happens

   thread_canary_free_zombies() (misc_thread_common.h#L146-L185) is marked "This must be called with the GIL". cffi_zombie_lock covers only taking a zombie off the list. PyThreadState_Clear() and PyThreadState_Delete() run outside that lock, and on a free-threaded build no GIL serializes them either. So several entering threads can clear several exited threads' states at the same time, each from a thread that does not own that state.

   On a free-threaded build, PyThreadState_Clear() hands the exited thread's delayed frees and mimalloc heaps over from whichever thread calls it (pystate.c#L1851-L1853):

       // Merge our queue of pointers to be freed into the interpreter queue.
       _PyMem_AbandonDelayed(tstate);
   
       _PyThreadState_ClearMimallocHeaps(tstate);

   _PyMem_AbandonDelayed() frees the queued blocks right away, on the calling thread (obmalloc.c#L1429-L1459). The crash is in mimalloc's local-free path (_mi_page_retire), which mimalloc takes only when it believes the calling thread owns the block's page. I have not established why it believes that here. One guess is that a new pthread can get the same mimalloc thread id as an exited one whose heaps have not been abandoned yet. Either way, the crash needs two zombies to be cleared at the same time: with the clearing serialized (below), it is gone. ctypes, whose PyGILState_Release() deletes a state on its own thread, does not crash.

   A fix that works here

   Holding one lock across the whole of thread_canary_free_zombies() on a free-threaded build is enough for this reproducer (the "with the patch" row above):

   --- a/src/c/misc_thread_common.h
   +++ b/src/c/misc_thread_common.h
   @@ -150,6 +150,10 @@ thread_canary_free_zombies(void)
        if (cffi_zombie_head.zombie_next == &cffi_zombie_head)
            return;    /* fast path */
    
   +#ifdef Py_GIL_DISABLED
   +    static PyMutex free_lock;
   +    PyMutex_Lock(&free_lock);
   +#endif
        while (1) {
            ThreadCanaryObj *ob;
            PyThreadState *tstate = NULL;
   @@ -181,6 +185,9 @@ thread_canary_free_zombies(void)
            PyThreadState_Delete(tstate);
            //fprintf(stderr, "thread_canary_free_zombie: cleared and deleted tstate=%p\n", tstate);
        }
   +#ifdef Py_GIL_DISABLED
   +    PyMutex_Unlock(&free_lock);
   +#endif
        //fprintf(stderr, "thread_canary_free_zombie: end\n");
    }

   This restores the one guarantee the function's comment asks for: zombies are cleared one at a time. It still clears each state from a thread that is not its own, so it does not address the leak above. Letting each thread delete its own state, as PyGILState_Release() does, would fix both, but I don't know what that costs the canary design. Until cffi changes, a program can avoid the crash by entering its callbacks under one lock in C (the "under one pthread_mutex_t" row).