Main thread livelocks on CPython 3.11 when a second thread dies while PendingCallback_DestroyQueue runs Python code #535

Versions: greenlet 3.5.2 and 3.5.6, CPython 3.11.15/3.11.16, Linux x86_64. The reproducer below also hangs 3/3 on stock 3.0.3, 3.1.1, 3.2.5, 3.3.2 and 3.4.0 (CPython 3.11). On CPython 3.12.3 with 3.5.2 the second pending call is still scheduled, but nothing hangs (6/6).

Summary

When a thread that owns greenlet state exits, AddToCleanupQueue schedules PendingCallback_DestroyQueue once the
queue goes from 0 to 1. The drain loop pops each state before DestroyOne(), so the queue is empty while the state is
being destroyed. Destroying a main greenlet can run Python code (weakref callbacks, __del__). If another thread dies in
that window, AddToCleanupQueue sees size() == 1 and schedules a second pending call while the first is still
executing.

On CPython 3.11 that is fatal under a trace function. make_pending_calls() is busy, so it returns without clearing
calls_to_do, and the eval breaker stays set. DO_TRACING on RESUME checks the eval breaker before advancing
next_instr, so the next Python function entry is re-dispatched forever, with the GIL held. coverage.py with
concurrency = greenlet hits this routinely: its C tracer keys a weakref.WeakKeyDictionary by greenlet.getcurrent()
in every thread, so every dying thread's main greenlet carries a Python callback.

Deterministic reproducer (no coverage needed)

import faulthandler, sys, threading, time, weakref
import greenlet
from greenlet._greenlet import get_pending_cleanup_count

faulthandler.dump_traceback_later(20, exit=True)
release_b = threading.Event()
state = {}

def _in_cleanup(_ref):          # runs on the main thread inside PendingCallback_DestroyQueue
    release_b.set()
    deadline = time.monotonic() + 10
    while get_pending_cleanup_count() == 0 and time.monotonic() < deadline:
        time.sleep(0.001)
    state["b_queued"] = get_pending_cleanup_count() > 0
    (lambda: None)()            # first function entry after B was queued -> hangs on 3.11

def _a():
    state["ref"] = weakref.ref(greenlet.getcurrent(), _in_cleanup)

def _b():
    greenlet.getcurrent()
    release_b.wait()

sys.settrace(lambda *a: None)
b = threading.Thread(target=_b); b.start()
a = threading.Thread(target=_a); a.start(); a.join()
for _ in range(5000):
    time.sleep(0.001)
    if "b_queued" in state and get_pending_cleanup_count() == 0:
        break
b.join()
print("GREEN", state.get("b_queued"))

3.5.2: hangs 3/3 at <lambda>. 3.5.6: hangs 3/3. With the patch below: GREEN True 3/3.

Proposed fix

Track whether the drain is running (under thread_states_to_destroy_lock), and don't schedule another pending call
while it is. The running drain picks up the new state on its next iteration.

+    static bool& draining()
+    {
+        static bool flag = false;
+        return flag;
+    }
...
-        if (mod_globs->thread_states_to_destroy.size() == 1) {
+        if (mod_globs->thread_states_to_destroy.size() == 1 && !draining()) {
...
                 if (mod_globs->thread_states_to_destroy.empty()) {
+                    draining() = false;
                     break;
                 }
+                draining() = true;
                 to_destroy = mod_globs->take_next_to_destroy();