Repository navigation
_py_abc Python implementation of abc is not thread-safe #130095
Description
Bug report
The update to the invalidation counter is not thread-safe and can lose updates in some Python implementations:
Failures seen on:
- Python 3.14t
- Python 3.9
- pypy3.10
- pypy3.11
But not on Python 3.10-3.14 with GIL due to limited GIL switch opportunities.
Lines 54 to 70 in 05e89c3
For example, consider the following repro, adapted from test_abc.test_registration_basics:
import _py_abc as abc # Use Python implementation of ABCs!!
import threading
import os
import sys
sys.setswitchinterval(1e-6)
N = 5
def run(b):
b.wait()
class A(metaclass=abc.ABCMeta):
pass
A.register(int)
if not isinstance(42, A):
print("Oops!")
os._exit(1)
def main():
for _ in range(10000):
threads = []
b = threading.Barrier(N)
for _ in range(N):
t = threading.Thread(target=run, args=(b,))
threads.append(t)
t.start()
for t in threads:
t.join()
if __name__ == "__main__":
main()
Activity
colesbury commentedon Feb 13, 2025
I'm not really sure what to do here. We could add a
threading.Lock()to protect_abc_invalidation_counter, but I'm not sure we want to import the threading module in_py_abc.py.cc @cfbolz in case you have thoughts on this.
picnixz commentedon Feb 14, 2025
Not sure if this is relevant: https://github.com/python/cpython/actions/runs/13335219139/job/37248893358?pr=129175 (I don't know if the test is testing the Python or the C implementation).
EDIT: I think this is the reproducer you had. But now we caught it in the CI
pythongh-130095: Mark _py_abc tests as not thread-safe
colesbury commentedon Feb 14, 2025
Thanks @picnixz. I put up a PR to avoid running them with
--parallel-threadsfor now until we figure out if and how to make_py_abcthread-safe.cfbolz commentedon Feb 14, 2025
yeah, importing threading in
_py_abcleads to a circular import problem if_abcis not available. What about usingitertools.count? that's supposed to be thread-safe, isn't it?cfbolz commentedon Feb 14, 2025
hm, doesn't quite work, because
itertools.counthas no way to read the current value, only a way to increase the count and get the next value.sergey-miryanov commentedon Feb 14, 2025
If I'm right, the problem comes from three bytecode operations:
LOAD_ATTR,BINARY_OP, andSTORE_ATTR. I think there are two ways to fix this: using a specialized operation for atomic increment/update or the same built-in function. Current main branch has only three uses of_Py_atomic_add_uint64-qsbr,_asynciomoduleand_abc, so there are no existing ways to do atomic update from python. I believe the quickest way to add private built-in for this, but the right one to add atomic ops.Sorry if I'm obvious.
colesbury commentedon Feb 14, 2025
The pure Python companion modules like
_py_abcaren't really used by CPython in normal circumstances outside of tests. They are primarily used by other Python implementations like PyPy. I don't think it makes much sense to consider CPython specific changes to bytecode.https://peps.python.org/pep-0399/
sergey-miryanov commentedon Feb 14, 2025
Thanks for the link - it is useful for my understanding.
I’m stick to atomic ops, but I’m sure you have much more and deeper understanding.
gh-130095: Mark `_py_abc` tests as not thread-safe (gh-130131)
cfbolz commentedon Feb 14, 2025
@colesbury if you are specifically concerned about pypy, we don't use
_py_abc.pyanymore either.colesbury commentedon Feb 14, 2025
Thanks for the clarification. I also noticed a failure when I substituted
abcfor_py_abcwithpypy311, so there might be a thread-safety issue there as well.If neither pypy nor CPython use
_py_abcoutside of tests, I'm not going to spend much more time on this.sergey-miryanov commentedon Oct 20, 2025
Do we want to fix this or may be deprecate it (if it is not used anymore by pypy and CPython)?
Usage over 15_000 pypi packages