Repository navigation
Re-entering pairwise.__next__() leaks references #109786
Description
Activity
Let's get the other issue fixed first. This one is somewhat exotic and the PR makes a mess of the code.
I don't see your leak reproducer, wrote one based on your test:
Leak reproducer
Whole code at Attempt This Online!
incref(x)artificially increases the refcount ofx:def incref(x): class I: count = 0 def __iter__(self): return self def __next__(self): self.count += 1 if self.count == 1: return next(pairs) if self.count == 3: return x return None pairs = pairwise(I()) next(pairs)
Demo increasing an object's refcount by 10000:
x = object() print(f'refcount before:', sys.getrefcount(x)) for _ in range(10000): incref(x) gc.collect() print('refcount after:', sys.getrefcount(x))
Output:
refcount before: 2 refcount after: 10002Demo leaking 10000 objects of 10 kB each:
tracemalloc.start() for _ in range(10000): incref(bytes(10000)) gc.collect() print('leaked memory:', tracemalloc.get_traced_memory()[0] // 10**6, 'MB')
Output:
leaked memory: 100 MBI added them as tests. You can run tests with the
-Roption to see leaks.Reacted by Stefan Pochmann- added a commit that references this issue
on Dec 4, 2023
Re-entering the
__next__()method of itertools.pairwise leaks references, because old hold a borrowed reference when the__next__()method of the underlying iterator is called. It may potentially lead to the use of a freed memory and a crash, but I only have reproducer for leaks.Even if the re-entrant call is not very meaningful, it should correctly count references. There are several ways to fix this issue. I choose the simplest one which matches the current results (only without leaks) in most of simple tests, even if there is no other reasons to prefer these results.
cc @rhettinger
Linked PRs