Flaky test_annotationlib test due to non-deterministic set iteration ordering

@JelleZijlstra

Seen in a 3.14 buildbot

test_type_repr (test.test_annotationlib.TestTypeRepr.test_type_repr) ... ok

======================================================================
FAIL: test_nonexistent_attribute (test.test_annotationlib.TestForwardRefFormat.test_nonexistent_attribute)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/buildbot/buildarea/3.14.ware-debian-x86.installed/build/target/lib/python3.14/test/test_annotationlib.py", line 161, in test_nonexistent_attribute
    self.assertEqual(epsilon_anno, support.EqualToForwardRef("some | {obj, module}", owner=f))
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: ForwardRef('some | {module, obj}', owner=[79 chars]810>) != EqualToForwardRef('some | {obj, module}',[86 chars]810>)

----------------------------------------------------------------------
Ran 110 tests in 0.079s

FAILED (failures=1)

https://buildbot.python.org/#/builders/1793/builds/879/steps/8/logs/stdio

I stuck this into Claude and it the cause was due to non-deterministic set ordering, which I think is correct. I stuck a random.shuffle(elts) below and was able to repro the issue locally:

cpython/Lib/annotationlib.py

Lines 390 to 399 in a01694d

    
           elif type(other) in (list, tuple, set): 
        
               extra_names = {} 
        
               elts = [] 
        
               for elt in other: 
        
                   new_elt, new_extra_names = self.__convert_to_ast(elt) 
        
                   if new_extra_names is not None: 
        
                       extra_names.update(new_extra_names) 
        
                   elts.append(new_elt) 
        
               ast_class = {list: ast.List, tuple: ast.Tuple, set: ast.Set}[type(other)] 
        
               return ast_class(elts), extra_names

Looks related to:

annotationlib: Improve test coverage #141174

cc @JelleZijlstra

Linked PRs

@colesbury

Hi @colesbury, this flaky test makes sense due to the non-deterministic set iteration ordering. I’d like to help investigate and possibly fix the test to make it deterministic. Could you please assign me so I can start working on it?

@Muawiya-contact

@Muawiya-contact please stop posting the same request under multiple issues, and furthermore, Jelle has already assigned himself here.

Thanks for reporting and sorry for the trouble. I'm sort of surprised this hasn't come up more, shouldn't this fail 50% of the time?

@StanFromIreland

@StanFromIreland Understood, thanks for the clarification. I’ll avoid cross-posting requests and won’t pursue assignment here since it’s already taken.

gh-144348: annotationlib: fix test that relies on set ordering (#144359)

pythongh-144348: annotationlib: fix test that relies on set ordering (p…

[3.14] gh-144348: annotationlib: fix test that relies on set ordering (…

I'm sort of surprised this hasn't come up more, shouldn't this fail 50% of the time?

I was also surprised it didn't fail more often. The elements are _Stringifier, which uses id(self) as the hash. From testing, it seems like the object address / id() are generally in order, so we nearly always get the same order, but that depends on the memory allocator.

It looks like the fixes are already merged, so I'll close this issue.

I see, that makes sense. So maybe some change to memory allocation landed and made the flake more likely to appear.

pythongh-144348: annotationlib: fix test that relies on set ordering (p…

Navigation Menu

Repository navigation

Flaky test_annotationlib test due to non-deterministic set iteration ordering #144348

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	elif type(other) in (list, tuple, set):
	extra_names = {}
	elts = []
	for elt in other:
	new_elt, new_extra_names = self.__convert_to_ast(elt)
	if new_extra_names is not None:
	extra_names.update(new_extra_names)
	elts.append(new_elt)
	ast_class = {list: ast.List, tuple: ast.Tuple, set: ast.Set}[type(other)]
	return ast_class(elts), extra_names

Flaky test_annotationlib test due to non-deterministic set iteration ordering #144348

Description

Linked PRs

Activity

Muawiya-contact commented on Jan 30, 2026

StanFromIreland commented on Jan 30, 2026

JelleZijlstra commented on Jan 31, 2026

Muawiya-contact commented on Jan 31, 2026

colesbury commented on Feb 2, 2026

JelleZijlstra commented on Feb 3, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions