Class-scope comprehension with lambda raises SystemError

[steve-s]

Bug report

Bug description:

class D:
    vals = [(lambda: __class__)() for _ in range(1)]

gives

SystemError: compiler_lookup_arg(name='__class__') with reftype=5 failed in D; freevars of code <lambda>: ('__class__',)

tested on main (f4bda4d) and on 3.12.8 (debug build). Issue #149806 looks related, but I tried that example on #149806 and it's not fixed by it.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-150700: Fix class-scope inline comprehensions when nested scopes reference __class__ and friends #150735
• [3.15] gh-150700: Fix class-scope inline comprehensions when nested scopes reference __class__ and friends (GH-150735) #151211
• [3.14] gh-150700: Fix class-scope inline comprehensions when nested scopes reference __class__ and friends (GH-150735) #151212

[terryjreedy]

Same on Windows back to 3.12. In 3.11, "NameError: cannot access free variable 'class' where it is not associated with a value in enclosing scope".

[StanFromIreland]

Out of curiosity, how was this found? I'm asking because it seems quite like our other OSS-Fuzz things (like the one you linked), and I'm wondering if we missed a finding?

[steve-s]

As part of implementation of PEP 709 in GraalPy done by my colleague @msimacek, during review I asked Codex to look for corner cases and gaps around the code handling the name resolution and it came up with this (manifested by different internal error). Then we realized it is also not working on CPython.

[johnslavik]

MREs: class C: [lambda: __class__ for () in ()], class C: [lambda: super() for () in ()]

[StanFromIreland]

Also don't forget __classdict__ and __conditional_annotations__ (which I suppose I technically broke some time ago), they can also be dropped.

[johnslavik]

Also don't forget __classdict__ and __conditional_annotations__ (which I suppose I technically broke some time ago), they can also be dropped.

I was just about to mention them, yes. class C: [lambda: __conditional_annotations__ for () in ()], class C: [lambda: __classdict__ for () in ()] reproduce too.

[johnslavik]

#149806 is unrelated because it fixes annotation visiting code and the issue here lies in inline_comprehension.
This is an edge case of #120295 (which is the likely cause of regression).

In inline_comprehension(), when __class__ / __classdict__ / __conditional_annotations__ appears as FREE in a comprehension's symbol table because a nested scope captured it (e.g. these lambdas), this name is still discarded from comp_free unconditionally. This prevents drop_class_free() from seeing it, so the appropriate ste_needs_(...) flag is never set on the enclosing class. That leads to codegen_make_closure() throwing SystemError when it couldn't find __class__ / __classdict__ / __conditional_annotations__ in the class's cellvars.

We could just only discard from comp_free when no child scope (e.g. a lambda) still needs the name as FREE. When a child scope does need it, keep it in comp_free so drop_class_free() can set the appropriate flag and the class creates the implicit cell.