Snapshot annotations of by-value __annotate__ functions

[BioGeek]

Fixes #585. Opened as a separate PR from #594 so that the two approaches can be compared side by side; only one of them should be merged.

What actually goes wrong

A PEP 649 __annotate__ function closes over the namespace in which the annotated object was defined. For a method, that is the namespace of the class, which for an ABC contains _abc_impl (an unpicklable _abc._abc_data object):

$ python3.14 -q  # Python 3.14.6
>>> import abc
>>> class Impl(abc.ABC):
...     attr: int
...     def m(self, x: int) -> str: ...
...
>>> sorted(Impl.m.__annotate__.__closure__[0].cell_contents)
['__abstractmethods__', '__annotate_func__', '__dict__', '__doc__',
 '__firstlineno__', '__module__', '__static_attributes__', '__weakref__',
 '_abc_impl', 'm']

Reading that expression inside-out:

• Impl.m is the plain function object of the method.
• .__annotate__ is the function the compiler emits per annotated object since PEP 649: annotations are no longer evaluated at definition time, they are computed on demand by this function. Impl.m.__annotate__(Format.VALUE)
  returns {'x': int, 'return': str}, which is what a first access to Impl.m.__annotations__ triggers.
• .__closure__ exists because that function runs later and still has to resolve the names used in the annotations as they were visible where the method was defined. For a method that scope is the class body, so the annotate
  function is a closure over the class namespace.
• [0].cell_contents unwraps the single captured cell, which is the dict used as that class namespace.
• sorted(...) lists its keys.

The interesting key is _abc_impl: writing def m(self, x: int) -> str in an ABC is enough for the method's annotate function to transitively hold a reference to the class's _abc._abc_data object, which pickle cannot serialize.

As long as that function is pickled by reference nothing happens. It becomes a problem whenever cloudpickle has to pickle it by value, and functools.update_wrapper is the common way for a bare __annotate__ function
to end up somewhere cloudpickle has to look at it: since Python 3.14 __annotate__ is part of WRAPPER_ASSIGNMENTS, so it is copied into the wrapper instance's __dict__.

There are two situations where the by-value path is taken:

1. The annotated class is itself pickled by value (defined in __main__, in a notebook, or otherwise dynamic).
2. CPython builds before the fix for gh-137814, where the
   __qualname__ of a method's __annotate__ function wrongly pointed at the class (Impl.__annotate__ instead of Impl.m.__annotate__), so the by-reference lookup failed for regular importable classes too.

Situation 2 is fixed upstream, in the 3.14 branch by gh-148221 and on main by gh-137842. On a patched CPython (checked with 3.14.6 and 3.15.0b3) the module-level reproducer of #585 already works with cloudpickle master. Situation 1 still fails on every 3.14+ build, and the fix below covers both.

The change

Follow what cloudpickle already does for the annotations of dynamic functions: _function_getstate stores __annotations__, i.e. it evaluates them at pickling time rather than shipping the lazy annotate closure. Do the same for a bare __annotate__ function that has to be pickled by value: evaluate it with annotationlib.call_annotate_function(func, Format.VALUE) and pickle a small annotate function returning that snapshot, supporting the VALUE, FORWARDREF and STRING formats.

If the annotations cannot be evaluated eagerly (unresolvable forward references, for example), the reducer returns NotImplemented and the generic dynamic function reducer is used, exactly as today — so nothing that works right now changes behaviour.

• cloudpickle/cloudpickle.py: _make_eager_annotate, _eager_annotate_reduce and a single hook in Pickler._function_reduce.
• tests/cloudpickle_test.py: regression test for the update_wrapper + annotated ABC method case of TypeError: cannot pickle '_abc._abc_data' object with python 3.14 and cloudpickle 3.1.2 #585, plus tests for pickling a bare __annotate__ function and for annotations that must stay lazy.
• CHANGES.md: changelog entry.

How this differs from #594

#594 detects an __annotate__ entry copied by update_wrapper (an object with __wrapped__ and a function called __annotate__ in its state) and drops it from the pickled state.

• It loses the annotations, including in cases that are not broken. With Fix Python 3.14 update_wrapper pickling with lazy annotations #594 applied on CPython 3.14.6, a wrapper around a plain annotated module-level function — which round-trips fine on master — comes back without any
  annotations at all (annotationlib.get_annotations(clone) raises TypeError: ... does not have annotations). Before 3.14, update_wrapper copied __annotations__ onto the wrapper and it survived pickling, so this is
  a behaviour regression.
• It keys on the wrong thing. __wrapped__ is incidental; the problem is the by-value pickling of an annotate closure. A bare Impl.m.__annotate__ still fails with Fix Python 3.14 update_wrapper pickling with lazy annotations #594.
• The heuristic can misfire. Any object carrying a __wrapped__ attribute and a function named __annotate__ loses that attribute silently, including when it was set deliberately.

This PR keeps the annotations, does not need the __wrapped__ heuristic, and adds no per-object cost in reducer_override.

Validation

case	master	#594	this PR
wrapper over annotated module-level function	ok, annotations kept	ok, annotations lost	ok, annotations kept
wrapper over ABC method, importable class, CPython 3.14.4	fails	ok, annotations lost	ok, annotations kept
wrapper over ABC method, class defined in __main__	fails	ok, annotations lost	ok, annotations kept
wrapper over function with unresolvable forward reference	ok	ok, annotations lost	ok, stays lazy
bare Impl.m.__annotate__ of a dynamic class	fails	fails	ok (VALUE and STRING)

Test suite, all green: 256 passed on CPython 3.14.4 (pre-gh-137814-fix build),
257 on 3.14.6, 256 on 3.15.0b3 and 252 on 3.12.

Reproducer used for the table:

import abc
from functools import update_wrapper

import annotationlib
import cloudpickle


class FuncWrapper:
    def __init__(self, function):
        self.function = function
        update_wrapper(self, self.function)


class Base(abc.ABC):
    @abc.abstractmethod
    def m(self, x: int) -> str: ...


class Impl(Base):
    attr: int

    def m(self, x: int) -> "str":
        return str(x)


clone = cloudpickle.loads(cloudpickle.dumps(FuncWrapper(Impl().m)))
print(annotationlib.get_annotations(clone))