PEP 689 -- Add an unstable C-API tier

[encukou]

On python-dev, it became clear that it would be useful to have a “semi-stable” “unstable” C-API tier which will stay stable within a minor release.

• This API will go in a new directory (Include/unstable/) - see Nick's reply
  You'll need to #define Py_USING_UNSTABLE_API.
  (name still up for bikeshedding).

• Since we're nearing Beta and there's no rush to break things, in 3.11
  you only get a warning if you try to use it without the opt-in #define.
  In 3.12 it'll fail.

• The functions will be renamed to drop the leading underscore. The old
  names will be available, and may be removed
  whenever the API changes. (Ideally, the underscore should always mark
  API that's fully private with no guarantees at all.)

• Get SC approval (this is now PEP-689 that will need acceptance)

• The API will be stable during a minor release. (As usual, for extreme
  cases, exceptions are possible with SC approval.)

• Docs:

  • Update https://devguide.python.org/c-api/
  • Update https://docs.python.org/3.10/c-api/stable.html
  • Add individual reference entries for the API
  • Document the new opt-in macro

• All ununstable API should be tested

  • PEP-523 PyCode_* functions: tested in test_code
  • ...

What should be here:

• Functions added in PEP-523
• PyCode_New, PyCode_NewWithPosOnlyArgs
• Ideally anything documented as subject to change between minor
  releases. (To be kind to users, if something is added later we should
  again have one release of compiler warnings before requiring the opt-in.
  Unless that API just changed and users would get errors anyway.)
• Cinder's dict writing API looks like it would feel at home here: add support for watching writes to selected dictionaries #91052 (comment)
• PyBytesObject.ob_shash sounds like a good candidate: see https://discuss.python.org/t/15108
• non-opaque access to frame structs and any other key APIs needed to
  implement alternate eval loops with comparable performance to the default
  eval loop (unless & until we can figure out stable public APIs that can
  deliver equivalent performance) - see Nick's reply
• C APIs that provide access to compiled code whether in AST or opcode
  form (the API itself may be stable, but the compiled code isn't, so this is
  kinda covered by your last point) - see Nick's reply
• _Py_HashDouble, see comment below

---

First PR: #91789

[vstinner]

This API will go in a new directory (Include/semi-stable/) - see Nick's reply

For the directory name, I suggest to omit - or _ and just put the two words as semistable, it's easier to type.

C compilers don't handle well header files with the same names in different directories. For Include/cpython/, all header files fail with #error if they are included directly. IMO we can do the same here to ensure that Include/semistable/ is not added to the search paths by mistake.

Another option is to add a prefix to each file, like pycore_ for the internal C API. For example, Include/internal/pycore_object.h is related to Include/object.h. But that's more annoying and confusing.

I would prefer that C extensions just include the main <Python.h> and that's it (no change). I would prefer that C extensions don't include directly <semistable/frame.h> for example.

[vstinner]

UPDATE: I completed my list, I forgot some removed functions.

Document the new opt-in macro

When building Python itself, does the Py_BUILD_CORE macro imply defining the Py_USING_SEMI_STABLE_API macro?

For me, the internal API (Py_BUILD_CORE) is a superset of semi-stable API (Py_USING_SEMI_STABLE_API).

APIs that provide access to compiled code whether in AST or opcode form (the API itself may be stable, but the compiled code isn't, so this is kinda covered by your last point)

In Python 3.10, Python-ast.h was moved to the internal C API as pycore_ast.h, and many API related to AST have been moved to the internal C API:

• PyST_GetScope()
• PySymtable_Build()
• PySymtable_BuildObject()
• PySymtable_Free()
• Py_SymtableString()
• Py_SymtableStringObject()
• PyAST_Compile()
• PyAST_CompileEx()
• PyAST_CompileObject()
• PyFuture_FromAST()
• PyFuture_FromASTObject()
• PyParser_ASTFromFile()
• PyParser_ASTFromFileObject()
• PyParser_ASTFromFilename()
• PyParser_ASTFromString()
• PyParser_ASTFromStringObject()
• PyArena_New()
• PyArena_Free()
• PyArena_Malloc()
• PyArena_AddPyObject()
• see: https://docs.python.org/dev/whatsnew/3.10.html#id4

Are there remaining "AST related" functions in the C API?

Obviously, compile() can still be called with PyCF_ONLY_AST flag, or call ast.parse(). But this API returns Python objects, not C structures.

[vstinner]

I would be nice to move the PyTypeObject to the semi-stable API, but IMO it's not possible to remove it from the default API because way too many C extensions use it. Modify all these extensions to define Py_USING_SEMI_STABLE_API is just annoying.

[encukou]

OK, I'll do semistable/, and I already have #error in my WIP.

When building Python itself, does the Py_BUILD_CORE macro imply defining the Py_USING_SEMI_STABLE_API macro?

Yes.

Are there remaining "AST related" functions in the C API?

@ncoghlan?

I would be nice to move the PyTypeObject to the semi-stable API

Definitely not in 3.11.

[encukou]

The old underscore-prefixed names can be deprecated static inline functions, rather than #defines as in the original plan.

[ncoghlan]

I forgot (or simply didn't notice) that the AST APIs had been migrated to the internal API.

I still think they'd be good candidates for being declared semi-stable - part of the rationale for the new tier is reducing the temptation for 3rd parties to include the internal API headers, and these were considered useful enough to be public APIs prior to 3.10

[vstinner]

Which projects use the C API of AST functions removed in Python 3.10? What is the advantage of adding back this removed API?

These functions were undocumented, excluded from the limited C API and had no test. The removal is related to the PEG parser PEP 617 which removed the old parser. The implementation modified the C code and in C, it was no longer possible to create struct _mod (mod_ty) which were arguments of the removed AST functions.

In Python 3.8, you needed the parser to get a node* object to then get a mod_ty object. But the whole C API of the parser has been removed in Python 3.10: the struct _node (node) type has been removed.

In Python 3.10 and 3.11, it's possible to browse AST using the Python API. Just use the Python ast module. IMO it's enough. There is no need to add a public C API for that.

[vstinner]

The old underscore-prefixed names can be deprecated static inline functions, rather than #defines as in the original plan.

If there is no plan to deprecate the old names, macros are fine. Like the current code in Include/cpython/abstract.h:

// Backwards compatibility aliases for API that was provisional in Python 3.8
#define _PyObject_Vectorcall PyObject_Vectorcall
#define _PyObject_VectorcallMethod PyObject_VectorcallMethod
#define _PyObject_FastCallDict PyObject_VectorcallDict
#define _PyVectorcall_Function PyVectorcall_Function
#define _PyObject_CallOneArg PyObject_CallOneArg
#define _PyObject_CallMethodNoArgs PyObject_CallMethodNoArgs
#define _PyObject_CallMethodOneArg PyObject_CallMethodOneArg

By the way, IMO it's time to deprecate these aliases :-)

Then nice advantage of a static inline function is that it can be decorated with Py_DEPRECATED(3.11).

[encukou]

Haha, _PyInterpreterState_GetEvalFrameFunc and _PyInterpreterState_SetEvalFrameFunc, are not mentioned in PEP 523 at all, but their docs say See PEP 523...

I guess they fall under Functions added in PEP-523. Off with their underscores!

[vstinner]

Haha, _PyInterpreterState_GetEvalFrameFunc and _PyInterpreterState_SetEvalFrameFunc, are not mentioned in PEP 523 at all, but their docs say See PEP 523...

These functions should be made public in the semi-stable API. The PyInterperterState structure was made opaque in Python 3.8 (released in 2019), PEP 523 was approved in 2016. I added these functions to keep PEP 523 usable. Otherwise, you had to include the internal C API to use the PEP, hum...

[encukou]

@vstinner I see the function is documented to take PyFrameObject: https://docs.python.org/3.10/c-api/init.html?highlight=pyframeevalfunction#c._PyFrameEvalFunction
But it actually takes a _PyInterpreterFrame. Should that be moved to semi-stable API too?

[encukou]

I found the commit that changed it and asked there: #88756 (comment)

[vstinner]

If you ask me, I suggest to move these functions and type to the semi-stable API:

• _PyFrameEvalFunction type
• _PyInterpreterState_GetEvalFrameFunc()
• _PyInterpreterState_SetEvalFrameFunc()
• PyCode_New()
• PyCode_NewWithPosOnlyArgs()

I would prefer to not expose _PyEval_EvalFrameDefault() directly. It should be the default value of _PyInterpreterState_GetEvalFrameFunc(). But if someone considers that it should be exposed, I'm also fine with that.