Implement PEP 793 – PyModExport

[encukou]

Feature or enhancement

Proposal:

See PEP 793. I'll copy the abstract for reference:

In this PEP, we propose a new entry point for C extension modules, by which one can define a module using an array of PyModuleDef_Slot structures without an enclosing PyModuleDef structure. This allows extension authors to avoid using a statically allocated PyObject, lifting the most common obstacle to making one compiled library file usable with both regular and free-threaded builds of CPython.

To make this viable, we also specify new module slot types to replace PyModuleDef’s fields, and to allow adding a token similar to the Py_tp_token used for type objects.

We also add an API for defining modules from slots dynamically.

The existing API (PyInit_*) is soft-deprecated. (That is: it will continue to work without warnings, and it’ll be fully documented and supported, but we plan to not add any new features to it.)

---

• Initial implementation (gh-140550: Initial implementation of PEP 793 – PyModExport #140556)
• Docs:
  • Reference docs (gh-140550: PEP 793 reference documentation #141197)
    • Soft-deprecate PyInit_* (gh-140550: PEP 793 reference documentation #141197)
    • Better explanation of module state (gh-140550: PEP 793 reference documentation #141197)
    • Give an example for tokens (gh-140550: PEP 793 reference documentation #141197)
  • What's New
    • a notice for DYNLOADFILE users
  • Porting guide HOWTO
    • Document that the export hook runs without GIL in FT builds (see #141780)
  • Tutorial update (gh-134160: Start "Extending and embedding" with a Diataxis-style tutorial #142314)
  • Mentions in PyImport_AppendInittab & friends
  • Make sure references to new API are linked:
    • PyMod_Execin PyImport_CreateModuleFromInitfunc docs is linked (gh-140550: PEP 793 reference documentation #141197)
    • PyMODEXPORT_FUNC in blurb for #141672
    • blurb for #141780
• xxlimited.c update (gh-140550: Update xxlimited with 3.15 limited API & PEP 697 #142827)
  • Remove/replace _Py_OPAQUE_PYOBJECT in xxlimited.c (by 3.15.0 beta)

Optional stretch goals:

• Remove the PyInit function from Lib/test/test_cext/extension.c (gh-140550: Remove PyInit function from test_cext #141511)
• Speed & size improvements
  • importdl: load the DLL once
  • bool for the GIL flag (gh-140550: Use a bool for the Py_mod_gil value #141519)
• No longer require spec.name in PyModule_FromSlotsAndSpec
• No longer require static allocation for Py_mod_methods
• Add a PyModule_GetStateWithToken() function

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

• gh-140550: Initial implementation of PEP 793 – PyModExport #140556
• gh-140550: Run make regen-limited-abi #141056
• gh-140550: PEP 793 reference documentation #141197
• gh-140550: Remove PyInit function from test_cext #141511
• gh-140550: Use a bool for the Py_mod_gil value #141519
• gh-140550: Correct error message for PyModExport (PEP 793) hook #142583
• gh-140550: Update xxlimited with 3.15 limited API & PEP 697 #142827
• gh-140550: allow slots that repeat information from PyModuleDef #144340

See also:

• gh-141671: PyMODINIT_FUNC: apply __declspec(dllexport) on Windows #141672
• gh-134160: Start "Extending and embedding" with a Diataxis-style tutorial #142314

[encukou]

I'd appreciate a review of the reference docs update: #141197