Repository navigation
[C API] PEP 674: Disallow using macros (Py_TYPE and Py_SIZE) as l-value #89639
Description
Activity
The Python C API provides "AS" functions to convert an object to another type, like PyFloat_AS_DOUBLE(). These macros can be abused to be used as l-value: "PyFloat_AS_DOUBLE(obj) = new_value;". It prevents to change the PyFloat implementation and makes life harder for Python implementations other than CPython.
I propose to convert these macros to static inline functions to disallow using them as l-value.
I made a similar change for Py_REFCNT(), Py_TYPE() and Py_SIZE(). For these functions, I added "SET" variants: Py_SET_REFCNT(), Py_SET_TYPE(), Py_SET_SIZE(). Here, I don't think that the l-value case is legit, and so I don't see the need to add a way to *set* a value.
For example, I don't think that PyFloat_SET_DOUBLE(obj, value) would make sense. A Python float object is supposed to be immutable.
These macros can be abused to be used as l-value
You could simply document, "don't do that". Also if these is a need to make an assignment, you're now going to have to create a new setter function to fill the need.
We really don't have to go on thin ice converting to functions that might or might not be inlined depending on compiler specific nuances.
AFAICT, no one has ever has problems with these being macros. There really isn't a problem to be solved and the "solution" may in fact introduce new problems that we didn't have before.
Put me down for a -1 on the these blanket macro-to-inline function rewrites. The premise is almost entirely a matter of option, "macros are bad, functions are good" and a naive assumption, "inline functions" always inline.
Raymond:
AFAICT, no one has ever has problems with these being macros.
This issue is about the API of PyFloat_AS_DOUBLE(). Implementing it as a macro or a static inline function is an implementation detail which doesn't matter. But I don't know how to disallow "PyFloat_AS_DOUBLE(obj) = value" if it is defined as a macro.
Have a look at the Facebook "nogil" project which is incompatible with accessing directly the PyObject.ob_refcnt member:
"Extensions must use Py_REFCNT and Py_SET_REFCNT instead of directly accessing reference count fields"
https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/editRaymond:
You could simply document, "don't do that".
Documentation doesn't work. Developers easily fall into traps when it's possible to fall. See bpo-30459 for such trap with PyList_SET_ITEM() and PyCell_SET() macros. They were misused by two Python projects.
Raymond:
We really don't have to go on thin ice converting to functions that might or might not be inlined depending on compiler specific nuances.
Do you have a concrete example where a static inline function is not inlined, whereas it was inlined when it was a macro? So far, I'm not aware of any performance issue like that.
There were attempts to use __attribute__((always_inline)) (Py_ALWAYS_INLINE), but so far, using it was not a clear win.
I searched for "PyFloat_AS_DOUBLE.*=" regex in the PyPI top 5000 projects. I couldn't find any project doing that.
I only found perfectly safe comparisons:
traits/ctraits.c: if (PyFloat_AS_DOUBLE(value) <= PyFloat_AS_DOUBLE(low)) {
traits/ctraits.c: if (PyFloat_AS_DOUBLE(value) >= PyFloat_AS_DOUBLE(high)) {
c/_cffi_backend.c: return PyFloat_AS_DOUBLE(ob) != 0.0;
pandas/_libs/src/klib/khash_python.h: ( PyFloat_AS_DOUBLE(a) == PyFloat_AS_DOUBLE(b) );I am with Raymond on this one.
If "protecting against wrong use" is the only reason to go down the slippery path of starting to rely on compiler optimizations for performance critical operations, the argument is not good enough.
If people do use macros in l-value mode, it's their problem when their code breaks, not ours. Please don't forget that we are operating under the consenting adults principle: we expect users of the CPython API to use it as documented and expect them to take care of the fallout, if things break when they don't.
We don't need to police developers into doing so.
For PyObject, I converted Py_REFCNT(), Py_TYPE() and Py_SIZE() to static inline functions to enforce the usage of Py_SET_REFCNT(), Py_SET_TYPE() and Py_SET_SIZE(). Only a minority of C extensions are affected by these changes. Also, there is more pressure from recent optimization projects to abstract accesses to PyObject members.
I agree that it doesn't seem that "AS" functions are abused to *set* the inner string:
- PyByteArray_AS_STRING()
- PyBytes_AS_STRING()
- PyFloat_AS_DOUBLE()
If "protecting against wrong use" is the only reason to go down the slippery path of starting to rely on compiler optimizations for performance critical operations, the argument is not good enough.
Again, I'm not aware of any performance issue caused by short static inline functions like Py_TYPE() or the proposed PyFloat_AS_DOUBLE(). If there is a problem, it should be addressed, since Python uses more and more static inline functions.
static inline functions is a common feature of C language. I'm not sure where your doubts of bad performance come from.
Using static inline functions has other advantages. It helps debugging and profiling, since the function name can be retrieved by debuggers and profilers when analysing the machine code. It also avoids macro pitfalls (like abusing a macro to use it as an l-value ;-)).
On 15.10.2021 11:43, STINNER Victor wrote:
Again, I'm not aware of any performance issue caused by short static inline functions like Py_TYPE() or the proposed PyFloat_AS_DOUBLE(). If there is a problem, it should be addressed, since Python uses more and more static inline functions.
static inline functions is a common feature of C language. I'm not sure where your doubts of bad performance come from.
Inlining is something that is completely under the control of the
used compilers. Compilers are free to not inline function marked for
inlining, which can result in significant slowdowns on platforms
which are e.g. restricted in RAM and thus emphasize on small code size,
or where the CPUs have small caches or not enough registers (think
micro-controllers).The reason why we have those macros is because we want the developers to be
able to make a conscious decision "please inline this code unconditionally
and regardless of platform or compiler". The developer will know better
what to do than the compiler.If the developer wants to pass control over to the compiler s/he can use
the corresponding C function, which is usually available (and then, in many
cases, also provides error handling).Using static inline functions has other advantages. It helps debugging and profiling, since the function name can be retrieved by debuggers and profilers when analysing the machine code. It also avoids macro pitfalls (like abusing a macro to use it as an l-value ;-)).
Perhaps, but then I never had to profile macro use in the past. Instead,
what I typically found was that using macros results in faster code when
used in inner loops, so profiling usually guided me to use macros instead
of functions.That said, the macros you have inlined so far were all really trivial,
so a compiler will most likely always inline them (the number of machine
code instructions for the call would be more than needed for
the actual operation).Perhaps we ought to have a threshold for making such decisions, e.g.
number of machine code instructions generated for the macro or so, to
not get into discussions every time :-)A blanket "static inline" is always better than a macro is not good
enough as an argument, though.Esp. in PGO driven optimizations the compiler could opt for using
the function call rather than inlining if it finds that the code
in question is not used much and it needs to save space to have
loops fit into CPU caches.gareth-rees commented
on Oct 15, 2021 gareth-reesmannequinMannequinIf the problem is accidental use of the result of PyFloat_AS_DOUBLE() as an lvalue, why not use the comma operator to ensure that the result is an rvalue?
The C99 standard says "A comma operator does not yield an lvalue" in §6.5.17; I imagine there is similar text in other versions of the standard.
The idea would be to define a helper macro like this:
/* As expr, but can only be used as an rvalue. */ #define Py_RVALUE(expr) ((void)0, (expr))
and then use the helper where needed, for example:
#define PyFloat_AS_DOUBLE(op) Py_RVALUE(((PyFloatObject *)(op))->ob_fval)
- changed the title
[-][C API] Convert "AS" functions, like PyFloat_AS_DOUBLE(), to static inline functions[/-][+][C API] Disallow using PyFloat_AS_DOUBLE() as l-value[/+]on Oct 15, 2021 - changed the title
[-][C API] Convert "AS" functions, like PyFloat_AS_DOUBLE(), to static inline functions[/-][+][C API] Disallow using PyFloat_AS_DOUBLE() as l-value[/+]on Oct 15, 2021 27 remaining items
In the PyPI top 5000, I found two projects using PyDescr_TYPE() and PyDescr_NAME() as l-value: M2Crypto and mecab-python3. In both cases, it was code generated by SWIG
I created bpo-46538 "[C API] Make the PyDescrObject structure opaque" to handle PyDescr_NAME() and PyDescr_TYPE() macros. But IMO it's not really worth it to make the PyDescrObject structure opaque. It's just too much work, whereas PyDescrObject is not performance sensitive. It's ok to continue exposing this structure in public for now.
I will exclude PyDescr_NAME() and PyDescr_TYPE() from the PEP-674.
datatable-1.0.0.tar.gz
I created h2oai/datatable#3231
pickle5-0.0.12: pickle5/_pickle.c
This project is a backport targeting Python 3.7 and older. I'm not sure if it makes sense to update to it to Python 3.11.
It's the same for pysha3 which targets Python <= 3.5.
- guppy3-3.1.2: src/sets/bitset.c and src/sets/nodeset.c
I created: zhuyifei1999/guppy3#40
- scipy-1.7.3: scipy/_lib/boost/boost/python/object/make_instance.hpp
This is a vendored the Boost.org python module which has already been fixed in boost 1.78.0 (commit: January 2021) by:
boostorg/python@500194escipy should just update its scipy/_lib/boost copy.
recordclass-0.16.3: lib/recordclass/_dataobject.c + code generated by Cython
I created: https://bitbucket.org/intellimath/recordclass/pull-requests/1/python-311-support-use-py_set_size
- zodbpickle-2.2.0: src/zodbpickle/_pickle_33.c
Technically, zodbpickle works on Python 3.11 and is not impacted by the Py_SIZE() change.
_pickle_33.c redefines the Py_SIZE() macro to continue using as an l-value:
zopefoundation/zodbpickle@8d99afcI proposed a PR to use Py_SET_SIZE() explicitly:
zopefoundation/zodbpickle#64- changed the title
[-][C API] PEP 674: Disallow using macros as l-value[/-][+][C API] PEP 674: Disallow using macros (Py_TYPE and Py_SIZE) as l-value[/+]on Jan 27, 2022 - changed the title
[-][C API] PEP 674: Disallow using macros as l-value[/-][+][C API] PEP 674: Disallow using macros (Py_TYPE and Py_SIZE) as l-value[/+]on Jan 27, 2022 pysha3-1.0.2
This module must not be used on Python 3.6 and newer which has a built-in support for SHA-3 hash functions. Example:
$ python3.6 Python 3.6.15 (default, Sep 5 2021, 00:00:00) >>> import hashlib >>> h=hashlib.new('sha3_224'); h.update(b'hello'); print(h.hexdigest()) b87f88c72702fff1748e58b87e9141a42c0dbedc29a78cb0d4a5cd81By the way, building pysha3 on Python 3.11 now fails with:
[Modules/_sha3/backport.inc:78](https://github.com/python/cpython/blob/main/Modules/_sha3/backport.inc#L78):10: fatal error: pystrhex.h: No such file or directoryThe pystrhex.h header file has been removed in Python 3.11 by bpo-45434. But I don't think that it's worth it trying to port it to Python 3.11, if the module must not be used on Python 3.6 and newer.
Environment markers can be used to skip the pysha3 dependency on Python 3.6 on newer.
Example: "pysha3; python_version < '3.6'"
Strictly speaking,
pysha3is not equivalent tohashlibbecause keccak functions are different. If the improvement in #83720 is done, pysha3 may be completely replaced.However, since pystrhex.h removed from Python 3.11, the pycryptodome is the only legit library supporting Ethereum keccak.
However, since pystrhex.h removed from Python 3.11
Technically, these functions are still exported and remain available in the internal C API: Include/internal/pycore_strhex.h
If someone wants a public C API for these, it should be asked. But by default, the private functions must not be used outside Python itself: https://docs.python.org/dev/c-api/stable.html
It's not hard to copy Python/pystrhex.c (174 lines) or reimplement it.
https://peps.python.org/pep-0674/ was deferred by the SC. For now, I prefer to close this issue.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: