Skip to content

PEP 670: Convert _PyObject_SIZE() and _PyObject_VAR_SIZE() macros to functions #99845

Closed
Closed
PEP 670: Convert _PyObject_SIZE() and _PyObject_VAR_SIZE() macros to functions#99845
Labels
type-bugAn unexpected behavior, bug, or error

Description

@vstinner
vstinner
opened on Nov 28, 2022

The _PyObject_SIZE() and _PyObject_VAR_SIZE() macros should be converted to functions: see PEP 670 for the rationale.

My problem is that I don't know if the return type should be signed (Py_ssize_t) or unsigned (size_t).

CPython usage of _PyObject_SIZE():

  • Signed: 18. Implementation of __sizeof__() methods.
  • Unsigned: 0
  • Implicit cast to unsigned: 2. Calls PyObject_Malloc(_PyObject_SIZE(tp)) and gc_alloc(_PyObject_SIZE(tp), presize) where the first argument type is size_t.

CPython usage of _PyObject_VAR_SIZE():

  • Signed: 5
  • Unsigned: 1
  • Implicit cast to unsigned: 1. Call _PyDebugAllocatorStats(..., _PyObject_VAR_SIZE(&PyTuple_Type, len)) where the argument type is size_t.

To get a container length, the C API uses signed type (Py_ssize_t): PyList_Size(), PyDict_Size(), Py_SIZE(), etc.

To allocate memory, the C API prefers unsigned type (size_t): PyMem_Malloc(), PyObject_Realloc(), etc.

Python allocator functions reject size greater than PY_SSIZE_T_MAX:

void *
PyMem_RawMalloc(size_t size)
{
    /*
     * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
     * Most python internals blindly use a signed Py_ssize_t to track
     * things without checking for overflows or negatives.
     * As size_t is unsigned, checking for size < 0 is not required.
     */
    if (size > (size_t)PY_SSIZE_T_MAX)
        return NULL;
    return _PyMem_Raw.malloc(_PyMem_Raw.ctx, size);
}

Some "sizeof" functions freely mix signed and unsigned types. Example:

static PyObject *
deque_sizeof(dequeobject *deque, void *unused)
{
    Py_ssize_t res;
    Py_ssize_t blocks;

    res = _PyObject_SIZE(Py_TYPE(deque));
    blocks = (size_t)(deque->leftindex + Py_SIZE(deque) + BLOCKLEN - 1) / BLOCKLEN;
    assert(deque->leftindex + Py_SIZE(deque) - 1 ==
           (blocks - 1) * BLOCKLEN + deque->rightindex);
    res += blocks * sizeof(block);
    return PyLong_FromSsize_t(res);
}

blocks and sizeof(block) are unsigned, but res is signed.

Another problem is that _PyObject_VAR_SIZE() has an undefined behavior on integer overflow. Maybe it should return SIZE_MAX on oveflow, to make sure that Python allocator function fail (return NULL)?

Linked PRs

Activity

  1. vstinner
    added
    type-bugAn unexpected behavior, bug, or error
    on Nov 28, 2022

  2. vstinner commented on Nov 28, 2022

    @vstinner
    vstinner
    on Nov 28, 2022
    MemberAuthor

    Aha, _PySys_GetSizeOf() returns size_t, but bytesio_sizeof() stores its result in a Py_ssize_t variable.

  3. vstinner commented on Nov 28, 2022

    @vstinner
    vstinner
    on Nov 28, 2022
    MemberAuthor

    @corona10 @erlend-aasland @serhiy-storchaka: Are you ok to use an unsigned type for _PyObject_SIZE() and _PyObject_VAR_SIZE() result? See PR #99846 and PR #99847 for concrete changes to prepare converting these macros to functions.

  4. vstinner commented on Nov 28, 2022

    @vstinner
    vstinner
    on Nov 28, 2022
    MemberAuthor

    In PyPI top 5000 projects, _PyObject_SIZE() and _PyObject_VAR_SIZE() are used with signed and unsigned types.

    numpy uses unsigned:

    numpy-1.23.2/numpy/core/src/multiarray/scalartypes.c.src: const size_t size = _PyObject_VAR_SIZE(type, nitems + 1);

    Cython gdb plugin treats _PyObject_VAR_SIZE() macro result as unsigned:

    Cython-0.29.32/Cython/Debugger/libpython.py: def _PyObject_VAR_SIZE(typeobj, nitems):
Cython-0.29.32/Cython/Debugger/libpython.py: if _PyObject_VAR_SIZE._type_size_t is None:
Cython-0.29.32/Cython/Debugger/libpython.py: _PyObject_VAR_SIZE._type_size_t = gdb.lookup_type('size_t')
Cython-0.29.32/Cython/Debugger/libpython.py: ).cast(_PyObject_VAR_SIZE._type_size_t)
Cython-0.29.32/Cython/Debugger/libpython.py: _PyObject_VAR_SIZE._type_size_t = None
Cython-0.29.32/Cython/Debugger/libpython.py: size = _PyObject_VAR_SIZE(typeobj, tsize)

    JPype uses signed and unsigned types:

    JPype1-1.4.0/native/python/pyjp_module.cpp: long v = _PyObject_VAR_SIZE(type, 1)+(PyJPValue_hasJavaSlot(type)?sizeof (JPValue):0);
JPype1-1.4.0/native/python/pyjp_value.cpp: const size_t size = _PyObject_VAR_SIZE(type, nitems + 1) + sizeof (JPValue);

	Py_ssize_t offset;
JPype1-1.4.0/native/python/pyjp_value.cpp: offset = _PyObject_VAR_SIZE(type, 1);

	Py_ssize_t offset;
JPype1-1.4.0/native/python/pyjp_value.cpp: offset = _PyObject_VAR_SIZE(type, sz + 1);

    frozendict uses signed type (copy of CPython Objects/dictobject.c which also uses signed types currently):

        Py_ssize_t size, usable, res;
frozendict-2.3.4/frozendict/src/3_10/cpython_src/Objects/dictobject_original.c: res = _PyObject_SIZE(Py_TYPE(mp));

    Py_ssize_t size, usable, res;
frozendict-2.3.4/frozendict/src/3_6/cpython_src/Objects/dictobject.c: res = _PyObject_SIZE(Py_TYPE(mp));
frozendict-2.3.4/frozendict/src/3_6/cpython_src/Objects/dictobject_original.c: res = _PyObject_SIZE(Py_TYPE(mp));
frozendict-2.3.4/frozendict/src/3_7/cpython_src/Objects/dictobject_original.c: res = _PyObject_SIZE(Py_TYPE(mp));
frozendict-2.3.4/frozendict/src/3_8/cpython_src/Objects/dictobject_original.c: res = _PyObject_SIZE(Py_TYPE(mp));
frozendict-2.3.4/frozendict/src/3_9/cpython_src/Objects/dictobject_original.c: res = _PyObject_SIZE(Py_TYPE(mp));

    pickle uses signed:

        Py_ssize_t res, s;
pickle5-0.0.12/pickle5/_pickle.c: res = _PyObject_SIZE(Py_TYPE(self));

    Py_ssize_t res;
pickle5-0.0.12/pickle5/_pickle.c: res = _PyObject_SIZE(Py_TYPE(self));

    recordclass uses signed:

    recordclass-0.17.5/lib/recordclass/_dataobject.c: // Py_ssize_t size = _PyObject_SIZE(type);
recordclass-0.17.5/lib/recordclass/_litetuple.c: Py_ssize_t size = _PyObject_VAR_SIZE(tp, nitems);

    Nuitka uses unsigned:

        size_t size;
Nuitka-1.0.6/nuitka/build/static_src/HelpersCalling2.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersCalling2.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersCalling2.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersCalling2.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersCalling2.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersCalling2.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersCalling2.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersCalling2.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersCalling2.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersCalling2.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersCalling2.c: size = _PyObject_VAR_SIZE(type, tsize);

    size_t size;
Nuitka-1.0.6/nuitka/build/static_src/HelpersAttributes.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersAttributes.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersAttributes.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersAttributes.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/static_src/HelpersAttributes.c: size = _PyObject_VAR_SIZE(type, tsize);
Nuitka-1.0.6/nuitka/build/include/nuitka/allocator.h: size_t size = _PyObject_VAR_SIZE(tp, nitems);

    size_t size;
Nuitka-1.0.6/nuitka/code_generation/templates_c/CodeTemplateCallsMethodPositional.c.j2: size = _PyObject_VAR_SIZE(type, tsize);

    pyobjc-core casts implicitly _PyObject_SIZE() result to unsigned (size_t):

    pyobjc-core-8.5/Modules/objc/varlist.m: result = (PyObjC_VarList*)PyObject_Malloc(_PyObject_SIZE(&PyObjC_VarList_Type)

  5. vstinner commented on Nov 28, 2022

    @vstinner
    vstinner
    on Nov 28, 2022
    MemberAuthor

    I created PR #99848 for Objects/dictobject.c.

  6. vstinner commented on Nov 28, 2022

    @vstinner
    vstinner
    on Nov 28, 2022
    MemberAuthor

    The final change should be PR #99850 which convert the two macros to functions.

  7. vstinner
    added 2 commits that reference this issue on Nov 29, 2022

    gh-99845: Change _PyDict_KeysSize() return type to size_t (#99848)

    4246fe9

    gh-99845: Clean up _PyObject_VAR_SIZE() usage (#99847)

    74d5f61

  8. vstinner commented on Nov 30, 2022

    @vstinner
    vstinner
    on Nov 30, 2022
    MemberAuthor

    I was confused by the "SIZE" name:

    • Py_SIZE() returns the number of elements in a container like a list or a dict: it has no unit.
    • _PyObject_SIZE() is different: it returns an object size in bytes: it has an unit.

    IMO for bytes, it's better to use unsigned numbers. In the Python API, it's not an issue, it's an object. For example, _PySys_GetSizeOf() and _PyType_PreHeaderSize() return types are unsigned (size_t).

  9. vstinner
    added 3 commits that reference this issue on Nov 30, 2022

    gh-99845: Use size_t type in __sizeof__() methods (#99846)

    85dd6cb

    gh-99845: PEP 670: Convert PyObject macros to functions (#99850)

    131801d

    gh-99845: _PyObject_DictPointer(): fix dictoffset cast (#99922)

    9707bf2
  10. miss-islington
    added 2 commits that reference this issue on Dec 1, 2022

    pythongh-99845: _PyObject_DictPointer(): fix dictoffset cast (pythonG…

    ba5d5b3

    gh-99845: _PyObject_DictPointer(): fix dictoffset cast (GH-99922)

    64dae2e
  11. carljm
    added a commit that references this issue on Dec 1, 2022

    Merge branch 'main' into fixinspect

    12f7659

  12. vstinner commented on Dec 5, 2022

    @vstinner
    vstinner
    on Dec 5, 2022
    MemberAuthor

    Fixed by 131801d

  13. vstinner
    closed this as completedon Dec 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions