gh-121153: Fix some errors with use of _PyLong_CompactValue()

[serhiy-storchaka]

• The result has type Py_ssize_t, not intptr_t.
• Type cast between unsigned and signed integer types should be explicit.
• Downcasting should be explicit.
• Fix integer overflow check in sum().

• Issue: Incorrect use of _PyLong_CompactValue() #121153

[vstinner]

Can _PyLong_CompactValue() really be outside [LONG_MIN; LONG_MAX] range? It should be a single digit in the range [-PyLong_BASE; PyLong_BASE] and should even fit into a C int.

Why not changing _PyLong_CompactValue() return type to long? It would make this code way simpler.

[vstinner]

(I'm not sure, correct me if I'm wrong.)

[serhiy-storchaka]

I do not know. But currently it is defined as returning Py_ssize_t. If we change its return value, we can change the code that uses it.

It perhaps does not make difference for most or all currently supported platforms, but it is easier to write the code that handles all theoretically possible cases than hope that the simpler code which works on your platform will work the same way on different platforms with different compilers.

[vstinner]

@markshannon: Is there a reason for _PyLong_CompactValue() return type to be Py_ssize_t rather long or even int?

[serhiy-storchaka]

Returning int from _PyLong_CompactValue() would simplify the code a lot.

[vstinner]

Returning int from _PyLong_CompactValue() would simplify the code a lot.

I created PR gh-121536 to only do that (don't try to simply code using _PyLong_CompactValue()).

[markshannon]

@markshannon: Is there a reason for _PyLong_CompactValue() return type to be Py_ssize_t rather long or even int?

Yes.
The representation of PyLongObject is internal. It is an implementation detail that it happens to return a value in the range -1<<30 <= val <= 1<<30. A different representation could well return values in the range -1<<62 <= val < 1<<62 (for 64 bit machines).

And please don't use long for new APIs, use a stdint.h type like intptr_t.

[serhiy-storchaka]

Changing the returning type to intptr_t needs more changes. So I'm going to merge this PR first, and if anybody want to change the returning type, it will be a separate PR.

[bedevere-app]

GH-121900 is a backport of this pull request to the 3.13 branch.

[bedevere-app]

GH-121901 is a backport of this pull request to the 3.12 branch.