Repository navigation
The C-API for Python to C integer conversion is, to be frank, a mess. #102471
Description
Activity
We also need a few functions for querying and extracting the value of a Python int.
We want to query its sign:
int PyInt_IsNegative(); int PyInt_IsPositive(); int PyInt_IsZero(); int PyInt_Sign();
We want to import and export the digits of an integer, and to know how many digits there are.
GNU's MP library hasmpz_importandmpz_export, which have quite a complex API, but might be a good model to use.
In addition we should provide a constant describing the "native" number of bits per digit, so that C extensions can extract the data efficiently.mpz_importandmpz_exporttake 6 parameters each, and four of those are small numbers describing the layout. Having many int parameters is hard to read and error-prone. We should combine the layout parameters into a single struct (of 32 bits or less).E.g.
typedef struct _PyIntExportLayout { uint8_t bits_per_digit, int8_t word_endian, int8_t array_endian, uint8_t digit_size, } PyIntExportLayout; PyLongObject *PyInt_Import(PyIntExportLayout layout, size_t count, const void *data); int PyInt_Import(PyLongObject *op, PyIntExportLayout layout, size_t count, void *data); size_t PyInt_DigitCount(PyLongObject *op, uint8_t bits_per_digit); const PyIntExportLayout PY_INT_NATIVE_LAYOUT; /* Use this when possible, for speed */
Reacted by Petr ViktorinHi. I'm the primary maintainer of
gmpy2. I'd like to provide some comments with my experiences using the C-API.I use
PyLong_AsLongAndOverflowwhen I want alongvalue or immediately proceed with the full conversion of PyLong tompzas quickly as possible. Avoiding the exception is a significant performance improvement.PyLong_AsUnsignedLongAndOverflowis used occasionally when GMP is expects anunsigned long.PyLong_As[Unsigned]LongLongAndOverflowwere used withMPIRto get 64-bit values on Windows. (MPIRextendedGMPto support 64-bit native integer sizes.)gmpy2doesn't currently use them but it would be nice if they could be kept.I like your
PyIntExportLayoutidea for specifying the . I have a question about the usage ofPyInt_Import- which side owns the conversion?Is
PyInt_Importintended to access external data (i.e. thempzdata) and create aPyLong? DoesPyIntExportLayoutthen specify the format of thempzdata?Would there be a corresponding
PyInt_Exportthat exports the value of aPyLonginto an external buffer with the format of the external buffer controlled byPyIntExportLayout? If so, who owns (CPython versus gmpy2) the memory allocated to the external buffer? (Note: GMP, MPFR, and MPC can use a different memory manager than CPython....)This is reversed from the current conversion direction. For
mpztoPyLong,gmpy2asks CPython to create a newPyLongwith sufficient space to store the output ofmpz_export. And forPyLongtompz,gmpy2creates a newmpzwith sufficient space to store the output ofmpz_import.I'll add another comment to the thread about the compact format.
Thanks for all the effort in improving CPython.
casevh
- added a commit that references this issue
on Jul 4, 2024 I had plans to add
PyLong_Import()andPyLong_Export()with GMP/libtommath inspired signatures. This is too general interface which allows to support many different representations.This is too general interface which allows to support many different representations.
This is relatively complex task, which is better suited to dedicated libraries. I would be rather surprised if some arbitrary precision math library lacks mpz_import/export-like functions.
If on CPython side we will have a "view" of integers as an array of digits - the rest of work could do any math library.
Then please used different names than
PyLong_Import()/PyLong_Export().- added a commit that references this issue
on Jul 8, 2024 - added a commit that references this issue
on Aug 6, 2024 We need a more consistent API for converting from Python integers to C integers and back again.
We should support both 32 bit and word size C integers. 32 bit, because we often want to store 32 bit values to save space on 64 bit machines, or for portability. We also want to support word size integers for performance and ease of coding.I added APIs for that with 4c6dca8:
- Signed:
- PyLong_FromInt32(), PyLong_FromInt64()
- PyLong_AsInt32(), PyLong_AsInt64()
- Unsigned:
- PyLong_FromUInt32(), PyLong_FromUInt64()
- PyLong_AsUInt32(), PyLong_AsUInt64()
Reacted by Erlend E. Aasland- Signed:
We want to query its sign:
int PyInt_Sign();PyLong_GetSign()was added to Python 3.14: https://docs.python.org/dev/c-api/long.html#c.PyLong_GetSignint PyInt_IsNegative();
int PyInt_IsPositive();
int PyInt_IsZero();There is an open discussion for these functions: capi-workgroup/decisions#29
PyLong_IsPositive(),PyLong_IsNegative()andPyLong_IsZero()were added to Python 3.14: https://docs.python.org/dev/c-api/long.html#c.PyLong_IsPositive- added a commit that references this issue
on Jan 6, 2025 - added 5 commits that reference this issue
on Jan 27, 2025
The C-API has built up over 30 years, in a haphazard way. So, it is no surprise that it is a bit of a mess.
What makes it worse is that it is based around the C
longtype, which is varies in size between architectures and operating systems in odd ways.C
longs are 32 bit on (almost?) all 32 bit machines, 64 bit on most 64 bit machines, except Windows when Clongs are 32 bits on 64 bit machines. In other words, it is not a useful fixed size, likeint32_t, nor does match the machine word size, likeintptr_t.We need a more consistent API for converting from Python integers to C integers and back again.
We should support both 32 bit and word size C integers. 32 bit, because we often want to store 32 bit values to save space on 64 bit machines, or for portability. We also want to support word size integers for performance and ease of coding.
This means we want 4 functions (2 sizes, 2 directions) to convert between C and Python integers.
Currently we have:
PyLong_FromSsize_tThe C API has a function to convert Python ints to
intptr_t, but it is missing efficient overflow handling.It also has a function with efficient overflow handling,
PyLong_AsLongAndOverflow, but that returns along.Here's what we want:
PyInt_AsInt32PyInt_FromInt32PyInt_AsSsize_tPyInt_FromSsize_tI'm using
PyIntprefix, now that Python 2 is history. It makes it clearer what is the new API.Note that I'm not handling unsigned values. I think the extra bit of precision is not worth the complexity of a larger API.
And if we decide that they are, we can always add them later.
Linked PRs