[3.13] _PyObject_Init() assertion fails with the stable ABI on Python built with assertions (extension built with Python 3.11) #121528

mgorny · 2024-07-09T09:47:37Z

Crash report

What happened?

Consider the following extension.

C code:

#include <Python.h>

static PyObject *
foo_bar(PyObject *self, PyObject *args)
{
	Py_INCREF(PyExc_TypeError);
	PyErr_SetString(PyExc_TypeError, "foo");
	return NULL;
}

static PyMethodDef foomethods[] = {
	{"bar", foo_bar, METH_VARARGS, ""},
	{NULL, NULL, 0, NULL},
};

static PyModuleDef foomodule = {
	PyModuleDef_HEAD_INIT,
	.m_name = "foo",
	.m_doc = "foo test module",
	.m_size = -1,
	.m_methods = foomethods,
};

PyMODINIT_FUNC
PyInit_foo(void)
{
	return PyModule_Create(&foomodule);
}

setup.py:

from setuptools import setup, Extension

setup(name='foo',
      version='0',
      ext_modules=[
          Extension('foo', ['foo.c'], py_limited_api='cp38'),
      ])

If I compile the extension using Python older than 3.12, and then run the method, Python 3.13.0b3 (built --with-assertions) crashes:

$ python3.11 setup.py build_ext -i
running build_ext
building 'foo' extension
creating build
creating build/temp.linux-x86_64-cpython-311
x86_64-pc-linux-gnu-gcc -Wsign-compare -fPIC -I/usr/include/python3.11 -c foo.c -o build/temp.linux-x86_64-cpython-311/foo.o
creating build/lib.linux-x86_64-cpython-311
x86_64-pc-linux-gnu-gcc -shared build/temp.linux-x86_64-cpython-311/foo.o -L/usr/lib64 -o build/lib.linux-x86_64-cpython-311/foo.abi3.so
copying build/lib.linux-x86_64-cpython-311/foo.abi3.so -> 
$ python3.13 -c 'import foo; foo.bar()'
python3.13: ./Include/internal/pycore_object.h:284: _PyObject_Init: Assertion `_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE) || _Py_IsImmortal(typeobj)' failed.
Aborted (core dumped)

I've been able to bisect it to c32dc47 (CC @markshannon). Originally hit it in extensions using PyO3, and reported to PyO3/pyo3#4311.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.0b3 (main, Jul 4 2024, 14:30:57) [GCC 14.1.1 20240622]

Linked PRs

The text was updated successfully, but these errors were encountered:

Zheaoli · 2024-07-09T19:06:23Z

The issue is confirmed, I have found the reason, but I'm not sure this is a bug, I prefer its undefined behavior.

It's caused by the Py_INCREF different behavior between 3.11 and 3.13(or master)

TL;DR;

For the 3.11 version of the Py_INCREF, we add the reference count directly without check FYI https://github.com/python/cpython/blob/3.11/Include/object.h#L491-L504

For the 3.13 version of the Py_INCREF, we add the reference count and check if it exceeds. If it exceeds, this means that this is an immortal object, we do noting except return the function. FYI https://github.com/python/cpython/blob/3.13/Include/object.h#L796-L846

And the function is an inline function, which is means we will expand the function into the binary during the compile. So in 3.11 version, the PyExc_TypeError's reference count is change from 4294967295(the immortal flag) to 4294967296. And the process will fail here https://github.com/python/cpython/pull/116115/files#diff-2a12f738a77b362d74a65949b58c37f2affcd15ba8b1c979b63bd00223b8a456R268

We can compare the assemble code to prove it

For 3.11

0000000000001120 <foo_bar>:
    1120:	48 83 ec 08          	sub    $0x8,%rsp
    1124:	48 8b 05 9d 2e 00 00 	mov    0x2e9d(%rip),%rax        # 3fc8 <PyExc_TypeError@Base>
    112b:	48 8d 35 ce 0e 00 00 	lea    0xece(%rip),%rsi        # 2000 <_fini+0xe9c>
    1132:	48 8b 38             	mov    (%rax),%rdi
    1135:	48 83 07 01          	addq   $0x1,(%rdi)
    1139:	e8 f2 fe ff ff       	call   1030 <PyErr_SetString@plt>
    113e:	31 c0                	xor    %eax,%eax
    1140:	48 83 c4 08          	add    $0x8,%rsp
    1144:	c3                   	ret
    1145:	66 66 2e 0f 1f 84 00 	data16 cs nopw 0x0(%rax,%rax,1)
    114c:	00 00 00 00

For 3.13

0000000000001120 <foo_bar>:
    1120:	48 83 ec 08          	sub    $0x8,%rsp
    1124:	48 8b 05 9d 2e 00 00 	mov    0x2e9d(%rip),%rax        # 3fc8 <PyExc_TypeError@Base>
    112b:	48 8b 38             	mov    (%rax),%rdi
    112e:	8b 07                	mov    (%rdi),%eax
    1130:	83 c0 01             	add    $0x1,%eax
    1133:	74 02                	je     1137 <foo_bar+0x17>
    1135:	89 07                	mov    %eax,(%rdi)
    1137:	48 8d 35 c2 0e 00 00 	lea    0xec2(%rip),%rsi        # 2000 <_fini+0xe9c>
    113e:	e8 ed fe ff ff       	call   1030 <PyErr_SetString@plt>
    1143:	31 c0                	xor    %eax,%eax
    1145:	48 83 c4 08          	add    $0x8,%rsp
    1149:	c3                   	ret
    114a:	66 0f 1f 44 00 00    	nopw   0x0(%rax,%rax,1)

The patch to fix this issue is simple ,we just the code back

    if (_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE)  ||  _Py_IsImmortal(typeobj)) {
        Py_INCREF(typeobj);
    }

I'm not sure this patch has side effect or not. I may need @markshannon help

Zheaoli · 2024-07-10T15:35:02Z

I setup a new disscussion here https://discuss.python.org/t/disable-the-inline-by-default-for-the-function-api-which-is-included-in-the-stable-abi/57832

Zheaoli · 2024-07-10T16:29:17Z

After dive more deeper, I think maybe its the user code issue.

I think you use Py_INCREF instead of Py_IncRef, Py_INCREF is not part of the stable ABI.

If you use the Py_IncRef I think the code here would be not existed

davidhewitt · 2024-07-10T18:35:21Z

The general impression I get from wjakob/nanobind#500 is that Py_INCREF is part of the limited API but is not explicitly listed due to tooling limitations. As a macro, it does not feature at the ABI level but its implementation does. Prior to 3.12, the implementation modified object reference counts directly. On 3.12 and later, the limited API implementation calls the private function _Py_IncRef.

Zheaoli · 2024-07-10T18:49:28Z

The general impression I get from wjakob/nanobind#500 is that Py_INCREF is part of the limited API but is not explicitly listed due to tooling limitations. As a macro, it does not feature at the ABI level but its implementation does. Prior to 3.12, the implementation modified object reference counts directly.

Yes, this is implemented as a Limit API/Stable ABI but actually it's not(can't guarantee the behavior and inlined). I'm not sure we should open a new discussion about this.

Without Py_LIMITED_API defined, some C API functions are inlined or replaced by macros. Defining Py_LIMITED_API disables this inlining, allowing stability as Python’s data structures are improved, but possibly reducing performance.

https://docs.python.org/3/c-api/stable.html#limited-api-scope-and-performance

Zheaoli · 2024-07-13T17:09:19Z

I think we might need a core to decide this issue should be fixed or not

@vstinner @Fidget-Spinner @markshannon @brettcannon

alex · 2024-07-13T17:12:17Z

It seems like this will break a large number of existing extensions if it's not resolved.

If the decision is not to fix this, then CPython needs to not load abi3 .sos that are impacted.

vstinner · 2024-07-13T17:31:00Z

In the limited C API version 3.12 and newer, Py_INCREF() is now implemented as a function call. You can build your C extension with Python 3.12 and the binary should work on Python 3.11 unless you use features specific to Python 3.12. Moreover, the binary (.abi3.so shared library) should work on Python 3.12 and newer.

Example.

setup.py

from setuptools import setup, Extension

setup(name='foo',
      version='0',
      ext_modules=[
          Extension('foo', ['foo.c'], py_limited_api='cp312'),
      ])

Build:

$ python3.12 setup.py build_ext -
(...)
$ ls *.so
foo.abi3.so*

It works on Python 3.11, 3.12, 3.13:

$ python3.11 -c 'import foo; foo.bar()'
TypeError: foo
$ python3.12 -c 'import foo; foo.bar()'
TypeError: foo
$ python3.13 -c 'import foo; foo.bar()'
TypeError: foo

It works even on Python 3.6:

$ python3.6 -c 'import foo; foo.bar()'
TypeError: foo

alex · 2024-07-13T17:33:38Z

My concern is primarily for the existing abi3 wheels on PyPI. The whole point is that maintainers could publish them and they'd continue to work.

vstinner · 2024-07-13T17:34:10Z

In the limited C API version 3.12 and newer, Py_INCREF() is now implemented as a function call.

Issue: PEP 683 (Immortal Objects): Implement Py_INCREF() as function call in limited C API 3.12 #105387
Discussion: https://discuss.python.org/t/limited-c-api-implement-py-incref-and-py-decref-as-function-calls/27592
Issue: Evolving the Stable ABI to use functions for INCREF, DECREF, TYPE capi-workgroup/problems#45

Zheaoli · 2024-07-13T17:37:34Z

In the limited C API version 3.12 and newer, Py_INCREF() is now implemented as a function call. You can build your C extension with Python 3.12 and the binary should work on Python 3.11 unless you use features specific to Python 3.12. Moreover, the binary (.abi3.so shared library) should work on Python 3.12 and newer.

Yes, the problem is the code which is using Py_INCREF and is compiled based on the Python < 3.12 for stable ABI can not be loaded in Python 3.13.

I'm not sure we should fix this or not.

vstinner · 2024-07-13T18:16:46Z

PEP 683 promises to not break the backward compatibility for the stable ABI, but the implementation doesn't seem to be fully compatible.

The problem is that _Py_IsImmortal() checks exactly if ref count is equal to _Py_IMMORTAL_REFCNT (refcnt == _Py_IMMORTAL_REFCNT).

This problem remains me the commit 5f6ac2d of the PR gh-112174.

Zheaoli · 2024-07-13T18:22:02Z

The problem is that _Py_IsImmortal() checks exactly if ref count is equal to _Py_IMMORTAL_REFCNT (refcnt == _Py_IMMORTAL_REFCNT).

Yes, I have mentioned this at here #121528 (comment)

But I think here's another core problem we change the way that we treat the object which is not in the heap and is not immortal object(From just check it to refuse the object by crashing process. I'm not sure this is a correct way.

Zheaoli · 2024-07-13T18:26:17Z

This problem remains me the commit 5f6ac2d of the PR gh-112174.

Sorry, I'm a little confused about the commit you mentioned because I'm not familiar with free threading. Would you mind give more detail here?

vstinner · 2024-07-13T18:31:06Z

Another interesting change: commit 0bb0d88 of issue gh-109496 (not directly related).

vstinner · 2024-07-13T18:46:48Z

Other interesting changes:

commit bd1e950 of issue Stable ABI doesn't work with PySide (Shiboken) on Python 3.12 #118997 <= the issue is was looking for!
commit 6199fe3 of issue 3.12.0b2: "Assertion failed: immortal object has less refcnt than expected _Py_IMMORTAL_REFCNT" #105587

vstinner · 2024-07-13T19:04:47Z

@mgorny:

$ python3.13 -c 'import foo; foo.bar()'
python3.13: ./Include/internal/pycore_object.h:284: _PyObject_Init: Assertion `_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE) || _Py_IsImmortal(typeobj)' failed.
Aborted (core dumped)

How did you install this python3.13? It seems to be a debug build of Python. Did you build it manually?

Zheaoli · 2024-07-13T19:10:04Z

How did you install this python3.13? It seems to be a debug build of Python. Did you build it manually?

it's not related debug build I guess. I can reproduce it in non-debug build Python 3.13(install by pyenv)

vstinner · 2024-07-13T19:15:10Z

it's not related debug build I guess. I can reproduce it in non-debug build Python 3.13(install by pyenv)

To check if you're using a debug build, you can check:

$ python3 -c 'import sys; print("debug build?", hasattr(sys, "gettotalrefcount"))'
debug build? False

I cannot reproduce the issue with a release build.

vstinner · 2024-07-13T19:16:25Z

The following code is a reference leak, you don't have to INCREF the TypeError exception:

	Py_INCREF(PyExc_TypeError);
	PyErr_SetString(PyExc_TypeError, "foo");
	return NULL;

You should just:

	PyErr_SetString(PyExc_TypeError, "foo");
	return NULL;

Zheaoli · 2024-07-13T19:21:16Z

it's not related debug build I guess. I can reproduce it in non-debug build Python 3.13(install by pyenv)

To check if you're using a debug build, you can check:
$ python3 -c 'import sys; print("debug build?", hasattr(sys, "gettotalrefcount"))'
debug build? False
I cannot reproduce the issue with a release build.

Weird, I can't reproduce this in non-debug build now. Sorry for the previous reply, maybe I mix up something

mgorny · 2024-07-13T19:24:15Z

We're building --with-assertions on Gentoo, as noted in the first comment of this thread.

The problem right now is that all current versions of PyO3 produce "broken" extensions, and it will take some time before packages update to the very newest version that fixes this, and every single package published on PyPI before that happens will remain broken.

Add _Py_IsImmortalLoose() function for assertions.

vstinner · 2024-07-13T19:37:59Z

I wrote PR gh-121725 to fix the assertion. I'm not sure if it's correct. It's an heuristic.

Add _Py_IsImmortalLoose() function for assertions.

…nGH-121725) Add _Py_IsImmortalLoose() function for assertions. (cherry picked from commit b826e45) Co-authored-by: Victor Stinner <[email protected]>

vstinner · 2024-07-17T19:51:43Z

Fixed by change b826e45 in the main branch. A backport will follow in the 3.13 branch.

Thanks for the bug report @mgorny. Thanks everyone who helped to analyze the root cause.

…21725) (#121936) gh-121528: Fix _PyObject_Init() assertion for stable ABI (GH-121725) Add _Py_IsImmortalLoose() function for assertions. (cherry picked from commit b826e45) Co-authored-by: Victor Stinner <[email protected]>

…ions. (python#121725)

mgorny added the type-crash A hard crash of the interpreter, possibly with a core dump label Jul 9, 2024

mgorny mentioned this issue Jul 9, 2024

[py3.13] rustworkx: python: ./Include/internal/pycore_object.h:268: _PyObject_Init: Assertion '_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE) || _Py_IsImmortal(typeobj)' failed. PyO3/pyo3#4311

Closed

Zheaoli mentioned this issue Jul 9, 2024

gh-121528: Fix the process crash when load extension when it's compiled with low version of the python. #121550

Closed

vstinner changed the title ~~[3.13] _PyObject_Init() assertion fails on a debug build with the stable ABI (extension built with Python 3.11)~~ [3.13] _PyObject_Init() assertion fails with the stable ABI on Python built with assertions (extension built with Python 3.11) Jul 13, 2024

vstinner added a commit to vstinner/cpython that referenced this issue Jul 13, 2024

pythongh-121528: Fix _PyObject_Init() assertion for stable ABI

734bc83

vstinner added a commit to vstinner/cpython that referenced this issue Jul 13, 2024

pythongh-121528: Fix _PyObject_Init() assertion for stable ABI

2918684

Add _Py_IsImmortalLoose() function for assertions.

bedevere-app bot mentioned this issue Jul 13, 2024

gh-121528: Fix _PyObject_Init() assertion for stable ABI #121725

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Jul 13, 2024

pythongh-121528: Fix _PyObject_Init() assertion for stable ABI

e09d70f

Add _Py_IsImmortalLoose() function for assertions.

vstinner added a commit that referenced this issue Jul 17, 2024

gh-121528: Fix _PyObject_Init() assertion for stable ABI (#121725)

b826e45

Add _Py_IsImmortalLoose() function for assertions.

bedevere-app bot mentioned this issue Jul 17, 2024

[3.13] gh-121528: Fix _PyObject_Init() assertion for stable ABI (GH-121725) #121936

Merged

vstinner closed this as completed Jul 17, 2024

encukou mentioned this issue Aug 30, 2024

Stable ABI: Objects/bytesobject.c:122: PyBytes_FromStringAndSize: Assertion '_Py_IsImmortal(op)' failed. #123091

Closed

encukou pushed a commit to encukou/cpython that referenced this issue Oct 3, 2024

[3.12] pythongh-121528: Add _Py_IsImmortalLoose() function for assert…

3af7834

…ions. (python#121725)

Uh oh!

[3.13] _PyObject_Init() assertion fails with the stable ABI on Python built with assertions (extension built with Python 3.11) #121528

[3.13] _PyObject_Init() assertion fails with the stable ABI on Python built with assertions (extension built with Python 3.11) #121528

Comments

mgorny commented Jul 9, 2024 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Zheaoli commented Jul 9, 2024

Uh oh!

Zheaoli commented Jul 10, 2024

Uh oh!

Zheaoli commented Jul 10, 2024

Uh oh!

davidhewitt commented Jul 10, 2024

Uh oh!

Zheaoli commented Jul 10, 2024

Uh oh!

Zheaoli commented Jul 13, 2024

Uh oh!

alex commented Jul 13, 2024

Uh oh!

vstinner commented Jul 13, 2024

Uh oh!

alex commented Jul 13, 2024

Uh oh!

vstinner commented Jul 13, 2024

Uh oh!

Zheaoli commented Jul 13, 2024

Uh oh!

vstinner commented Jul 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Zheaoli commented Jul 13, 2024

Uh oh!

Zheaoli commented Jul 13, 2024

Uh oh!

vstinner commented Jul 13, 2024

Uh oh!

vstinner commented Jul 13, 2024

Uh oh!

vstinner commented Jul 13, 2024

Uh oh!

Zheaoli commented Jul 13, 2024

Uh oh!

vstinner commented Jul 13, 2024

Uh oh!

vstinner commented Jul 13, 2024

Uh oh!

Zheaoli commented Jul 13, 2024

Uh oh!

mgorny commented Jul 13, 2024

Uh oh!

vstinner commented Jul 13, 2024

Uh oh!

vstinner commented Jul 17, 2024

Uh oh!

mgorny commented Jul 9, 2024 •

edited by bedevere-app bot

Loading

vstinner commented Jul 13, 2024 •

edited

Loading