Support for tp_free and/or custom delete

[Walter-Reactor]

I'm aware that generally speaking nanobind does not support c++ types which overload operators new/delete, however I'm working on a project where, at the very least, I need to support custom deleters. I'd prefer not to give up on nanobind and resort to pybind11, so I've experimented with overriding the tp_dealloc slot. Unfortunately the inst_dealloc doesn't have a header declaration, so I've had to copy/paste the definition. In doing so, I noticed that the implementation never appears to call tp_free, even though there are some ifdefs that store that function pointer. Am I perhaps missing something?
Solutions that could work for me that I can think of:

1. Just live with a copy pasted & slightly tweaked inst_dealloc which will call tp_free instead of an untyped delete, and put my custom deleter in the tp_free slot
2. modify nanobind to expose inst_dealloc in headers so I can call write a shim tp_dealloc which will call tp_free and manually clear the cpp_delete flag, then call the default tp_dealloc
3. On module init create a dummy class and read out the tp_dealloc slot, and store that address to a global static which I then reference in my shim tp_dealloc.
4. Submit a patch adding support for calling the tp_free slot to nanobind, maybe becoming an optional feature allowing declaring a custom free function in the class definition via .def(nb::delete<T>())?

Would the nanobind maintainers be open to a patch adding that kind of support?

For background, I'm working on a project exposing an API written in Rust to multiple languages via a bunch of #[repr(C)] structs and pointers, and since I already have to write codegen for C++, I've been using nanobind to create the python bindings. The only hitch is that almost everything is allocated by the rust heap, and must be freed by it.

[Walter-Reactor]

For reference, the code below is enough to do what I need, but it's a bit ugly & I'd like to either improve it or upstream a nicer way.

static void (*nb_tp_dealloc)(void *) = nullptr;

void diplomat_tp_dealloc(PyObject *self)
{
	using namespace nb::detail;
	printf("diplomat_tp_dealloc %d\n", __LINE__);
	PyTypeObject *tp = Py_TYPE(self);
	nb_inst *inst = (nb_inst *)self;
	if (inst->cpp_delete)
	{
		inst->cpp_delete = false;
		printf("diplomat_tp_dealloc %d\n", __LINE__);
		auto tp_free = (freefunc)(PyType_GetSlot(tp, Py_tp_free));
		(*tp_free)(self);
	}
	(*nb_tp_dealloc)(self);
}

struct Dummy
{
};

NB_MODULE(somelib, somelib_mod)
{
	{
		nb::class_<Dummy> dummy(somelib_mod, "__dummy__");
		nb_tp_dealloc = (void (*)(void *))nb::type_get_slot(dummy, Py_tp_dealloc);
	}
	...
	PyType_Slot slots[] = {
		 {Py_tp_free, (void *)MyType::operator delete },
		 {Py_tp_dealloc, (void *)diplomat_tp_dealloc},
		 {0, nullptr}};
	nb::class_<MyType>(somelib_mod, "MyType", nb::type_slots(slots))
}

[wjakob]

Nanobind doesn't use tp_free, but rather frees objects using the following lines in nb_type.cpp:

    if (NB_UNLIKELY(gc))
        PyObject_GC_Del(self);
    else
        PyObject_Free(self);

It uses those operations because those same objects were also allocated in a custom way:

    nb_inst *self;
    if (NB_LIKELY(!gc)) {
        self = (nb_inst *) PyObject_Malloc(sizeof(nb_inst));
        if (!self)
            return PyErr_NoMemory();
        PyObject_Init((PyObject *) self, tp);
    } else {
        self = (nb_inst *) PyType_GenericAlloc(tp, 0);
        if (!self)
            return nullptr;
    }

Of your various options, I would be open to approach 2, since it does not impose additional runtime costs.

[wjakob]

I would prefer if this code path in nanobind doesn't have to be changed.

[wjakob]

PyType_GetSlot is an opaque call across a shared library and cannot be inlined, so this does add a tiny bit of extra runtime cost to support what I would call a niche use case. My general policy in this project is to avoid such changes.

[Walter-Reactor]

Gotcha. Thanks for the replies! If I manage to figure out a way to support custom deleters for unique_ptr/shared_ptr that was lower overhead, would you be open to a PR? The way I'm using GetSlot is admittedly a bit of a hack, and I suspect there's better ways to deal with this.

[wjakob]

I can't promise ahead of time-- it really depends on the details. Best to make a very draft PR before polishing your code.

[Walter-Reactor]

as long as it's not a no-in-advance-by-design! I'll see what I can come up with.