Best practices for mixed python/C++ APIs?

[bethebunny]

I frequently find myself wanting to implement types in C++ that then have extended functionality in Python for richer pythonic interfaces. I haven't found a good way to implement this in pybind or nanobind.

• The nanobind docs recommend subclassing the C++ type in Python as a "rich wrapper". This has some annoying issues:
  • If you return the C++ type from some function in C++, you'll get the parent type and not the python subclass
  • In general you still have 2 types floating around and occasionally need to reason about which one you have
  • It's not even easy to manually "downcast" to the Python type from the C++ type; the only way I'm aware of is assigning directly to __class__ which is IMO extremely bad practice
    • This prevents common patterns like static factory methods
• Have a Python type which internally holds an instance of the C++ type
  • This still has the complexity of having 2 different types floating around
  • You need to be careful to wrap any returned C++ calls with their wrapping Python type
  • Any functionality on the C++ type needs to be explicitly forwarded through the Python type
• Design your abstractions around the Python <-> C++ interop boundary
  • Every type you have is either all-C++ or all-Python
  • This is great, if your design happens to fit this criteria already
  • It's really annoying when your abstractions to cleanly fit around the boundary; you either need to invent concepts which exist entirely for technical reasons, or eg. turn what should be methods into free functions
• Define mixin types in Python and subclass them from C++
  • This resolves most of the above issues
  • There's a tight coupling between the Python mixin type and the C++ type -- ie you couldn't move or rename your Python type without updating the C++ code that imports it
  • From the perspective of Python this isn't an explicit relationship. When you look at the mixin class, it's not clear that it's going to be mixed in to your C++ type, or vice versa.
• Define types in C++ and monkey-patch Python functions on to them
  • For instance, nb::class_<Foo>(...); then in Python, def _foo_some_complex_fn(self, ...): ...; Foo.some_complex_fn = _foo_some_complex_fn
  • This allows the python type and C++ type to always be unified, ie. you'll never have to reason about which one you have
  • It allows defining functions on the type either in C++ or in Python
  • Patching functions onto types is still generally bad practice (though IMO not nearly as bad as assigning to __class__)

Are there any other patterns folks have come up with? Does anything fit more naturally with Nanobind? Is there an easier way to "downcast" to your Python wrapper type that I'm not missing? Could Nanobind support these patterns better?

[wjakob]

Not sure if I can contribute much here. I don't think there is anything particularly objectionable about the monkey-patch approach you proposed. Can you give an example of the mixin strategy, I can't imagine what this looks like in practice.

[bethebunny]

Roughly something like

# mixins.py
class FooMixin:
  def foo(self): ...

// module.cpp
...
auto foo_mixin = nb::module_::import_("mixins").attr("FooMixin");
nb::class_<Foo>(m, "Foo", foo_mixin);
...

It's similar to the monkey patch approach, but trades off being less explicit for allowing you to write the methods in a normal class namespace, and dispatching through normal python inheritance.

[wjakob]

To be honest, I'm surprised that actually works :D. I don't think we test such mix-in inheritance relationships from a Python type object anywhere in the test suite.