How can I set up nested __init__.py files in my source folder so that my modules are actually found?

[dograt]

Pretty much the title. I've seen the nanobind example project, but it's a pretty trivial example. It's not clear to me how I should structure my __init__.py files such that relative imports actually work. For example, I have a bunch of nested submodules, such that I can do the following:

import my_example
myexample.m1.m2.function(argument)

However, the following does NOT work:

from myexample.m1.m2 import function

I get the error: ModuleNotFoundError: No module named 'myexample.m1'. So, it seems that I need to create my own __init__.py hierarchy so that these modules can get imported properly. This seems to help with the imports up until nanobind needs to actually find the function, and then it gets very unhappy with my manually specified __all__. Has anybody gotten imports working properly this way?

Edit: I will add that I can get imports sort of working if I use sys.modules, but this is a hack and I'd really rather not use it!

[wjakob]

One thing you can do is to look up modules from Python via nb::module_::import_ and directly place bindings there. That lets you create modules combining a mixture of declarations made with standard Python code, and C++ extensions.

[dograt]

Thanks for the reply! I'm not sure I am following you, though. How would I invoke nb::module_::import_ in Python?

What worked for me is that I ended up manually recreating the directory structure of the modules, and using a bunch of relative includes in each module file. For example, myexample/m1/m2.py would contain from ..myexample_ext.m1.m2 import *.

I'm not sure if this is the approach that you're describing, but it seems to work well. I would be open to another method if it is more automagical, since I don't love needing to do relative imports.

[smurfix]

How would I invoke nb::module_::import_ in Python?

That's just import. Or maybe importlib.import_module.

As for the original question, you could also take nanobind's synthetic submodules and just add them to sys.modules. Python will find them and not try to go to the file system.

Recursively auto-generating these assignments when your program first imports myexample is reasonably easy: loop through the module's attributes (skip anything starting with an underline); if the value is a module, add to sys.modules and descend into it.