Build static module with Pybind11 and link against the interpreter

[allsey87]

I am embedding Python in an application and I would like to statically link against a Pybind11 module so that it is available without having to use dlopen etc.

I think I have to add a line in Python-*/Modules/Setup under the *static* section. I believe I will be able to hack something together, but I am wondering if there is a standard way of doing this?

In #466 @wjakob mentions that:

Python extension modules have to be dynamic.

But I think that is not entirely true, unless it is a limitation of Pybind11, right?

[allsey87]

I have tried adding the following line to my Setup.local

_MODULE_pybind11 path/to/some/libMODULE.a

I set the identifier of the module like this since Python links against the symbol PyInit_MODULE and Pybind11 produces the symbol PyInit__MODULE_pybind11, so by setting the identifier to _MODULE_pybind11 the two line up.

I confirmed that the symbol requested by Python exists in the libMODULE.a (statically linked Pybind11 module) but it seems to not be getting included. I am currently trying to bring in the whole archive by manipulating the linker flags as specified in tweaking compiler linker flags. I am currently trying with the following line but it still isn't quite working:

_MODULE_pybind11 -Xlinker --whole-archive path/to/some/libMODULE.a

Using nm, I can see libpython3.11a contains:

U PyInit__MODULE_pybind11

while libMODULE.a contains:

T PyInit__MODULE_pybind11

So it seems like it is just a question of getting that symbol from A to B...

[allsey87]

I have something somewhat working (somewhat being the keyword there). It is sufficient to just have a single line in Setup.local under *static* that contains the identifier of the package:

_MODULE_pybind11

This is sufficient to cause the interpreter library to be built with the unresolved symbol U PyInit__MODULE_pybind11, this symbol can then be provided during linking the executable that embeds the interpreter. Now, the catch is that the module has to be imported by the name _MODULE_pybind11 and not MODULE and I think this may be causing other issues too since although I can import the module, I cannot see anything on it (e.g., no __version__ field, etc).

[allsey87]

Ok, I overlooked that the module I was trying to build actually contains a mixture of Python code and C++ code. Hence the issue was that I had MODULE (Python code) and _MODULE_pybind11 (C++ code). The code of the C++ module itself defined the name _MODULE_pybind11 and it is used internally by MODULE.

I am still trying to patch everything up and get stuff working, but in summary, it seems that it is pretty straight forward to statically link a Pybind11 library to the Python interpreter.

[emmenlau]

Dear @allsey87 and all, we are currently facing the same issue, and would be really happy about insights about your solution. Could you share what you ended up doing? And did you ever face this in a cross-platform environment where Windows is also targetted?