Properly setting RPATH in Python extensions

[voidtrance]

I have an application which includes a Python extension. The extension links against a shared library that is part of the application. Both the extension and shared library get installed in the site-packages directory.

I am trying to figure out how to properly set RPATH in the meson build so the extension can find the shared library. I have a small example that illustrates the issue:
rpath_test.tar.gz

The following are the steps to setup, build, and run the example.

venv setup commands:

# cd /tmp
# virtualenv py-venv
# /tmp/py-venv/bin/pip install -r <path to rpath_test>/virtualenv.txt

Build command:

# cd <path to rpath_test>
# /tmp/py-venv/bin/python -m build -w .

Wheel install command:

# /tmp/py-venv/bin/pip install ./dist/test-0.0.1-cp311-cp311-linux_x86_64.whl

Test execution command:

# /tmp/py-venv/bin/python /tmp/py-venv/bin/test_app.py

When executing from the editable build, I get:

# ./test_app.py 
This is a debug print

When executing from the virtual environment, I get:

# ~/tmp/py-venv/bin/python ~/tmp/py-venv/bin/test_app.py 
Traceback (most recent call last):
  File "/home/dharalanov/tmp/py-venv/bin/test_app.py", line 2, in <module>
    import mytest.pyext
  File "/home/dharalanov/tmp/py-venv/lib64/python3.11/site-packages/mytest/pyext/__init__.py", line 1, in <module>
    import mytest.ext
  File "/home/dharalanov/tmp/py-venv/lib64/python3.11/site-packages/mytest/ext/__init__.py", line 1, in <module>
    from mytest.ext._test import *
ImportError: libmylib.so: cannot open shared object file: No such file or directory

When looking at the extension binary, RPATH is not even set. All I can see is:

# readelf -d /tmp/py-venv/lib64/python3.11/site-packages/mytest/ext/_test.cpython-311-x86_64-linux-gnu.so 

Dynamic section at offset 0x2e28 contains 22 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libmylib.so]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../lib]
 0x000000000000000c (INIT)               0x1000
 0x000000000000000d (FINI)               0x11fc
 0x0000000000000019 (INIT_ARRAY)         0x3e10
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x3e18
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x300
 0x0000000000000005 (STRTAB)             0x430
 0x0000000000000006 (SYMTAB)             0x328
 0x000000000000000a (STRSZ)              198 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x3fe8
 0x0000000000000002 (PLTRELSZ)           144 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x678
 0x0000000000000007 (RELA)               0x4f8
 0x0000000000000008 (RELASZ)             384 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffff9 (RELACOUNT)          12
 0x0000000000000000 (NULL)               0x0

Not only is RPATH not set but RUNPATH is also incorrect.

What is the correct way to link the extension so RPATH is set correctly?

[rgommers]

@voidtrance isn't this essentially the same question as gh-662? Or is it a follow-up question for the same package (that discussion did touch on install_rpath)?

Also, it would be good to put the example in a standalone GitHub repo, that's a lot easier to look at and discuss than a tarball.

[voidtrance]

Yes, it did touch on it but the question was different. It was about installation paths, not RPATH. So, I thought it would be better to start a new discussions question specific to the issue.

[voidtrance]

Can anyone help with this? Is this a meson-python issue or pure meson issue?

[voidtrance]

meson-python does not apply the RPATH added with the Meson's install_rpath argument. This feature can be implemented. However, I'm not sure whether Meson exposes install_rpath the metadata. If it does not, this feature should be added to Meson before meson-python can implement support for install_rpath. I'll have a closer look when I have some time.

Hi, have you had a chance to look at this? My application is DOA without it as the Python extensions can't find the shared library.

Thank you.

[voidtrance]

meson-python does not apply the RPATH added with the Meson's install_rpath argument. This feature can be implemented. However, I'm not sure whether Meson exposes install_rpath the metadata. If it does not, this feature should be added to Meson before meson-python can implement support for install_rpath. I'll have a closer look when I have some time.

I just went back to the Meson documentation for py_installation.extension_module(). It states that:

All positional and keyword arguments are the same as for shared_module(), excluding name_suffix and name_prefix, and with the addition of the following:

This should mean that install_rpath should be accepted and set. Is the documentation incorrect?

[dnicolodi]

intall_path is applied when meson install is run. meson-python does not meson install to create a Python wheel.

[dnicolodi]

I've started looking into this. Meson did not export the metadata required by meson-python for applying the install_path. I've extended Meson to add the metadata mesonbuild/meson#13625 The changes required in meson-python are a bit more involved and I haven't had time to work on this yet.

As a work-around, you can set an RPATH explicitly passing linker arguments. For example:

py.extension_module(
    'example',
    'example.c',
    link_with: example_lib,
    link_args: ['-Wl,-rpath,example'],
    install: true,
)

Meson emits a warning when it detects this use of link_args but it is harmless.

[voidtrance]

I've started looking into this. Meson did not export the metadata required by meson-python for applying the install_path. I've extended Meson to add the metadata mesonbuild/meson#13625 The changes required in meson-python are a bit more involved and I haven't had time to work on this yet.

As a work-around, you can set an RPATH explicitly passing linker arguments. For example:

py.extension_module(
    'example',
    'example.c',
    link_with: example_lib,
    link_args: ['-Wl,-rpath,example'],
    install: true,
)

Meson emits a warning when it detects this use of link_args but it is harmless.

Thank you for the suggestion and for the work!