editable install cannot find pybind built .so

[HernandoR]

error

after pip install -e .
import odometry_py gives
ModuleNotFoundError: No module named 'odometry_py'

expected behaviour

import odometry_py as opy

details

I have such a project that uses pybind11 build cpp codes into a .so file then been called from a python script as follows:

try:
    sys.path.append(os.path.split(Path(__file__).resolve())[0] + "/../build")
    import odometry_py as opy

i'm recently migrating the project to vcpkg and uv managed environment. therefore with following pyproject.toml

[build-system]
requires = ["scikit-build-core>=0.10", "pybind11", "jinja2"]
build-backend = "scikit_build_core.build"

[project]
name = "odometry_py"
version = "0.0.1"
description = "A minimal example package (with pybind11)"
readme = "README.md"
requires-python = ">=3.9, <3.11"
classifiers = [
    "Development Status :: 4 - Beta",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3 :: Only",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
]
dependencies = [
    "scikit-learn>=1.6.1",
    "scipy>=1.13.1",
]

[tool.scikit-build]
wheel.expand-macos-universal-tags = true
minimum-version = "build-system.requires"
build-dir = "build/{wheel_tag}"
[tool.scikit-build.cmake]
build-type = "Release"
args = ["-GNinja"]
[tool.scikit-build.cmake.define]
CMAKE_CXX_STANDARD = "17"
CMAKE_CXX_FLAGS = "-Wno-deprecated-declarations"
CMAKE_EXPORT_COMPILE_COMMANDS = "ON"
CMAKE_TOOLCHAIN_FILE = "/home/lz/vcpkg/scripts/buildsystems/vcpkg.cmake"
# VCPKG_HOST_TRIPLET = "x64-linux"
# VCPKG_TARGET_TRIPLET = "x64-linux-static"

both uv build and cmake build can build the lib:

build/cp310-cp310-linux_x86_64/odometry_py.cpython-310-x86_64-linux-gnu.so
build/odometry_py.cpython-310-x86_64-linux-gnu.so

uv pip install -e . also build the file and crate corresponding dir in venv

.venv/lib/python3.10/site-packages/_odometry_py_editable.pth
.venv/lib/python3.10/site-packages/_odometry_py_editable.py
.venv/lib/python3.10/site-packages/odometry_py-0.0.1.dist-info/

but in python, import odometry_py as opy throws ModuleNotFoundError: No module named 'odometry_py'

request

if anyone can have a look on what did i missed?

[LecrisUT]

One concern is that the module does not belong to any package. I don't think that path is covered by the tests (still WIP to get all cases covered #906). Just so that we can narrow the issue, try to:

• Create a src/odometry_py/__init__.py (could be something else, just minimizing the changes needed in the pyproject.toml)
• Add to your pyproject.toml

  [tool.scikit-build]
  wheel.install-dir = "odometry_py"

• Rename your python module in the CMakeLists.txt to anything else (common pattern I've seen is to just make it private like _odometry_py, but anything will be fin), and then try to import odometry_py._odometry_py

This would narrow down the issue if this is an issue with top-level modules.

[HernandoR]

One concern is that the module does not belong to any package. I don't think that path is covered by the tests (still WIP to get all cases covered #906). Just so that we can narrow the issue, try to:

• Create a src/odometry_py/__init__.py (could be something else, just minimizing the changes needed in the pyproject.toml)
• Add to your pyproject.toml
  [tool.scikit-build]
  wheel.install-dir = "odometry_py"
• Rename your python module in the CMakeLists.txt to anything else (common pattern I've seen is to just make it private like _odometry_py, but anything will be fin), and then try to import odometry_py._odometry_py

This would narrow down the issue if this is an issue with top-level modules.

I'm afraid it was not working as follows:

Is it because of my project structure? How would it supposed to be?

[LecrisUT]

Can you share your project or at least your CMakeLists.txt, it seems the install part is not configured properly. I also noticed that you might have had a name clash between the python module and compiled one so maybe the python sources is also necessary

[HernandoR]

here is the CMakeList.txt, I'm not sure what i should share of python sources.

cmake_minimum_required(VERSION 3.15)
project(lidar_odometry)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse3 -pthread -fext-numeric-literals -fPIC -fopenmp")
set(CMAKE_BUILD_TYPE "Release")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS} -O3")

set(THIRDPARTY ${CMAKE_SOURCE_DIR}/thirdparty/x86_64)

# find_package(PCL REQUIRED) # pcl is not using inner libs
find_package(PCL CONFIG REQUIRED)
# add pcl_io_ply to PCL_LIBRARIES
list(APPEND PCL_LIBRARIES pcl_io_ply)

list(APPEND CMAKE_MODULE_PATH ${CMAKE_SOURCE_DIR}/cmake)
find_package(GLOG REQUIRED)
find_package(CERES REQUIRED)
find_package(GFLAGS REQUIRED)
find_package(GTEST REQUIRED)
find_package(FLANN REQUIRED)
find_package(GFLAGS REQUIRED)
find_package(NLOHMANNJSON REQUIRED)
find_package(EIGEN REQUIRED)

# set(CMAKE_CXX_FLAGS "-Wall -fopenmp")
find_package(OpenMP REQUIRED)
if(OpenMP_FOUND)
    message(STATUS "########## Found openmp ##########") 
    set(CMAKE_C_FLAGS ${CMAKE_C_FLAGS} ${OPENMP_C_FLAGS})
    set(CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS} ${OPENMP_CXX_FLAGS})
else()
    message(FATAL_ERROR "Openmp not found!")
endif()
add_definitions(-DUSE_OPENMP)

set(CMAKE_EXPORT_COMPILE_COMMANDS ON CACHE INTERNAL "") # works

option(BUILD_WITH_GUI "Build gui viewer for lio" OFF)
if(BUILD_WITH_GUI)
    add_definitions(-D WITH_LIO_GUI)
    find_package(Pangolin REQUIRED)
endif()

include_directories(src/)

file(GLOB_RECURSE ALL_LIBRARY_HDRS "include/*.h")
file(GLOB_RECURSE ALL_LIBRARY_SRCS "src/*.cpp" "src/*.hpp" "src/*.h")

set(LIO_CORE_LIB lio_core_lib)
add_library(${LIO_CORE_LIB} SHARED ${ALL_LIBRARY_SRCS})

find_package(OpenCV REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS})
message("OpenCV_INCLUDE_DIRS:" ${OpenCV_INCLUDE_DIRS})
message("OpenCV_LIBS:" ${OpenCV_LIBS})
message("opencv version:" ${OpenCV_VERSION})


# # add link directories


target_include_directories(${LIO_CORE_LIB}
    PUBLIC
    ${CMAKE_CURRENT_SOURCE_DIR}/include
    ${CMAKE_CURRENT_SOURCE_DIR}
    ${GLOG_INCLUDE_DIR}
    ${CERES_INCLUDE_DIR}
    ${FLANN_INCLUDE_DIR}
    ${EIGEN_INCLUDE_DIR}
    ${GTEST_INCLUDE_DIR}
    ${GFLAGS_INCLUDE_DIR}
    ${NLOHMANNJSON_INCLUDE_DIR}
    ${PCL_INCLUDE_DIRS})

option(GCC_LOW_VERSION "gcc version" OFF)
if(GCC_LOW_VERSION)
    set(DEPENDS_LIBS     
        ${GLOG_LIBRARIES}
        ${CERES_LIBRARIES}
        ${FLANN_LIBRARIES}
        ${PCL_LIBRARIES}
        ${GFLAGS_LIBRARIES}
        ${GTEST_LIBRARIES}
        ${OpenCV_LIBS})
else()
    set(DEPENDS_LIBS     
        ${GLOG_LIBRARIES}
        ${CERES_LIBRARIES}
        ${FLANN_LIBRARIES}
        ${PCL_LIBRARIES}
        ${GFLAGS_LIBRARIES}
        ${GTEST_LIBRARIES}
        ${OpenCV_LIBS}
        -lstdc++fs)
    add_definitions(-Dexpfs)
endif()

if(BUILD_WITH_GUI)
    list(APPEND DEPENDS_LIBS ${Pangolin_LIBRARIES})
endif()

target_link_libraries(${LIO_CORE_LIB} PUBLIC ${DEPENDS_LIBS})

add_subdirectory(apps)



option(BUILD_WITH_PYTHON_WRAPPER "Build python wrapper for lio" ON)
if (BUILD_WITH_PYTHON_WRAPPER)
    find_package(Python3 3.10 EXACT REQUIRED COMPONENTS Interpreter Development)
    find_package(pybind11 CONFIG)
    message("python version:" ${Python3_VERSION})
    message("python stdlib:" ${Python3_STDLIB})
    pybind11_add_module(_odometry_py python_bind/odometry_py_bind.cpp)
    target_link_libraries(_odometry_py PRIVATE pybind11::module ${LIO_CORE_LIB} lio_io liotool ${DEPENDS_LIBS})
endif()

[LecrisUT]

Not sure if you hit a limit, but it cut off at the relevant part that would have had install instructions. Python sources wise the output of tree should do.

[HernandoR]

Not sure if you hit a limit, but it cut off at the relevant part that would have had install instructions. Python sources wise the output of tree should do.

i don't think there is a install instruction in my CMakeList.txt, is it a must?

.
├── apps
├── .clang-format
├── cmake
├── CMakeLists.txt
├── .git
├── .gitignore
├── .gitlab-ci.yml
├── include
├── python_bind
├── README.md
├── run_mapping.sh
├── scripts
├── src
├── thirdparty
├── vcpkg.json
└── .venv

there's no python project strucure in the project, but only .py scripts in the pthon_bind, which act as an entry script:

./python_bind
├── odometry_py_bind.cpp
├── batch_run_lidar_odo_from_clip.py
├── run_lidar_odo_from_clip_postpatch.py
└── test_bind.py

[LecrisUT]

i don't think there is a install instruction in my CMakeList.txt, is it a must?

Yes, it must have one. But that would also break in non-editable pip install. Does normal pip install work? If not it could be a simple "fix" to your initial problem (will still have other design issues later on).

there's no python project strucure in the project, but only .py scripts in the pthon_bind, which act as an entry script:

Entry scripts for the user? If so you still have much to configure. Those files are also the one that require the dependencies in your pyproject.toml?

[HernandoR]

Though I have moved forward from the original project, my latter experience could almost conclude that the install instruction in the cmake was the issue to my original issue.

Yes, these scripts must run in a venv which is specified by the dependencies in my pyproject.toml, you can imagine they are pre-written examples for the potential users to use tha cpp lib.
and.. Yes, the project's structure has tons of optimizations that should be done (yet I don't think the owner will ever do)