Skip to content

Commit 92bf684

Browse files
wjakobwjakob
committed
packaging documentation
1 parent 9a3eff6 commit 92bf684

File tree

1 file changed

+194
-124
lines changed

1 file changed

+194
-124
lines changed

‎docs/packaging.rst

+194-124
Original file line numberDiff line numberDiff line change
@@ -18,113 +18,143 @@ Note that all of the recommended practices have already been implemented in the
1818
which is a minimal C++ project with nanobind-based bindings. You may therefore
1919
prefer to clone this repository and modify its contents.
2020

21-
Step 1: Specify build dependencies
22-
----------------------------------
21+
Step 1: Overview
22+
----------------
2323

24-
In the root directory of your project, create a file named ``pyproject.toml``
25-
listing build-time dependencies. Runtime dependencies don't need to be added
26-
here. The following core dependencies are required by nanobind:
24+
The example project has a simple directory structure:
25+
26+
.. code-block:: text
27+
28+
├── README.md
29+
├── CMakeLists.txt
30+
├── pyproject.toml
31+
└── src/
32+
├── my_ext.cpp
33+
└── my_ext/
34+
└── __init__.py
35+
36+
The file ``CMakeLists.txt`` contains the C++-specifics part of the build
37+
system, while ``pyproject.toml`` explains how to turn the example into a wheel.
38+
The file ``README.md`` should ideally explain how to use the project in more
39+
detail. Its contents are arbitrary, but the file must exist for the following
40+
build system to work.
41+
42+
All source code is located in a ``src`` directory containing the Python package
43+
as a subdirectory.
44+
45+
Compilation will turn ``my_ext.cpp`` into a shared library in the package
46+
directory, which has an underscored platform-dependent name (e.g.,
47+
``_my_ext_impl.cpython-311-darwin.so``) to indicate that it is an
48+
implementation detail. The ``src/my_ext/__init__.py`` imports the extension and
49+
exposes relevant functionality. In this small example project, it only contains
50+
a single line:
51+
52+
.. code-block:: python
53+
54+
from ._my_ext_impl import hello
55+
56+
The file ``src/my_ext.cpp`` contains minimal bindings for an example function:
57+
58+
.. code-block:: cpp
59+
60+
#include <nanobind/nanobind.h>
61+
62+
NB_MODULE(_my_ext_impl, m) {
63+
m.def("hello", []() { return "Hello world!"; });
64+
}
65+
66+
The next two steps will set up the infrastructure needed for wheel generation.
67+
68+
Step 2: Specify build dependencies and metadata
69+
-----------------------------------------------
70+
71+
In the root directory of the project, create a file named ``pyproject.toml``
72+
listing *build-time dependencies*. Note that runtime dependencies *do not* need
73+
to be added here. The following core dependencies are required by nanobind:
2774

2875
.. code-block:: toml
2976
3077
[build-system]
31-
requires = [
32-
"setuptools>=42",
33-
"wheel",
34-
"scikit-build>=0.16.7",
35-
"cmake>=3.18",
36-
"nanobind>=1.1.0",
37-
"ninja; platform_system!='Windows'"
38-
]
78+
requires = ["scikit-build-core >=0.4.3", "nanobind >=1.3.1"]
79+
build-backend = "scikit_build_core.build"
3980
40-
build-backend = "setuptools.build_meta"
81+
You may need to increase the minimum nanobind version in the above snippet if
82+
you are using features from versions newer than 1.3.1.
4183

42-
Step 2: Create a ``setup.py`` file
43-
----------------------------------
84+
Just below the list of build-time requirements, specify project metadata including:
4485

45-
Most wheels are built using the `setuptools
46-
<https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/>`__
47-
package. We also use it here in combination with `scikit-build
48-
<https://scikit-build.readthedocs.io/en/latest>`__, which can be thought of as
49-
a glue layer connecting setuptools with CMake.
86+
- The project's name (which must be a valid package name)
87+
- The version number
88+
- A brief (1-line) description of the project
89+
- The name of a more detailed README file
90+
- The list of authors with email addresses.
91+
- The software license
92+
- The project web page
93+
- Runtime dependencies, if applicable
5094

51-
To do so, create the file ``setup.py`` at the root of your project directory.
52-
This file provides metadata about the project and also constitutes the entry
53-
point of the build system.
95+
An example is shown below:
5496

55-
.. code-block:: python
97+
.. code-block:: toml
5698
57-
import sys
58-
59-
try:
60-
from skbuild import setup
61-
import nanobind
62-
except ImportError:
63-
print("The preferred way to invoke 'setup.py' is via pip, as in 'pip "
64-
"install .'. If you wish to run the setup script directly, you must "
65-
"first install the build dependencies listed in pyproject.toml!",
66-
file=sys.stderr)
67-
raise
68-
69-
setup(
70-
name="my_ext", # <- The package name (e.g. for PyPI) goes here
71-
version="0.0.1", # <- The current version number.
72-
author="Your name",
73-
author_email="[email protected]",
74-
description="A brief description of what the package does",
75-
long_description="A long format description in Markdown format",
76-
long_description_content_type='text/markdown',
77-
url="https://github.com/username/repository_name",
78-
python_requires=">=3.8",
79-
license="BSD",
80-
packages=['my_ext'], # <- The package will install one module named 'my_ext'
81-
package_dir={'': 'src'}, # <- Root directory containing the Python package
82-
cmake_install_dir="src/my_ext", # <- CMake will place the compiled extension here
83-
include_package_data=True
84-
)
85-
86-
The warning message at the top will be explained shortly. This particular
87-
``setup.py`` file assumes the following project directory structure:
99+
[project]
100+
name = "my_ext"
101+
version = "0.0.1"
102+
description = "A brief description of what this project does"
103+
readme = "README.md"
104+
requires-python = ">=3.8"
105+
authors = [
106+
{ name = "Your Name", email = "[email protected]" },
107+
]
108+
classifiers = [
109+
"License :: BSD",
110+
]
111+
# Optional: runtime dependency specification
112+
# dependencies = [ "cryptography >=41.0" ]
88113
89-
.. code-block:: text
114+
[project.urls]
115+
Homepage = "https://github.com/your/project"
90116
91-
├── CMakeLists.txt
92-
├── pyproject.toml
93-
├── setup.py
94-
└── src/
95-
├── my_ext.cpp
96-
└── my_ext/
97-
└── __init__.py
117+
We will use `scikit-build-core
118+
<https://github.com/scikit-build/scikit-build-core>`__ to build wheels, and
119+
this tool also has its own configuration block in ``pyproject.toml``. The
120+
following defaults are recommended:
98121

99-
In other words, the source code is located in a ``src`` directory containing
100-
the Python package as a subdirectory. Naturally, the code can also be
101-
arranged differently, but this will require corresponding modifications in
102-
``setup.py``.
122+
.. code-block:: toml
123+
124+
[tool.scikit-build]
125+
# Protect the configuration against future changes in scikit-build-core
126+
minimum-version = "0.4"
127+
128+
# Setuptools-style build caching in a local directory
129+
build-dir = "build/{wheel_tag}"
103130
104-
In practice, it can be convenient to add further code that extracts relevant
105-
fields like version number, short/long description, etc. from code or the
106-
project's README file to avoid duplication.
131+
# Build stable ABI wheels for CPython 3.12+
132+
wheel.py-api = "cp312"
107133
108-
Step 3: Create a CMake build system
134+
Step 3: Set up a CMake build system
109135
-----------------------------------
110136

111137
Next, we will set up a suitable ``CMakeLists.txt`` file in the root directory.
112-
Since this build system is designed to be invoked through ``setup.py`` and
113-
``scikit-build``, it does not make sense to perform a standalone CMake build.
114-
The message at the top warns users attempting to do this.
138+
Since this build system is designed to be invoked through
139+
``scikit-build-core``, it does not make sense to perform a standalone CMake
140+
build. The message at the top warns users attempting to do this.
115141

116142
.. code-block:: cmake
117143
118-
cmake_minimum_required(VERSION 3.18...3.22)
119-
project(my_ext)
144+
# Set the minimum CMake version and policies for highest tested version
145+
cmake_minimum_required(VERSION 3.15...3.26)
120146
147+
# Set up the project and ensure there is a working C++ compiler
148+
project(my_ext LANGUAGES CXX)
149+
150+
# Warn if the user invokes CMake directly
121151
if (NOT SKBUILD)
122152
message(WARNING "\
123-
This CMake file is meant to be executed using 'scikit-build'. Running
124-
it directly will almost certainly not produce the desired result. If
125-
you are a user trying to install this package, please use the command
126-
below, which will install all necessary build dependencies, compile
127-
the package in an isolated environment, and then install it.
153+
This CMake file is meant to be executed using 'scikit-build-core'.
154+
Running it directly will almost certainly not produce the desired
155+
result. If you are a user trying to install this package, use the
156+
command below, which will install all necessary build dependencies,
157+
compile the package in an isolated environment, and then install it.
128158
=====================================================================
129159
$ pip install .
130160
=====================================================================
@@ -133,64 +163,54 @@ The message at the top warns users attempting to do this.
133163
in your environment once and use the following command that avoids
134164
a costly creation of a new virtual environment at every compilation:
135165
=====================================================================
136-
$ python setup.py install
137-
=====================================================================")
166+
$ pip install nanobind scikit-build-core[pyproject]
167+
$ pip install --no-build-isolation -ve .
168+
=====================================================================
169+
You may optionally add -Ceditable.rebuild=true to auto-rebuild when
170+
the package is imported. Otherwise, you need to rerun the above
171+
after editing C++ files.")
138172
endif()
139173
140-
# Perform a release build by default
141-
if (NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
142-
set(CMAKE_BUILD_TYPE Release CACHE STRING "Choose the type of build." FORCE)
143-
set_property(CACHE CMAKE_BUILD_TYPE PROPERTY STRINGS "Debug" "Release" "MinSizeRel" "RelWithDebInfo")
144-
endif()
174+
Next, import Python and nanobind including the ``Development.SABIModule``
175+
component that can be used to create `stable ABI
176+
<https://docs.python.org/3/c-api/stable.html>`__ builds.
145177

146-
# Create CMake targets for Python components needed by nanobind
147-
find_package(Python 3.8 COMPONENTS Interpreter Development.Module REQUIRED)
178+
.. code-block:: cmake
148179
149-
# Determine the nanobind CMake include path and register it
150-
execute_process(
151-
COMMAND "${Python_EXECUTABLE}" -m nanobind --cmake_dir
152-
OUTPUT_STRIP_TRAILING_WHITESPACE OUTPUT_VARIABLE NB_DIR)
153-
list(APPEND CMAKE_PREFIX_PATH "${NB_DIR}")
180+
# Try to import all Python components potentially needed by nanobind
181+
find_package(Python 3.8
182+
REQUIRED COMPONENTS Interpreter Development.Module
183+
OPTIONAL_COMPONENTS Development.SABIModule)
154184
155185
# Import nanobind through CMake's find_package mechanism
156186
find_package(nanobind CONFIG REQUIRED)
157187
158-
# We are now ready to compile the actual extension module
159-
nanobind_add_module(
160-
_my_ext_impl
161-
src/my_ext.cpp
162-
)
163-
164-
# Install directive for scikit-build
165-
install(TARGETS _my_ext_impl LIBRARY DESTINATION .)
188+
The last two steps build and install the actual extension
166189

167-
A simple definition of ``src/my_ext.cpp`` could contain the following:
168-
169-
.. code-block:: cpp
190+
.. code-block:: cmake
170191
171-
#include <nanobind/nanobind.h>
192+
# We are now ready to compile the actual extension module
193+
nanobind_add_module(
194+
# Name of the extension
195+
_my_ext_impl
172196
173-
NB_MODULE(_my_ext_impl, m) {
174-
m.def("hello", []() { return "Hello world!"; });
175-
}
197+
# Target the stable ABI for Python 3.12+, which reduces
198+
# the number of binary wheels that must be built. This
199+
# does nothing on older Python versions
200+
STABLE_ABI
176201
177-
Compilation and installation will turn this binding code into a shared library
178-
located in the ``src/my_ext`` directory with an undescored platform-dependent
179-
name (e.g., ``_my_ext_impl.cpython-311-darwin.so``) indicating that the
180-
extension is an implementation detail. The ``__init__.py`` file in the same
181-
directory has the purpose of importing the extension and exposing relevant
182-
functionality, e.g.:
202+
# Source code goes here
203+
src/my_ext.cpp
204+
)
183205
184-
.. code-block:: python
206+
# Install directive for scikit-build-core
207+
install(TARGETS _my_ext_impl LIBRARY DESTINATION my_ext)
185208
186-
from ._my_ext_impl import hello
187209
188210
Step 4: Install the package locally
189211
-----------------------------------
190212

191-
It used to be common to run ``setup.py`` files directly (as in ``python
192-
setup.py install``), but this is fragile when the environment doesn't have the
193-
exact right versions of all build dependencies. The recommended method is via
213+
To install the package, run
194214

195215
.. code-block:: bash
196216
@@ -214,7 +234,38 @@ instead of installing the package.
214234
215235
$ pip wheel .
216236
217-
Step 5: Build wheels in the cloud
237+
Step 5: Incremental rebuilds
238+
----------------------------
239+
240+
The ``pip install`` and ``pip wheel`` commands are extremely conservative to
241+
ensure reproducible builds. They create a pristine virtual environment and
242+
install build-time dependencies before compiling the extension *from scratch*.
243+
244+
It can be frustrating to wait for this lengthy sequence of steps after every
245+
small change to a source file during the active development phase of a project.
246+
To avoid this, first install the project's build dependencies, e.g.:
247+
248+
.. code-block:: bash
249+
250+
$ pip install nanobind scikit-build-core[pyproject]
251+
252+
Next, install the project without build isolation to enable incremental builds:
253+
254+
.. code-block:: bash
255+
256+
$ pip install --no-build-isolation -ve .
257+
258+
This command will need to be run after every change to reinstall the updated package.
259+
For an even more interactive experience, use
260+
261+
.. code-block:: bash
262+
263+
$ pip install --no-build-isolation -Ceditable.rebuild=true -ve .
264+
265+
This will automatically rebuild any code (if needed) whenever the ``my_ext``
266+
package is imported into a Python session.
267+
268+
Step 6: Build wheels in the cloud
218269
---------------------------------
219270

220271
On my machine, the previous step produced a file named
@@ -223,7 +274,7 @@ Python 3.11 running on an arm64 macOS machine. Other Python versions
223274
and operating systems will each require their own wheels, which leads
224275
to a challenging build matrix.
225276

226-
In the future (once Python 3.12 is more widespread), nanobind's Stable ABI
277+
In the future (once Python 3.12 is more widespread), nanobind's stable ABI
227278
support will help to reduce the size of this build matrix. More information
228279
about this will be added here at a later point.
229280

@@ -239,6 +290,25 @@ after every commit, which is perhaps a bit excessive. In this case, you can
239290
still trigger the action manually on the *Actions* tab of the GitHub project
240291
page.
241292

293+
Furthermore, append the following cibuildwheel-specific configuration to ``pyproject.toml``:
294+
295+
.. code-block:: toml
296+
297+
[tool.cibuildwheel]
298+
# 32-bit builds are not supported by nanobind
299+
archs = ["auto64"]
300+
301+
# Necessary to see build output from the actual compilation
302+
build-verbosity = 1
303+
304+
# Optional: run pytest to ensure that the package was correctly built
305+
# test-command = "pytest {project}/tests"
306+
# test-requires = "pytest"
307+
308+
# Needed for full C++17 support on macOS
309+
[tool.cibuildwheel.macos.environment]
310+
MACOSX_DEPLOYMENT_TARGET = "10.14"
311+
242312
Following each run, the action provides a downloadable *build artifact*, which
243313
is a ZIP file containing all the individual wheel files for each platform.
244314

0 commit comments

Comments
 (0)