Skip to content

Commit 652bc79

Browse files
mayantaylorritvikrao
mayantaylor
and
ritvikrao
authored
Modernize build system to use pyproject.toml and build wheels (#267)
* greenlet 3.0.0 drops support for python 3.6 * draft pyproject.toml * adding templated build wheel yaml * set up build wheel action manually specifying package dir formatting typo manual cibuildwheel command? one dir down? adding debug info typo testing git installation clean up print statements typos removing 'build' dependency separating workflows adding testpypi workflow * set up testpypi upload action * don't need cmake? not used * cleanup * adding pypy builds * work around pypy dev0 checkout * adding comments * updating installation instructions * action to publish to pypi * tmp commit to get action to flush * tmp commit to get action to flush on PR * restoring upload action * adding source dist * fetch full depth * adding source dist to upload * line break * experiment without github charm token * cleanup - somehow checkout charm_src got duplicated * remove manual requirements installation instructions (handled by pip install) --------- Co-authored-by: Ritvik Rao <[email protected]>
1 parent 6df825d commit 652bc79

File tree

7 files changed

+227
-75
lines changed

7 files changed

+227
-75
lines changed

‎.github/workflows/build_wheels.yml

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
name: Build wheels
2+
3+
on: workflow_dispatch
4+
5+
jobs:
6+
build_sdist:
7+
name: Build source distribution
8+
runs-on: ubuntu-latest
9+
steps:
10+
- uses: actions/checkout@v4
11+
with:
12+
fetch-depth: 0 # Fetch full history, necessary for git describe in setup.py
13+
14+
- name: Build sdist
15+
run: pipx run build --sdist
16+
17+
- uses: actions/upload-artifact@v4
18+
with:
19+
name: cibw-sdist
20+
path: dist/*.tar.gz
21+
22+
build_wheels:
23+
name: Build wheels on ${{ matrix.os }}
24+
runs-on: ${{ matrix.os }}
25+
strategy:
26+
matrix:
27+
# macos-13 is an intel runner, macos-14 is apple silicon
28+
os: [ubuntu-latest, macos-13, macos-14]
29+
30+
steps:
31+
- uses: actions/setup-python@v5
32+
with:
33+
python-version: 3.8
34+
if: runner.os == 'macOS' && runner.arch == 'ARM64'
35+
36+
- uses: actions/checkout@v4
37+
with:
38+
fetch-depth: 0 # Fetch full history, necessary for git describe in setup.py
39+
40+
- name: Checkout charm_src # is this best practice? should we use a submodule?
41+
uses: actions/checkout@v4
42+
with:
43+
repository: charmplusplus/charm
44+
path: charm_src/charm
45+
fetch-depth: 0 # Fetch full history, necessary for commit_str parsing in setup.py
46+
47+
- name: Build wheels
48+
env:
49+
CIBW_SKIP: "*-manylinux_i686 *-manylinux_ppc64le *-manylinux_s390x *musllinux_*"
50+
uses: pypa/[email protected]
51+
52+
- uses: actions/upload-artifact@v4
53+
with:
54+
name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
55+
path: ./wheelhouse/*.whl

‎.github/workflows/upload_pypi.yml

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
name: Publish wheels to pypi
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
workflow_id:
7+
description: "The workflow ID to pull wheels from"
8+
required: true
9+
type: string
10+
11+
jobs:
12+
publish:
13+
name: Publish Python 🐍 distribution 📦 to PyPI
14+
runs-on: ubuntu-latest
15+
16+
environment:
17+
name: pypi
18+
url: https://pypi.org/p/charm4py
19+
20+
permissions:
21+
id-token: write # IMPORTANT: mandatory for trusted publishing
22+
23+
steps:
24+
- name: Download all the dists
25+
uses: dawidd6/action-download-artifact@v2
26+
with:
27+
# download all the wheels
28+
path: unmerged
29+
run_id: ${{ github.event.inputs.workflow_id }}
30+
workflow: build_wheels.yml
31+
32+
- name: Merge files to dist
33+
run: |
34+
mkdir dist
35+
mv unmerged/*/*.whl dist
36+
mv unmerged/*/*.tar.gz dist
37+
38+
- name: Publish distribution 📦 to PyPI
39+
uses: pypa/gh-action-pypi-publish@release/v1

‎.github/workflows/upload_testpypi.yml

+41
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
name: Publish wheels to testpypi
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
workflow_id:
7+
description: "The workflow ID to pull wheels from"
8+
required: true
9+
type: string
10+
11+
jobs:
12+
publish_test:
13+
name: Publish Python 🐍 distribution 📦 to TestPyPI
14+
runs-on: ubuntu-latest
15+
16+
environment:
17+
name: testpypi
18+
url: https://test.pypi.org/p/charm4py
19+
20+
permissions:
21+
id-token: write # IMPORTANT: mandatory for trusted publishing
22+
23+
steps:
24+
- name: Download all the dists
25+
uses: dawidd6/action-download-artifact@v2
26+
with:
27+
# download all the wheels
28+
path: unmerged
29+
run_id: ${{ github.event.inputs.workflow_id }}
30+
workflow: build_wheels.yml
31+
32+
- name: Merge files to dist
33+
run: |
34+
mkdir dist
35+
mv unmerged/*/*.whl dist
36+
mv unmerged/*/*.tar.gz dist
37+
38+
- name: Publish distribution 📦 to TestPyPI
39+
uses: pypa/gh-action-pypi-publish@release/v1
40+
with:
41+
repository-url: https://test.pypi.org/legacy/

‎docs/install.rst

+39-39
Original file line numberDiff line numberDiff line change
@@ -7,82 +7,82 @@ Install
77
Charm4py runs on Linux, macOS, Windows, Raspberry Pi, and a wide variety of clusters and
88
supercomputer environments (including many supercomputers in the TOP500).
99

10-
Charm4py runs on Python 3.4+. Charm4py has been tested with the
10+
Charm4py runs on Python 3.7+. Charm4py has been tested with the
1111
following Python implementations:
1212
CPython (most common implementation) and PyPy_.
1313

1414

1515
.. _PyPy: https://pypy.org
1616

17+
Installing Charm4Py binaries (via pip)
18+
---------------------------------------
1719

18-
Installing Charm4Py on a laptop/personal machine
19-
------------------------------------------------
20+
The easiest way to install Charm4Py is via pip. Currently, pip wheels are available for Linux and macOS.
2021

21-
This install process covers the installation of Charm4Py on a laptop or personal machine, as opposed to a cluster.
22+
To install the latest release of Charm4Py, run::
2223

23-
Before installing, you need the following prerequisites:
24-
- CPython: numpy, greenlet and cython (``pip3 install 'numpy>=1.10.0' cython greenlet``)
25-
- PyPy: none
24+
$ pip install charm4py
2625

27-
You can get these prerequisites by running the following command::
26+
This will install the latest stable release of Charm4Py, using the default underlying Charm++ build
27+
(see the `Charm++ manual`_ for more information on the different builds of Charm++). If you want to
28+
use a specific Charm++ build, you can install and build Charm4Py from source. Note that the source distribution
29+
is available via "pip install", but the standard from source build process is via "git clone", as outlined below.
2830

29-
$ pip3 install -r requirements.txt
31+
Installing Charm4Py from source
32+
------------------------------------------------------------
33+
34+
This install process covers the installation of Charm4Py from source.
3035

3136
The first step is to clone the Charm4py repository from Git::
3237

3338
$ git clone https://github.com/charmplusplus/charm4py.git
3439
$ cd charm4py
3540

36-
Next, create a folder called charm_src in the charm4py repo, and then clone the Charm++ repo
37-
into that folder::
41+
Next, clone the Charm++ repo into charm_src::
3842

39-
$ mkdir charm_src && cd charm_src
40-
$ git clone https://github.com/charmplusplus/charm.git
43+
$ git clone https://github.com/charmplusplus/charm.git charm_src/charm
4144

42-
Once this is done, there are two ways to build Charm4py. The first way is to change back up
43-
into the Charm4Py directory and run the install script::
45+
Once this is done, there are two ways to build Charm4py. The first is to simply run the installation
46+
from the Charm4py root. This method will use the default Charm++ backend::
4447

4548
$ cd ..
46-
$ python3 setup.py install
49+
$ pip install .
4750

48-
The other option is to manually build Charm++ before building Charm4py. To do this, change to
49-
the charm directory and run the following build command::
51+
The other option is to manually build Charm++ before building Charm4py. This may be necessary
52+
if you want to configure Charm++ differently from the default. To do this, change to
53+
the charm directory and run the following build command, then build Charm4Py::
5054

5155
$ cd charm
52-
$ ./build charm4py netlrts-<os>-<architecture> -j<N> --with-production
53-
54-
For building on a laptop, you must use a netlrts build of Charm4Py. Check the Charm++ documentation
55-
to identify the correct os and architecture command to pass into the build command. The -j option
56-
is a cmake option that launches N threads for the make.
57-
58-
Then, return to the charm4py directory and run setup.py::
59-
56+
$ ./build charm4py <target-architecture> -j<N> --with-production
6057
$ cd ../..
61-
$ python3 setup.py install
58+
$ pip install .
6259

60+
Finally, if necessary, when installing dependencies or when running the install script, add the --user
61+
option to the Python command to complete the installation without permission errors.
6362

6463
After building, you can run Charm4py examples. One example you can try is
6564
array_hello.py, which can be run as follows::
6665

6766
$ cd examples/hello
6867
$ python -m charmrun.start +p2 array_hello.py
6968

70-
Installing Charm4Py on a cluster machine
71-
----------------------------------------
69+
Choosing your target architecture when building from source
70+
------------------------------------------------------------
7271

73-
To install Charm4Py on a cluster machine, you will generally follow the same steps as above, but
74-
with the following changes. First, when building Charm++, use the MPI build instead of the netlrts
75-
build::
72+
When building from source, as described above, you must chose the appropriate target architecture.
7673

77-
$ ./build charm4py mpi-<os>-<architecture> -j<N> --with-production
78-
79-
Next, pass in the MPI option to the python setup script::
80-
81-
$ python3 setup.py install --mpi
74+
For building on a laptop or personal machine, you must use a netlrts build of Charm4Py.
75+
For example, to build for a personal machine running macOS with ARM processors, using 4 cmake
76+
threads, you would run::
77+
78+
$ ./build charm4py netlrts-darwin-arm -j4 --with-production
8279

83-
Finally, if necessary, when installing dependencies or when running the install script, add the --user
84-
option to the Python command to complete the installation without permission errors.
80+
To install Charm4Py on a cluster machine, you will generally want to chose a different backend.
81+
For example, to use Charm4Py with MPI, build the Charm backend as follows::
8582

83+
$ ./build charm4py mpi-<os>-<architecture> -j<N> --with-production
8684

85+
Check the Charm++ documentation to identify the correct os and architecture command
86+
to pass into the build command.
8787

8888
.. _manual: https://charm.readthedocs.io/en/latest/charm++/manual.html#installing-charm

‎pyproject.toml

+44
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
[build-system]
2+
requires = ["setuptools>=43.0.0", "cython>=3.0.0", "numpy>=1.10.0"]
3+
build-backend = "setuptools.build_meta"
4+
5+
[project]
6+
description = 'Charm4py Parallel Programming Framework'
7+
authors = [
8+
{ name = "Juan Galvez", email = "[email protected]" },
9+
{ name = "Maya Taylor", email = "[email protected]" },
10+
]
11+
maintainers = [{ name = "Maya Taylor", email = "[email protected]" }]
12+
name = "charm4py"
13+
dynamic = ["version"]
14+
readme = { file = "README.rst", content-type = "text/x-rst" }
15+
keywords = [
16+
"parallel",
17+
"parallel programming",
18+
"distributed computing",
19+
"distributed",
20+
"hpc",
21+
"HPC",
22+
"runtime",
23+
]
24+
requires-python = ">=3.7"
25+
dependencies = ["numpy>=1.10.0", "greenlet>=3.0.0"]
26+
classifiers = [
27+
'Intended Audience :: Developers',
28+
'License :: Free for non-commercial use',
29+
'Operating System :: MacOS :: MacOS X',
30+
'Operating System :: POSIX',
31+
'Operating System :: POSIX :: Linux',
32+
'Operating System :: Microsoft :: Windows',
33+
'Programming Language :: Python',
34+
'Programming Language :: Python :: 3',
35+
'Topic :: System :: Distributed Computing',
36+
'Topic :: System :: Clustering',
37+
]
38+
39+
[project.urls]
40+
Documentation = "https://charm4py.readthedocs.io"
41+
Repository = "https://github.com/charmplusplus/charm4py"
42+
43+
[project.scripts]
44+
charmrun = 'charmrun.start:start'

‎requirements.txt

+2-5
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,3 @@
1-
cython>=3.0.0
2-
numpy
1+
numpy>=1.10.0
32
greenlet>=3.0.0
4-
5-
# Required for the charm++ build:
6-
cmake
3+
cython>=3.0.0

‎setup.py

+7-31
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,13 @@ def check_libcharm_version(charm_src_dir):
118118
def check_cffi():
119119
try:
120120
import cffi
121-
version = tuple(int(v) for v in cffi.__version__.split('.'))
121+
version_str = cffi.__version__.split('.')
122+
123+
# pypy3.9 returns version string like '1.17.0.dev0'
124+
if (len(version_str) > 3):
125+
version_str = version_str[:3]
126+
127+
version = tuple(int(v) for v in version_str)
122128
if version < (1, 7):
123129
raise DistutilsSetupError('Charm4py requires cffi >= 1.7. '
124130
'Installed version is ' + cffi.__version__)
@@ -369,42 +375,12 @@ def install(self):
369375
additional_setup_keywords['cffi_modules'] = 'charm4py/charmlib/charmlib_cffi_build.py:ffibuilder'
370376

371377

372-
with open('README.rst', 'r') as f:
373-
long_description = f.read()
374-
375-
376378
setuptools.setup(
377-
name='charm4py',
378379
version=charm4py_version,
379-
author='Juan Galvez and individual contributors',
380-
author_email='[email protected]',
381-
description='Charm4py Parallel Programming Framework',
382-
long_description=long_description,
383-
url='https://github.com/charmplusplus/charm4py',
384-
keywords='parallel parallel-programming distributed distributed-computing hpc HPC runtime',
385380
packages=setuptools.find_packages(),
386381
package_data={
387382
'charm4py': ['libcharm_version'],
388383
},
389-
entry_points={
390-
'console_scripts': [
391-
'charmrun = charmrun.start:start',
392-
],
393-
},
394-
install_requires=['numpy>=1.10.0', 'greenlet>=3.0.0', 'cython>=3.0.0', 'cmake'],
395-
python_requires='~=3.6',
396-
classifiers=[
397-
'Intended Audience :: Developers',
398-
'License :: Free for non-commercial use',
399-
'Operating System :: MacOS :: MacOS X',
400-
'Operating System :: POSIX',
401-
'Operating System :: POSIX :: Linux',
402-
'Operating System :: Microsoft :: Windows',
403-
'Programming Language :: Python',
404-
'Programming Language :: Python :: 3',
405-
'Topic :: System :: Distributed Computing',
406-
'Topic :: System :: Clustering',
407-
],
408384
ext_modules=extensions,
409385
cmdclass = {'build_py': custom_build_py,
410386
'build_ext': custom_build_ext,

0 commit comments

Comments
 (0)