Skip to content

Conversation

enwask
Copy link
Contributor

@enwask enwask commented Jul 23, 2025

Adds an additional base layer for building Python 3.13 from source with disable-gil, and makes the necessary changes to the rest of the build process as well as Docker build workflows.

Copy link
Contributor

@mloubout mloubout left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments but it looks good

@mloubout mloubout added dependencies Pull requests that update a dependency file installation labels Jul 24, 2025
@ggorman ggorman requested a review from Copilot July 24, 2025 08:52
Copilot

This comment was marked as outdated.

@enwask enwask requested a review from Copilot July 24, 2025 09:14
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for Python 3.13 free-threaded (GIL-disabled) Docker images to the Devito project. The changes enable building Docker containers with a custom Python 3.13 build that has the Global Interpreter Lock (GIL) disabled, allowing for true parallelism in Python code.

Key changes include:

  • Creation of a new Dockerfile for building Python 3.13 from source with GIL disabled
  • Modification of existing Dockerfiles to conditionally use either system Python or the custom free-threaded build
  • Updates to GitHub Actions workflows to build both standard and nogil variants of all Docker images

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
docker/Dockerfile.nogil New Dockerfile that builds Python 3.13 from source with --disable-gil flag
docker/Dockerfile.nvidia Modified to accept parameterized base image and conditionally install Python
docker/Dockerfile.intel Updated to use configurable base image and conditional Python installation
docker/Dockerfile.devito Added logic to handle free-threaded builds and skip incompatible packages like Jupyter
docker/Dockerfile.cpu Modified to support parameterized base image and conditional Python setup
docker/Dockerfile.amd Updated to copy Python build from base image and handle conditional installation
.github/workflows/docker-devito.yml Added nogil variants for all existing Docker image configurations
.github/workflows/docker-bases.yml Extended workflow matrix to build both standard and nogil versions of all base images
Comments suppressed due to low confidence (1)

zlib1g-dev

# Clone CPython repository at the 3.13 branch
RUN git clone --depth 1 --branch 3.13 https://github.com/python/cpython.git /usr/src/cpython
Copy link

Copilot AI Jul 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a specific branch '3.13' may cause issues if the branch is updated or removed. Consider pinning to a specific commit hash or tag for reproducible builds.

Suggested change
RUN git clone --depth 1 --branch 3.13 https://github.com/python/cpython.git /usr/src/cpython
RUN git clone --depth 1 https://github.com/python/cpython.git /usr/src/cpython && \
cd /usr/src/cpython && \
git checkout <COMMIT_HASH>

Copilot uses AI. Check for mistakes.

Copy link
Contributor

@EdCaunt EdCaunt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would probably be worth adding a run to CI-core which runs the tests with free-threaded Python

@enwask
Copy link
Contributor Author

enwask commented Jul 24, 2025

It would probably be worth adding a run to CI-core which runs the tests with free-threaded Python

Thought about this but I think lots of things will fail off the bat so not sure it should be in CI yet

@EdCaunt
Copy link
Contributor

EdCaunt commented Jul 24, 2025

Thought about this but I think lots of things will fail off the bat so not sure it should be in CI yet

That's a reasonable contention, but they will all need fixing eventually. We've been tripped up like this before (CXX anyone?)

@enwask
Copy link
Contributor Author

enwask commented Jul 24, 2025

That's a reasonable contention, but they will all need fixing eventually. We've been tripped up like this before (CXX anyone?)

Yeah, just wasn't sure that we want CI to always be red from now on—I think this is above my paygrade though so I'll defer to consensus

@enwask enwask requested a review from mloubout July 24, 2025 14:01
@enwask enwask force-pushed the nogil branch 2 times, most recently from 9fda9b2 to 61635e3 Compare July 24, 2025 15:04
@mloubout
Copy link
Contributor

I agree with @EdCaunt that at least one basic test needs to be added to the base no-mpi (i.e., pytest-core-nompi.py, possibly with just test_operator.py). This test file contains very little runtime and mostly just operator build, which should ensure that the base of Devito works with it.

@mloubout
Copy link
Contributor

Copy link

codecov bot commented Jul 29, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.73%. Comparing base (69ffd48) to head (3054e73).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2681      +/-   ##
==========================================
- Coverage   92.10%   87.73%   -4.37%     
==========================================
  Files         248      248              
  Lines       49654    49654              
  Branches     4368     4368              
==========================================
- Hits        45734    43566    -2168     
- Misses       3213     5326    +2113     
- Partials      707      762      +55     
Flag Coverage Δ
pytest-gpu-aomp-amdgpuX ?
pytest-gpu-nvc-nvidiaX ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Contributor

@mloubout mloubout left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just need the CI fixed but GTG. Need some rebase/cleanup of commits.

@enwask enwask force-pushed the nogil branch 3 times, most recently from de3b990 to 95a7e31 Compare July 29, 2025 20:34
cpu:
type: boolean
default: false
default: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this switch?

Copy link
Contributor Author

@enwask enwask Aug 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default is used for automated runs, so with a default of false none of the automated runs actually built any images

@EdCaunt
Copy link
Contributor

EdCaunt commented Aug 11, 2025

Commit history will want a cleanup before merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file installation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants