Skip to content

FIX: modal_setup.py deploys on a new Modal workspace (image builder 2026.10) - #212

Merged
ConalMullan merged 1 commit into
digitalsamba:mainfrom
jdela141252-lab:fix/modal-builder-version
Nov 12, 2026
Merged

ConalMullan merged 1 commit into
digitalsamba:mainfrom
jdela141252-lab:fix/modal-builder-version

Conversation

@jdela141252-lab

@jdela141252-lab jdela141252-lab commented Nov 10, 2026 •

Copy link
Copy Markdown

What broke

A Modal workspace created now (seen 2026-11-10) builds images with builder version 2026.10 by default. The client in uv.lock is modal==1.5.0, which only knows 2023.12, 2024.04, 2024.10 and 2025.06, so every modal deploy from /setup failed before building anything:

This version of the modal client supports the following image builder versions: {...}.
You are using '2026.10'. Please update your client library (pip install --upgrade modal).

Exporting MODAL_IMAGE_BUILDER_VERSION=2025.06 by hand fixed it: qwen3_tts, flux2 and music_gen built and deployed on 2025.06.

Why the toolkit needs it: without it, /setup can't deploy any Modal app for anyone who signs up for Modal today. It is a fix to an existing tool, not a feature: no new dependency, service, licence or cost.

The fix

tools/modal_setup.py recognises that error, retries the step with MODAL_IMAGE_BUILDER_VERSION=2025.06, and keeps the variable for the rest of the run, so only the first app fails once. It applies to SoulX's populate_weights step as well.

  • Only when needed. A workspace whose builder version the client accepts is never pinned. Pinning every deploy would rebuild all of an older workspace's cached images (ltx2's is ~80GB) under a builder they were never built with.
  • The user's choice wins. A version set in the shell, in .env (the tool loads it) or in ~/.modal.toml is left alone. If the client refuses that one, the new hint says to set a listed version or remove it.
  • The wrapped panel. The Modal CLI prints errors in a rich panel wrapped at 80 columns when piped, which splits "supports the following image builder versions" across two lines. Output is now flattened (borders dropped, lines joined) before matching, which also makes the existing hints (payment method, rate limit, Hugging Face access) survive a wrap.
  • --dry-run and --json show the version when one is set.

Docs: docs/modal-setup.md gets a section with the error, the bash and PowerShell exports for a deploy by hand, and why not to set it on an older workspace. /setup says the first deploy failing and retrying is expected, and .env.example points deploys by hand at the variable. CHANGELOG entry under Unreleased.

Why not bump modal instead

uv lock --upgrade-package modal would pick 1.6.3, which does accept 2026.10 (checked in an isolated env). In 1.6.3's source, builder 2026.10:

  • drops Python 3.10 (SUPPORTED_PYTHON_SERIES["2026.10"] starts at 3.11), and docker/modal-sadtalker and docker/modal-propainter use add_python="3.10", so those two would stop building on a new workspace;
  • splits the registry image and standalone Python into separate layers, and the FROM line of a Dockerfile into its own image.

So a bump alone would trade one failure for another. It needs either this pin or sadtalker and propainter moved to 3.11 and rebuilt, and that is worth its own PR with real deploys. A comment beside FALLBACK_BUILDER_VERSION says so. All nine apps already install fastapi[standard] themselves, so 2025.06 (which no longer puts the client's dependencies in the image) doesn't break their web endpoints.

Testing

  • tests/python/test_modal_setup.py (8 tests, deploy subprocess faked): the real wrapped panel from modal 1.5.0 at 80 columns and the bare message both trigger one retry, a later app starts pinned, SoulX's weights step retries, a version from the env var or ~/.modal.toml is not overridden and gets the hint, other failures aren't retried, an accepted workspace is never pinned, and the dry run shows the variable.
  • The detector was checked against the message modal 1.5.0's own _get_image_builder_version("2026.10") raises, for both the workspace default and the env var source.
  • uv run tools/copy_lint.py --diff origin/main --commits origin/main is clean; CI's ruff selection passes.
  • Not run: a live deploy through the new retry path. The 2025.06 deploys themselves were verified by hand on the new workspace.

🤖 Generated with Claude Code

… builder version (2026.10) is retried with 2025.06, which the rest of the run keeps

A Modal workspace created now defaults to image builder 2026.10. The Modal
client in uv.lock (1.5.0) only knows 2023.12 to 2025.06, so every deploy failed
before building anything until MODAL_IMAGE_BUILDER_VERSION was exported by hand.

modal_setup.py now recognises that error (after joining the lines of the CLI's
wrapped error panel), retries the step with 2025.06 and keeps it for later apps
in the run. A version the user set (shell, .env or ~/.modal.toml) is left alone
and gets a hint instead, and a workspace the client accepts is never pinned, so
its cached images aren't rebuilt.

Bumping modal (1.6.3 accepts 2026.10) is not enough on its own: 2026.10 drops
Python 3.10, which the sadtalker and propainter images use.

Docs: modal-setup.md, /setup and .env.example tell deploys by hand to export
the variable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdela141252-lab jdela141252-lab changed the title FIX: modal_setup.py deploys on a new Modal workspace: a refused image… FIX: modal_setup.py deploys on a new Modal workspace (image builder 2026.10) Nov 10, 2026
@ConalMullan

Copy link
Copy Markdown
Collaborator

Thanks, this is a clear fix and a careful write-up. One small thing: the comment beside FALLBACK_BUILDER_VERSION names sadtalker and dewatermark as the apps on Python 3.10, while the description says sadtalker and propainter. If dewatermark is the propainter image under another name, a word in the comment saying so would save the next reader a search. Not a blocker; merging once CI is green.

@ConalMullan
ConalMullan merged commit a036a27 into digitalsamba:main Nov 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants