Skip to content

Add shared Podman and Apptainer runtimes - #2528

Open
xeophon wants to merge 12 commits into
mainfrom
feat/local-container-runtimes
Open

xeophon wants to merge 12 commits into
mainfrom
feat/local-container-runtimes

Conversation

@xeophon

@xeophon xeophon commented Sep 4, 2026

Copy link
Copy Markdown
Member

Overview

Adds podman and apptainer as local runtimes through one shared command, process, and file implementation. Docker and Podman use engine-managed bridge networking and the existing Verifiers proxy so host callbacks and published services remain reachable during restricted execution.

Details

  • Extract the common CLI execution machinery from Docker into ContainerRuntime; Podman shares Docker's lifecycle and networking with native NVIDIA CDI device selection. Use native detached exec for Docker/Podman background servers.
  • Use private container and instance names, and provision MCP runtimes through the existing context manager so failed startup releases owned resources.
  • Run Apptainer in contained instances with writable workspaces initialized from the image, cached SIF images, and process-group cancellation. Apptainer uses host networking and rejects execution-time network policies; its NVIDIA GPU count and local disk requests are advisory.
  • Send framework-owned host-loopback HTTP(S) callbacks through one tokenized listener, preserving certificate checks, IPv6 destinations, and redirect credential boundaries. Scope cookie names and paths to each callback so redirected sessions can establish their own cookies while retaining upstream TLS and credential boundaries. Bound request-body reads and close active handlers during teardown while allowing long-running callback responses.
  • Reuse task images containing Python 3 for Linux callback listeners; other images use the shared Python helper. Offline prerequisites are documented in the architecture guide.
  • Cache the Alpine networking helper with iptables installed. Preserve host-published service traffic while blocking peer requests and replies after the egress cut, retain background-server proxy settings, and route MCP exposure through Runtime.expose().

Performance

Isolating the helper-cache change on main (1e3e729) on an Apple M4 Max running macOS 27.0 and OrbStack Docker 29.4.0 reduced median DockerRuntime.start() + prepare_execution() latency from 0.823 s to 0.291 s: 64.6% lower latency (2.82×), saving 0.531 s per restricted start. This used six warm samples per variant, interleaved in three ABBA blocks after warmups. One cold helper build took 1.107 s, with 1.486 s for that first full start and policy application; base images were already cached. These measurements cover the helper cache on this machine only. Whole-PR performance, registry-cold pulls, Linux, model latency, and full evaluation throughput remain unmeasured.

Supersedes #2469, #2470, #2473, and #2509. Related to #2319 and #2359.


Note

High Risk
Large changes to Docker networking, egress proxy, and host callback routing affect restricted execution and interception/MCP reachability; container naming and expose semantics also changed.

Overview
Adds podman and apptainer as first-class local runtimes and refactors Docker onto a shared ContainerRuntime (CLI exec, processes, file I/O). Podman reuses Docker’s bridge networking, egress proxy, and published service port; Apptainer runs contained instances on the host network with cached SIF images and does not support execution-time network policies.

Docker/Podman now use bridge networking (not host), UUID container names, a cached localhost/verifiers-network:1 helper for restricted egress cuts, and host_url routes loopback framework HTTP(S) through the egress proxy’s tokenized callback URLs (cookies, redirects, auth scoping). Runtime.expose always returns a URL; container engines publish SERVICE_PORT to loopback. MCP serving uses provision_runtime, fixes trailing-slash URL building, and tunnels local services only for remote consumers.

Docs, pytest marks, and e2e placements cover the new runtimes (skipped when the CLI is missing).

Reviewed by Cursor Bugbot for commit f4faa84. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add shared PodmanRuntime and ApptainerRuntime with common ContainerRuntime base

  • Extracts shared container CLI logic into ContainerRuntime, ContainerConfig, and ContainerProcess in container.py, letting Docker, Podman, and Apptainer reuse process management, file transfer, and command execution.
  • Adds PodmanRuntime as a DockerRuntime specialization and a standalone ApptainerRuntime with image pull/caching, instance lifecycle, and workdir validation in apptainer.py.
  • Introduces a callback URL system in egress.py that routes loopback HTTP(S) services through the proxy with capability-scoped cookies, origin-bound authorization, and redirect conversion.
  • Reworks DockerRuntime startup in init.py to use an engine selector, generated container names, bridge networking with published SERVICE_PORT, and iptables-based restricted-network policy instead of package-install cuts.
  • Fixes trailing-slash URL joins in Rollout and MCP server state URL construction; updates reachable_url and Runtime.expose so non-colocated consumers receive the runtime's exposed URL.
  • Behavioral Change: DockerRuntime.host_url now returns proxy callback URLs for all loopback hosts (not just two hardcoded names) regardless of restriction state; NetworkPolicy.permits rejects loopback hosts before framework-route matching; Runtime.expose base implementation returns an HTTP URL instead of None; _PASS_LISTENER no longer sets SO_REUSEADDR.

Macroscope summarized f4faa84.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 4, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-14T19:24:57.942288Z 708dbeb New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Comment thread verifiers/v1/runtimes/docker/egress.py
Comment thread verifiers/v1/runtimes/docker/__init__.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0373dc6973

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/runtimes/docker/__init__.py
Comment thread verifiers/v1/runtimes/docker/__init__.py
Comment thread verifiers/v1/runtimes/docker/__init__.py
Comment thread verifiers/v1/runtimes/docker/__init__.py
@macroscopeapp

macroscopeapp Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds substantial Podman and Apptainer runtime capabilities and rewrites shared container execution, Docker networking, port exposure, and host-callback proxy behavior. The changes affect existing production paths and sandbox isolation, with an unresolved resource-leak finding in shared process handling, so the scope and risk warrant human review.

Not approved because:

  • 1 blocking correctness issue found at or above your repo's Minimum Blocking Severity

Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4caf3c4057

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/runtimes/apptainer.py
Comment thread verifiers/v1/runtimes/docker/__init__.py Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/runtimes/docker/egress.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ef30215d05

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/runtimes/apptainer.py
Comment thread tests/v1/test_e2e.py Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/runtimes/docker/__init__.py
Comment thread verifiers/v1/runtimes/container.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6112ce7447

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/runtimes/docker/egress.py
Comment thread verifiers/v1/runtimes/docker/egress.py Outdated
Comment thread verifiers/v1/runtimes/docker/egress.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c3c46865fd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/runtimes/docker/egress.py Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/runtimes/docker/egress.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 38cf0c2941

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/runtimes/docker/egress.py
Comment thread verifiers/v1/runtimes/docker/egress.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 72bfe978e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/runtimes/docker/egress.py Outdated
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@xeophon
xeophon force-pushed the feat/local-container-runtimes branch from f012c79 to c2d39dc Compare September 14, 2026 14:16
Comment thread verifiers/v1/runtimes/__init__.py
Comment thread verifiers/v1/runtimes/docker/__init__.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c2d39dcb24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/runtimes/apptainer.py
Comment thread verifiers/v1/runtimes/docker/egress.py

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c2d39dc. Configure here.

Comment thread verifiers/v1/runtimes/apptainer.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a1e83fd601

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/runtimes/docker/egress.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 56e9b94b31

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/runtimes/docker/egress.py
@xeophon
xeophon force-pushed the feat/local-container-runtimes branch from 56e9b94 to 708dbeb Compare September 14, 2026 19:16

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 708dbebc0b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/runtimes/docker/egress.py
# post-setsid PID before exec preserves it as the target PID.
wrapper = (
"if setsid -w true >/dev/null 2>&1; then "
'exec setsid -w sh -c \'echo $$ > "$1"; shift; exec "$@"\' '

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium runtimes/container.py:167

open_process leaves each /tmp/vf-process-*.pid file behind after the process exits, so long-lived shared container and Apptainer runtimes accumulate one file per session and can eventually exhaust their writable workspace. Pass the PID-file path to ContainerProcess and remove it when wait() completes, while retaining the existing startup-cleanup path for failures.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/runtimes/container.py around line 167:

`open_process` leaves each `/tmp/vf-process-*.pid` file behind after the process exits, so long-lived shared container and Apptainer runtimes accumulate one file per session and can eventually exhaust their writable workspace. Pass the PID-file path to `ContainerProcess` and remove it when `wait()` completes, while retaining the existing startup-cleanup path for failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant