feat(v1): Podman and Apptainer runtimes, private container networking

[xeophon]

Superseded by #2528.

What this does

Adds two new local runtimes, type = "podman" and type = "apptainer", and moves all local containers onto a private network so a task's ports can never collide with the host's.

• Fixes DockerRuntime uses --network host for unrestricted tasks, silently corrupting grading when host ports collide #2319 (Docker used --network host, so a task binding port 8000 could silently talk to an unrelated host process and get graded wrong).
• Resolves Support for other runtimes like apptainer or enroot #2359 by adding Apptainer for HPC clusters. Enroot is not part of this PR.
• Supersedes Isolate Docker runtime networking #2469, Add Podman runtime support #2470 and Add Apptainer runtime support #2473. Those three did the same things with a lot more code; this PR replaces them with one smaller change.

How it is built

One shared base for every local container. runtimes/container.py holds what used to be the engine-independent half of the Docker runtime: running a command, opening a live process, background processes, reading and writing files, and sending signals. All of it goes through a single method, _exec(env), that returns "the host command that runs something inside this container". A runtime only has to provide _exec, start and cleanup.

Podman is Docker with a different binary. PodmanRuntime is a two-line subclass of DockerRuntime. Podman accepts the same commands and even aliases host.docker.internal, so there is nothing else to write. Two small differences are handled inside the Docker code: Podman refuses a --workdir that does not exist in the image (Docker creates it), so the workdir is now created with mkdir -p right after start; and Podman's --gpus takes all where Docker takes a count.

Apptainer is a small subclass too. It starts an unprivileged instance, binds a host directory as the workdir, and backs the contained /tmp and $HOME with disk instead of a tiny tmpfs. Images given as Docker references are pulled to a SIF once and cached under ~/.cache/verifiers, the same way an engine keeps a pulled image. A local .sif path is used as is. Apptainer always shares the host network, so it has no egress fields; a task that asks for a network policy on it gets a clear error.

How the networking works now

Every Docker/Podman container runs on the engine's bridge network. That is the fix for #2319. Two things then have to keep working:

The container must reach the interception server on the host. The interception server (and any host-side tool server) listens on the host's loopback. On Linux, host_url leaves the URL unchanged and remembers its port. Before the next command runs in the container, one small helper container that shares the network namespace binds a listener at that port on the container's loopback and hands the socket back to us; we relay every connection to the host's loopback. Inside the container, http://127.0.0.1:PORT simply works, with no proxy settings needed, so harnesses that ignore HTTP_PROXY (Node based ones) keep working. On macOS, host_url rewrites loopback to host.docker.internal, as before.

Egress must be cuttable. Restricted runtimes keep the existing HTTP(S) policy proxy and the existing route cut. The proxy is now an ordinary host-loopback service reached like any other. On Linux the cut leaves loopback alone, so the doors keep working. On macOS, restricted framework traffic goes through the proxy as before (the proxy dials the host's loopback for host.docker.internal), so the cut opens only the proxy port. The proxy-inside-the-container listener and the made-up vf.host.internal name are gone; the engine's own alias does that job.

The host must reach a tool server placed inside a container. The fixed service port is published to a host loopback port, and Runtime.expose returns it. Runtime.expose now defaults to host loopback, so the MCP launch code has one rule for every runtime instead of three cases.

Behavior changes for Docker

• Private bridge network instead of the host network (the point of the PR).
• The workdir is created after start (as root, like run --workdir did) instead of via run --workdir.
• Helper images are fully qualified (docker.io/library/...) and the socket directory is mounted with --volume ...:Z, so Podman and SELinux hosts work.
• run_background starts the process in the background inside the container instead of exec --detach, which Apptainer does not have.
• In restricted mode the proxy environment is set from the start, not only after the cut. Setup traffic goes direct until the cut anyway.

How I tested it

Automated: uv run pytest tests/v1 -m "not e2e" (82 tests), uv run pre-commit run --all-files, and ty check verifiers all pass. The e2e suite needs PRIME_API_KEY, which I do not have here.

Live, with two throwaway scripts (below) run against each engine. The first covers the process side: run, environment values containing commas, stdin closed for plain runs, binary write/read, a live process fed on stdin and then terminated with its child reaped, a background server, and installing and running a uv script inside. The second covers networking: a host HTTP server reached from inside through host_url, proof that the namespace is private (a port busy on the host is free inside), a server inside reached from the host through expose, two doors opened in one batch, and the restricted flow: host reachable before and after the cut, a blocked host gets 403, an allowed host works through the proxy, raw TCP egress is dead, and live processes see the proxy environment.

Engine	Where	Result
Docker 29	macOS, OrbStack	both scripts pass, both network modes
Podman 4.9 rootless	Ubuntu 24.04 arm64 VM	both scripts pass, both network modes
Apptainer 1.5.3	Ubuntu 24.04 arm64 VM	both scripts pass (unrestricted only, by design)

Not tested by hand: Podman on macOS through podman machine and GPU flags. Docker on Linux is covered by CI's live E2E suite, which passed on this branch. After the review fixes, both macOS Docker runs were repeated; Linux is re-verified by CI on the final commit.

Two bugs the live tests caught that unit tests would not have: rootless Podman defaults to pasta, which has no eth0, so --network bridge is now explicit for both engines; and the listener-planting helper dropped its first socket to garbage collection when two ports were requested at once.

Smoke script 1: processes and files (smoke_runtime.py <docker|podman|apptainer> [--skip-restricted])

"""Smoke-test the refactored container runtime against local Docker."""

import asyncio
import sys

from verifiers.v1.runtimes import provision_runtime
from verifiers.v1.runtimes import RuntimeConfig
from pydantic import TypeAdapter
KIND = sys.argv[1]
def Config(**kw):
    return TypeAdapter(RuntimeConfig).validate_python({"type": KIND, **kw})


async def collect(stream):
    return b"".join([chunk async for chunk in stream])


async def unrestricted():
    print("== unrestricted ==")
    async with provision_runtime(Config(), env={"BASE": "1"}) as rt:
        r = await rt.run(["sh", "-c", "echo hi $BASE $EXTRA; pwd"], {"EXTRA": "a,b=c"})
        assert r.exit_code == 0, r
        assert r.stdout.strip() == "hi 1 a,b=c\n/app".replace("\n", "\n"), repr(r.stdout)
        # stdin of `run` is EOF, not the terminal
        r = await rt.run(["sh", "-c", "cat; echo done"], {})
        assert r.stdout.strip() == "done", repr(r.stdout)
        # write / read (binary-safe), relative to workdir
        data = bytes(range(256)) * 10
        await rt.write("sub/dir/blob.bin", data)
        assert await rt.read("sub/dir/blob.bin") == data
        assert await rt.read("/app/sub/dir/blob.bin", max_bytes=len(data)) == data
        try:
            await rt.read("missing.txt")
        except Exception as e:
            print("  read missing ->", type(e).__name__, str(e)[:60])
        else:
            raise AssertionError("read of missing file did not raise")
        # live process: stdin echo, then terminate reaps the group
        p = await rt.open_process(["sh", "-c", "read x; echo got:$x; sleep 30 & wait"], {})
        await p.write(b"ping\n")
        stdout = asyncio.create_task(collect(p.stdout))
        await asyncio.sleep(0.5)
        await p.terminate()
        code = await asyncio.wait_for(p.wait(), 10)
        out = await asyncio.wait_for(stdout, 10)
        assert b"got:ping" in out, out
        print("  live process exit", code, "stdout", out)
        left = await rt.run(["sh", "-c", "cat /proc/[0-9]*/cmdline 2>/dev/null | tr '\\0' ' ' | grep -c 'sleep 3[0]' || true"], {})
        assert left.stdout.strip() == "0", ("sleep survived terminate", left.stdout)
        # background server: returns immediately, keeps running, logs to file
        t = asyncio.get_running_loop().time()
        await rt.run_background(["sh", "-c", "echo started; sleep 60"], {}, "bg.log")
        assert asyncio.get_running_loop().time() - t < 3, "run_background blocked"
        await asyncio.sleep(0.5)
        assert (await rt.read("bg.log")).strip() == b"started"
        running = await rt.run(["sh", "-c", "cat /proc/[0-9]*/cmdline 2>/dev/null | tr '\\0' ' ' | grep -c 'sleep 6[0]'"], {})
        assert running.stdout.strip() == "1", running
        print("  uv script:", (await rt.run_uv_script("# /// script\n# dependencies = []\n# ///\nprint('uv ok')")).stdout.strip())
    print("  ok")


async def restricted():
    print("== restricted ==")
    cfg = Config(image="docker.io/library/python:3.11-slim", block=["example.com"])
    async with provision_runtime(cfg) as rt:
        assert rt.network_restricted
        # setup phase: direct egress works
        r = await rt.run(["python3", "-c", "import urllib.request;print(urllib.request.urlopen('http://example.com', timeout=10).status)"], {})
        assert r.stdout.strip() == "200", r
        await rt.prepare_execution(["http://127.0.0.1:1/v1"])
        # after the cut: blocked destination fails, allowed one works through the proxy
        blocked = await rt.run(["python3", "-c", "import urllib.request;print(urllib.request.urlopen('http://example.com', timeout=10).status)"], {})
        assert blocked.exit_code != 0, blocked
        print("  blocked ->", blocked.stderr.strip().splitlines()[-1][:80])
        allowed = await rt.run(["python3", "-c", "import urllib.request;print(urllib.request.urlopen('https://pypi.org/simple/', timeout=20).status)"], {})
        assert allowed.stdout.strip() == "200", allowed
        # env plumbing after the cut includes the proxy for run + open_process
        p = await rt.open_process(["sh", "-c", "echo $HTTPS_PROXY | sed 's/:[^@]*@/:***@/'"], {})
        out = await collect(p.stdout)
        assert b"http:***@" in out, out
        print("  proxy env in live process:", out.strip())
        assert await p.wait() == 0
    print("  ok")


async def main():
    await unrestricted()
    if "--skip-restricted" not in sys.argv:
        await restricted()


asyncio.run(main())

Smoke script 2: networking (smoke_net.py <docker|podman|apptainer> [--skip-restricted])

"""Network smoke test: private netns, host loopback doors, published service port, cut."""

import asyncio
import sys

from aiohttp import web
from pydantic import TypeAdapter

from verifiers.v1.runtimes import RuntimeConfig, provision_runtime
from verifiers.v1.runtimes.base import SERVICE_PORT

KIND = sys.argv[1]
PY = "import sys,urllib.request;print(urllib.request.urlopen(sys.argv[1], timeout=15).read().decode())"
FETCH = ["python3", "-c", PY]


def Config(**kw):
    return TypeAdapter(RuntimeConfig).validate_python({"type": KIND, **kw})


async def host_server():
    """A host-loopback HTTP service standing in for the interception server."""
    app = web.Application()
    app.router.add_get("/ping", lambda r: web.Response(text=f"pong {r.path_qs}"))
    runner = web.AppRunner(app)
    await runner.setup()
    site = web.TCPSite(runner, "127.0.0.1", 0)
    await site.start()
    return runner, site._server.sockets[0].getsockname()[1]


async def main():
    runner, port = await host_server()
    try:
        print(f"== {KIND} unrestricted ==")
        async with provision_runtime(Config()) as rt:
            url = rt.host_url(f"http://127.0.0.1:{port}")
            print("  host_url ->", url)
            r = await rt.run([*FETCH, f"{url}/ping?a=1"], {})
            assert r.stdout.strip() == "pong /ping?a=1", r
            # private netns: a port busy on the host (and not yet door-ed) is free inside
            runner2, port2 = await host_server()
            if KIND != "apptainer":
                r = await rt.run(["python3", "-c", "import socket;s=socket.socket();s.bind(('127.0.0.1',%d));print('bound')" % port2], {})
                assert r.stdout.strip() == "bound", ("host port leaked into the container", r)
            # long idle connection through the door survives (no relay timeout)
            # published service port: a server inside on 0.0.0.0:SERVICE_PORT is reachable at expose()
            await rt.run_background(["python3", "-m", "http.server", str(SERVICE_PORT), "--bind", "0.0.0.0", "-d", "/tmp"], {}, "srv.log")
            exposed = await rt.expose(SERVICE_PORT)
            print("  expose ->", exposed)
            import urllib.request
            for _ in range(50):
                try:
                    code = urllib.request.urlopen(exposed, timeout=2).status
                    break
                except Exception:
                    await asyncio.sleep(0.2)
            assert code == 200, code
            # a second host_url'd port opens another door in the same container
            url2 = rt.host_url(f"http://127.0.0.1:{port2}")
            r = await rt.run([*FETCH, f"{url2}/ping"], {})
            assert r.stdout.strip() == "pong /ping", r
            await runner2.cleanup()
            print("  ok")

        if "--skip-restricted" in sys.argv:
            return
        print(f"== {KIND} restricted ==")
        async with provision_runtime(Config(block=["example.com"])) as rt:
            url = rt.host_url(f"http://127.0.0.1:{port}")
            # setup phase: host reachable, egress open
            r = await rt.run([*FETCH, f"{url}/ping"], {})
            assert r.stdout.strip() == "pong /ping", r
            r = await rt.run([*FETCH, "http://example.com"], {})
            assert r.exit_code == 0, r
            await rt.prepare_execution([f"{url}/v1"])
            # after the cut: host still reachable, blocked host denied, allowed host via proxy
            r = await rt.run([*FETCH, f"{url}/ping"], {})
            assert r.stdout.strip() == "pong /ping", ("interception unreachable after cut", r)
            r = await rt.run([*FETCH, "http://example.com"], {})
            assert r.exit_code != 0, r
            print("  blocked ->", r.stderr.strip().splitlines()[-1][:70])
            r = await rt.run([*FETCH, "https://pypi.org/simple/"], {})
            assert r.exit_code == 0, r
            # raw egress (proxy bypass) is cut
            r = await rt.run(["python3", "-c", "import socket;s=socket.socket();s.settimeout(3);s.connect(('1.1.1.1',80))"], {})
            assert r.exit_code != 0, ("raw egress survived the cut", r)
            # live process sees proxy env
            p = await rt.open_process(["sh", "-c", "echo $HTTPS_PROXY $NO_PROXY"], {})
            out = b"".join([c async for c in p.stdout])
            assert b"@" in out and b"localhost,127.0.0.1" in out, out
            assert await p.wait() == 0
            print("  ok")
    finally:
        await runner.cleanup()


asyncio.run(main())

Known limits

• Podman GPU requests emit --gpus all, which needs Podman 5. Docker keeps its device count.
• Apptainer maps any GPU request to --nv (all NVIDIA GPUs); the count is advisory.
• Apptainer cannot restrict egress without root or CNI setup, so allow/block are rejected for it.

---

Note

High Risk
Changes sandbox networking, egress restriction, and service URL exposure across Docker/Podman/MCP—mistakes could break grading, interception reachability, or leak/block traffic incorrectly.

Overview
Adds podman and apptainer as v1 runtimes (wired through RuntimeConfig, public exports, and architecture docs) and refactors local containers around a shared ContainerRuntime in container.py so Docker/Podman only implement _exec, start, and cleanup.

Docker/Podman now run on the engine bridge (not host networking), publish the fixed service port to host loopback, and implement expose for that URL. On Linux, host_url schedules loopback “doors” (listeners in the container netns relayed to the host) so harnesses can reach interception on 127.0.0.1 without relying on HTTP_PROXY; restricted egress keeps the policy proxy on host loopback and updates the network cut / HOST_ALIAS (host.docker.internal) accordingly. Podman is a thin subclass; Apptainer runs unprivileged instances on the host network with cached SIF pulls.

Runtime.expose now always returns a concrete URL (default http://127.0.0.1:{port}), which unifies MCP reachable_url (always expose, Prime Tunnel only when a local runtime serves a remote consumer) and gates MCP_HOST on both exposed and published_port.

^{Reviewed by Cursor Bugbot for commit dc14ff9. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add PodmanRuntime and ApptainerRuntime and refactor container networking

• Extracts shared ContainerRuntime and ContainerConfig base classes so Docker and Podman share container lifecycle and execution logic. PodmanRuntime reuses the Docker implementation with the podman CLI.
• Adds ApptainerRuntime for unprivileged local instances using the host network, with local image file support and a digest-keyed SIF cache.
• Updates restricted container networking: Linux containers use lazily-created host loopback relay doors, and non-Linux containers use host.docker.internal for host resolution.
• Behavioral Change: Runtime.expose() now returns a concrete loopback HTTP URL instead of None. EgressProxy.start no longer accepts an externally supplied listener socket. The MCP launcher omits MCP_HOST when serve_in_runtime is called with exposed=False.

^{Macroscope summarized dc14ff9.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-02T17:22:53.081605Z	1e43eaa	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 309046be9e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds two production runtime modes and materially changes existing Docker networking, egress enforcement, process execution, service exposure, and MCP routing. The cross-cutting runtime and network-boundary changes are too substantial for automatic approval.

^{You can add or adjust custom eligibility rules. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1e43eaa. Configure here.}