Skip to content

Validate requested GPU types in Docker runtime - #2545

Open
xeophon wants to merge 2 commits into
mainfrom
docker-gpu-type-check
Open

Validate requested GPU types in Docker runtime#2545
xeophon wants to merge 2 commits into
mainfrom
docker-gpu-type-check

Conversation

@xeophon

@xeophon xeophon commented Sep 6, 2026

Copy link
Copy Markdown
Member

Typed GPU requests such as H100:2 keep Docker's --gpus 2 allocation and check the exposed devices before task setup. Startup fails with a clear error when the GPU type or count does not match, or when the device query cannot run.

The check uses one nvidia-smi invocation inside the started container, so it inspects devices exposed by the selected Docker daemon. Product names match case-insensitively at whitespace and hyphen boundaries. Count-only and no-GPU requests retain their existing behavior.

Separately provisioned MCP server runtimes register teardown before startup, so partially started containers are removed when startup fails or is cancelled.


Note

Medium Risk
GPU-backed sandboxes can fail at startup when the image lacks nvidia-smi or NVIDIA product names do not match the configured type string, though mis-provisioned GPUs are caught earlier with clearer errors.

Overview
Docker runtime now verifies typed GPU configs (e.g. H100:2) after docker run succeeds, instead of only passing --gpus with the parsed count.

When both a GPU type and count come from parse_gpu, the runtime runs nvidia-smi inside the new container and fails startup with SandboxError if the query fails, the number of exposed devices differs from the count, or any device name does not match the requested type (case-insensitive match at whitespace/hyphen boundaries via re). Count-only specs (2) and no-GPU configs are unchanged—no type check runs without a type.

The DockerConfig.gpu field docstring is updated to describe the type[:count] format and this validation behavior.

Reviewed by Cursor Bugbot for commit 2843527. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Validate requested GPU types in DockerRuntime.start and fix cleanup ordering in mcp._serve

  • DockerRuntime.start now runs nvidia-smi after container startup for typed GPU requests and rejects startup if the query fails, the exposed GPU count differs, or device names do not match the requested type (case-insensitive, whitespace/hyphen boundaries). Count-only GPU requests skip this check.
  • Updates DockerConfig GPU field docs to describe type-with-count and count-only forms plus the nvidia-smi verification requirement.
  • mcp._serve registers the runtime stop callback before awaiting startup for separately provisioned runtimes, so runtime.stop() runs during cleanup if runtime.start() raises.
  • Behavioral Change: typed GPU requests that previously started with mismatched or unqueryable GPUs now raise SandboxError; existing configs with correct types are unaffected.

Macroscope summarized a091797.

Comment thread verifiers/v1/runtimes/docker/__init__.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2843527193

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/runtimes/docker/__init__.py
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-06T15:06:32.331480Z 2843527 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@macroscopeapp

macroscopeapp Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR changes an existing Docker startup path by validating typed GPUs after container creation and adjusting startup teardown ordering. An unresolved review concern identifies a possible leaked container when GPU verification fails, so the cleanup behavior should receive human confirmation.

You can add or adjust custom eligibility rules. Learn more.

@xeophon
xeophon enabled auto-merge (squash) September 7, 2026 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant