[Power] Require B200 and B300 multinode telemetry / 强制启用 B200 和 B300 多节点功耗采集 - #2688
Conversation
1ecb394 to
a52acf5
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a52acf5097
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
edwingao28
changed the title
…-power # Conflicts: # perf-changelog.yaml
…-power # Conflicts: # .github/workflows/test-process-result.yml # perf-changelog.yaml # runners/launch_h200-dgxc-slurm.sh # utils/agentic/aggregation/power_adapter.py # utils/agentic/aggregation/test_power_adapter.py # utils/test_aggregate_power_multinode.py # utils/test_h200_power_official_contract.py
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f791a15. Configure here.
…-power # Conflicts: # .github/workflows/test-process-result.yml # perf-changelog.yaml # utils/test_gb200_power_official_contract.py
…-power # Conflicts: # perf-changelog.yaml
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
…-power # Conflicts: # perf-changelog.yaml
1 participant
What changed
launch_b200-dgxc.sh,launch_b200-nscale-slurm.sh, andlaunch_b300-nv.sh.dcgm-powerwithrequired: trueon 37 fixed-sequence recipes (21 B200 + 16 B300): 18 SGLang and 19 vLLM, including 15 dedicated-infrastructure topologies — both pools are back in service.e5c837f06a362dc888dfea2ee588e9f19c298270; preserve every existing non-power clone revision and keep Kimi-K3 and AgentX recipes unchanged.Why required
required: truepromotes exporter startup timeouts, launch failures, and endpoint resolution failures to blocking validation failures inutils/aggregate_power_multinode.py, so a recipe cannot publish a result whose power collection never started.Stack
6ad27ddbe6761df4e7947c62cb1422db51837620).e5c837f06a362dc888dfea2ee588e9f19c298270.mainafter PR-A merges.Validation
bash -n;git diff --checkpassed.中文说明
变更内容
launch_b200-dgxc.sh、launch_b200-nscale-slurm.sh和launch_b300-nv.sh。dcgm-power并设为required: true：其中 18 个 SGLang、19 个 vLLM，含 15 个独立基础设施节点拓扑——两个 pool 均已恢复服务。e5c837f06a362dc888dfea2ee588e9f19c298270；保留全部现有非功耗 clone revision，且不修改 Kimi-K3 和 AgentX recipe。为何设为 required
required: true会在utils/aggregate_power_multinode.py中把 exporter 启动超时、启动失败和 endpoint 解析失败提升为阻断性校验失败，确保功耗采集未启动的 run 无法发布结果。依赖关系
6ad27ddbe6761df4e7947c62cb1422db51837620）。e5c837f06a362dc888dfea2ee588e9f19c298270。main。验证
bash -n；git diff --check通过。Note
Medium Risk
Touches production Slurm launch paths and makes power collection blocking for 37 official lanes; mis-gating or fork pin drift could break benchmarks or reject valid runs, though contract tests and preserved non-power clone paths limit blast radius.
Overview
Requires mandatory DCGM power telemetry on 37 fixed-sequence multinode benchmark recipes (21 B200, 16 B300) for DeepSeek-V4 and Kimi K2.6 FP4 SGLang/vLLM lanes. Each recipe gains a
telemetryblock withprovider: dcgm-powerandrequired: true, so failed exporter startup or collection blocks result publication downstream.Wires B200/B300 Slurm launchers (
launch_b200-dgxc.sh,launch_b200-nscale-slurm.sh,launch_b300-nv.sh) to detect power-enabled recipes from YAML, enforce per-cluster lane guards, and only then clone a pinnedsrt-slurmproducer fork, stage the NVIDIA DCGM exporter image intosrtslurm.yaml, and archive producer/exporter provenance under runpower/logs. Non-power runs keep their existing clone pins.Adds
utils/test_b200_b300_power_official_contract.pyplus CI path/pytest coverage intest-process-result.yml, documents the change inperf-changelog.yaml, and setsset_cuda_visible_devices: trueon affected vLLM DeepSeek-V4 B200 recipes. Nscale preflight skips are broadened to alldsv4configs when weights are node-local.Reviewed by Cursor Bugbot for commit c99c52c. Bugbot is set up for automated code reviews on this repo. Configure here.