Test AIPerf empty-content TTFT on MiniMax TRT - #2682
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
cquil11
changed the title
a5960e3 to
ecb89bc
Compare
|
Claude finished @cquil11's task in 3m 49s —— View job PR Review
SummaryReviewed all 3 changed files. The benchmark script change is sound: One 🟡 warning posted inline on No blocking issues found. |
|
+viz @ajcasagrande |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32305145240 |
ecb89bc to
f17aecd
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| @@ -1 +1 @@ | |||
| Subproject commit 754356e9a39acc6cc6afb242d123bb57c3fb6f75 | |||
| Subproject commit e8ece5d29bd0285af1ced36fbc4c670328b2d58c | |||
There was a problem hiding this comment.
🟡 WARNING: Shared submodule pinned to an unmerged PR head
Why it matters: utils/aiperf is consumed by every agentic recipe (all of them build REPLAY_CMD from the same submodule), not just minimaxm3-fp4-b300-trtllm-agentic-mtp. Two things can't be verified from this diff alone:
- Ancestry: if
e8ece5d29is not a fast-forward descendant of the current pin754356e9a(e.g. aiperf PR [NVIDIA] Update B200 SGL FP8 and add FP4 TP4 #42 is based on an older or diverged base), this pin silently changes or rolls back aiperf behavior for all other agentic sweeps, while the perf-changelog entry scopes the change to the MiniMax config only. It's only safe if every behavior change in the754356e9a..e8ece5d29range is gated behind the opt-in--allow-empty-contentflag. - Reachability: the submodule remote is
SemiAnalysisAI/aiperf.git, but the PR description says the commit is the head ofSemiAnalysisAI/agentx-harnessPR [NVIDIA] Update B200 SGL FP8 and add FP4 TP4 #42 — a different repo name. If the commit only exists outside the aiperf repo network,git submodule updatewill fail on fresh checkouts. Additionally, once PR [NVIDIA] Update B200 SGL FP8 and add FP4 TP4 #42 is rebased or squash-merged, this dangling commit stays reachable only through the PR ref, which is fragile for long-term reproducibility onmain.
Fix: Confirm in the aiperf repo that git merge-base --is-ancestor 754356e9a e8ece5d29 holds and that the range is fully gated behind --allow-empty-content; then re-pin to the merged mainline aiperf commit once PR #42 lands, before (or promptly after) this PR merges to main.
f17aecd to
1f037ea
Compare
Pin the AIPerf submodule to PR #42 and enable --allow-empty-content only for the MiniMax-M3 B300 TensorRT-LLM AgentX recipe. Signed-off-by: Cam Quilici <cjquilici@gmail.com>
1f037ea to
e874f66
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32305345287 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32310661720 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32310661720 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32310661720 |
|
Closing because the project scope is now limited to porting active, nondeprecated AMD multi-node configurations to srt-slurm. This PR is outside that scope. |
|
Reopening: this PR predates the recent out-of-scope configuration-porting detour and was closed by mistake during an overly broad cleanup. Its prior state is being restored. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ef9707f. Configure here.
| run_eval --port "$PORT" | ||
| else | ||
| build_replay_cmd "$RESULT_DIR" | ||
| REPLAY_CMD+=" --allow-empty-content" |
There was a problem hiding this comment.
AIPerf empty-content flag unpinned
High Severity
The recipe now passes --allow-empty-content to AIPerf, but this commit does not pin utils/aiperf to e8ece5d29bd0285af1ced36fbc4c670328b2d58c. install_agentic_deps installs the local submodule, and current AIPerf does not accept that flag, so the MiniMax TRT AgentX replay will fail at CLI parse before TTFT can be measured.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit ef9707f. Configure here.
1 participant
Summary
utils/aiperfto the exact head of SemiAnalysisAI/agentx-harness PR [NVIDIA] Update B200 SGL FP8 and add FP4 TP4 #42 (e8ece5d29bd0285af1ced36fbc4c670328b2d58c).--allow-empty-contentonly for the MiniMax-M3 NVFP4 B300 TensorRT-LLM AgentX recipe.Why
The existing MiniMax-M3 TRT-LLM configuration uses
stream_interval: 100. Its first generated reasoning delimiter is suppressed by the reasoning parser, so current AIPerf waits for later non-empty content before recording TTFT. This PR tests AIPerf PR #42 against that concrete serving path.Validation
bash -n benchmarks/single_node/agentic/minimaxm3_fp4_b300_trt_mtp.sh26 passedNote
Low Risk
Benchmark script and perf changelog only; no production serving or auth paths change.
Overview
This wires AIPerf empty-content TTFT handling into the MiniMax-M3 NVFP4 B300 TensorRT-LLM AgentX benchmark: after
build_replay_cmd, the replay path now appends--allow-empty-contentbeforerun_agentic_replay_and_write_outputs(throughput runs only; eval unchanged).perf-changelog.yamladds an agentic-coding entry forminimaxm3-fp4-b300-trtllm-agentic-mtpdescribing validation of AIPerf PR #42 on this recipe (including the aiperf pin and flag called out in the changelog text).Reviewed by Cursor Bugbot for commit ef9707f. Bugbot is set up for automated code reviews on this repo. Configure here.