Tune B200 DSV4 SGLang AgentX prefill scheduling / 调整 B200 DSV4 SGLang AgentX 预填充调度 - #2718
Tune B200 DSV4 SGLang AgentX prefill scheduling / 调整 B200 DSV4 SGLang AgentX 预填充调度#2718nvpohanh wants to merge 14 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 206cb37. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32720253003 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32720253003 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32924641821 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32930559234 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32958611057 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32958611057 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33043221347 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33045888662 |
f8cdaf4 to
2151f50
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33087599126 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33087599126 |
2151f50 to
24918be
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33134754389 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33144525538 |
调整 B200 DSV4 AgentX 预填充调度。
更新性能变更日志中的 PR 链接。
修正性能变更日志中的 PR 链接回填。
更新 B200 DSV4 AgentX 高并发性能曲线配置。
简化 B200 DSV4 AgentX 启动参数分支。
统一使用 0.90 的静态显存比例，为 B200 FP4 indexer 保留瞬时工作区。
将 c96 纳入较小的分块预填充预算和紧凑 CUDA Graph 配置，以避免 FP4 索引器工作区内存不足。
将 TP8 HiCache 比例降至 2.75，同时保留 DEP8 比例 8，避免 NScale 1.7 TiB 主机内存上的 V4 分页池初始化失败。\n\n改用支持小数的比例上限检查，并在性能变更日志中记录实测的主机内存限制。
将 B200 SGLang 并发 160 的 mem-fraction-static 从 0.90 降至 0.88，为 DeepSeek V4 FP4 indexer 的上下文相关工作区释放约 3.57 GiB HBM。\n\n保留其他并发点和 HiCache 比例不变，并更新现有性能变更日志说明。
将所有 B200 DP-attention 配置统一为每 rank 6144 token 的 chunked prefill 和 CUDA graph 最大 batch 32。\n\n移除并发 96/128/160 的重复条件分支，并同步更新现有性能变更日志。
在全部 B200 DEP8 并发点统一启用 total_requests、每秒负载快照和基于 correlation ID 的会话路由，并保留并发 160 的专用显存余量设置。
将 B200 DSV4 SGLang AgentX 配置更新到 nightly-dev-20260827-20621aa1，并同步记录性能变更日志。
将 B200 DSV4 SGLang AgentX 镜像回退到 nightly-dev-20260821-f825d729，以验证 c128 OOM 是否由较新的 nightly 引入。
将所有 B200 DSV4 DP-attention 配置的 mem-fraction-static 统一降至 0.88，并恢复 nightly-dev-20260827-20621aa1，以为 FP4 indexer 的动态工作区预留显存。
76398d3 to
30f2fb4
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33154296172 |
1 participant
[by Codex]
Refresh the B200 DeepSeek V4 SGLang AgentX submission with validated high-concurrency HiCache profiles:
--prefill-decode-intervalto DEP8 profiles only; use interval 24 for concurrency 64, 96, and 128.--chunked-prefill-size 49152,--cuda-graph-max-bs 32, and--load-balance-method total_requests.--mem-fraction-static 0.90across every TP and DP profile. The official c128 sweep showed that 0.91 left insufficient transient workspace for the FP4 indexer.Full-hour validation reached 34.587 tok/s/user and 43,382.65 tok/s/GPU at concurrency 128, and 28.916 tok/s/user and 50,468.23 tok/s/GPU at concurrency 160. No SGLang or AIPerf source changes are required.
使用经过验证的高并发 HiCache 配置刷新 B200 DeepSeek V4 SGLang AgentX 提交：
--prefill-decode-interval；并发 64、96 和 128 使用 interval 24。--chunked-prefill-size 49152、--cuda-graph-max-bs 32和--load-balance-method total_requests。--mem-fraction-static 0.90。官方 c128 sweep 表明，0.91 无法为 FP4 indexer 保留足够的瞬时工作区。完整一小时验证中，并发 128 达到 34.587 tok/s/user 和 43,382.65 tok/s/GPU，并发 160 达到 28.916 tok/s/user 和 50,468.23 tok/s/GPU。不需要修改 SGLang 或 AIPerf 源代码。
Note
Low Risk
Benchmark and YAML sweep changes only; no application runtime or security-sensitive logic.
Overview
Refreshes the B200 DeepSeek V4 SGLang AgentX + HiCache + MTP recipe and master config after validated high-concurrency runs on NScale.
The launch script now sizes HiCache by topology: host/device ratio 2.75 for replicated TP8 (fits ~121 GiB/rank on 1.7 TiB hosts) and 8 for sharded DP8, with float-safe ratio cap checks. DP-attention (DEP8) paths drop
--enable-prefill-delayer, use--prefill-decode-interval24 (20 at concurrency 160),mem-fraction-static0.88 (0.90 for non-DP), a 6144 tokens/rank chunked-prefill budget, CUDA graph max batch 32,total_requestsDP load balancing, one-second load snapshots, and correlation-ID session routing. The SGLang router switches tocache_awarepolicy with an optional balance threshold at conc 160. Weight load adds multithreaded model-loader config.nvidia-masterbumps the SGLang image tonightly-dev-20260827-20621aa1, trims TP HiCache concurrency points, and extends the DP HiCache sweep to 64–160 with the sglang-router.perf-changelog.yamldocuments the curve refresh.Reviewed by Cursor Bugbot for commit 30f2fb4. Bugbot is set up for automated code reviews on this repo. Configure here.