feat(agentx): retune Kimi-K3 FP4 MI355X ATOM DSpark recipe on _0821 (mirror of #2723) - #2725
feat(agentx): retune Kimi-K3 FP4 MI355X ATOM DSpark recipe on _0821 (mirror of #2723)#2725seungrokj wants to merge 8 commits into
Conversation
Refresh the MI355X Kimi-K3 ATOM AgentX submission onto image kimi_k3_agentic_0821 and replace concurrency [1, 4, 8, 10] with [1, 2, 4, 8, 12, 16, 32, 40, 56], retuning every point. Concurrency 1-4 is the latency floor: GPU-resident, no decode context parallelism, 7 draft tokens at golden AL 3.84, 8192-token prefill step. From concurrency 8 up decode is KV-bandwidth-bound over 100k+ token agentic contexts, so decode-context-parallel-size 8 shards the KV read across all 8 GPUs and the LMCache DRAM tier backs the paged KV. Concurrency 8 and 12 run 3 draft tokens at golden AL 3.00 with ReplaySSM and 96 in-GPU state checkpoint slots; from concurrency 16 up the KDA recurrent state moves to ATOM's CPU state-offload tier instead. Concurrency 32 and up serve without a draft model. Both acceptance lengths come from the committed golden curve in golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml (7 -> 3.84, 3 -> 3.00). Evaluations drop the flag and use real acceptance. LMCACHE_MAX_LOCAL_CPU_SIZE and OFFLOAD_STATE_CPU_SIZE are per rank, so the aggregate TOTAL_CPU_DRAM_GB is divided by TP as the agentic README requires: dram-utilization 0.534 gives 200 GB/rank of paged KV at concurrency 8 and 12, and 0.598 gives 224 GB/rank from concurrency 16 up, split 192 GB paged KV plus 32 GB for the state tier. 将 MI355X 上 Kimi-K3 的 ATOM AgentX 提交刷新到 kimi_k3_agentic_0821 镜像， 并把并发点从 [1, 4, 8, 10] 换成 [1, 2, 4, 8, 12, 16, 32, 40, 56]，逐点重调。 并发 1-4 为时延下界：全部驻留 GPU，不开 decode context parallel，草稿 7 token 对应 golden AL 3.84，prefill 步长 8192。并发 8 起，10 万 token 以上的 agentic 上下文使 decode 受 KV 带宽约束，因此用 decode-context-parallel-size 8 把 KV 读取 分摊到 8 张卡，并由 LMCache DRAM 层承接分页 KV。并发 8 与 12 使用草稿 3 token （golden AL 3.00）、开启 ReplaySSM 并保留 96 个 GPU 内 state checkpoint 槽位； 并发 16 起，KDA 循环状态改由 ATOM 的 CPU state-offload 层承载。并发 32 及以上 不加载草稿模型。 两个接受长度均取自仓库内已提交的 golden 曲线（7 -> 3.84，3 -> 3.00）；评测 （eval）不传该参数，使用真实接受率。 LMCACHE_MAX_LOCAL_CPU_SIZE 与 OFFLOAD_STATE_CPU_SIZE 均为每 rank 设置，因此按 agentic README 的要求将聚合预算 TOTAL_CPU_DRAM_GB 除以 TP：dram-utilization 0.534 在并发 8/12 下给到每 rank 200 GB 分页 KV；0.598 在并发 16 起给到每 rank 224 GB，拆为 192 GB 分页 KV + 32 GB state 层。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…CL barrier Every rank pins its own LMCache pool, and at the measured sizes (200 GB/rank at concurrency 8 and 12, 192 GB/rank above that) the eight ranks finished pinning more than 600 s apart on the CI nodes. 600 s is exactly PyTorch's hardcoded kProcessGroupNCCLDefaultTimeout, so the ranks that reached the barrier at the end of allocate_kv_cache() first timed out waiting for rank 0 to publish the ncclUniqueId, and their ModelRunner processes died before the server ever served a request. Every concurrency point with KV offload hung; the three GPU-resident points finished normally. Halve the paged-KV half of the budget: dram-utilization 0.534 -> 0.268 (100 GB/rank) and 0.598 -> 0.343 (128 GB/rank, split 96 GB paged KV plus the unchanged 32 GB state tier). Aggregate pinned memory drops from 1600 to 800 GB at concurrency 8 and 12, and from 1792 to 1024 GB above that. The smaller pool costs little: the CPU tier's measured hit rate is 0.0% at concurrency 8 and 12 and 0.3-2.4% above that, and 96 GB/rank still holds roughly 53M tokens, about 530 full 100k-token contexts. 每个 rank 各自 pin 自己的 LMCache 池。按实测尺寸（并发 8/12 每 rank 200 GB， 以上每 rank 192 GB），八个 rank 在 CI 节点上完成 pin 的时间相差超过 600 秒， 而 600 秒正是 PyTorch 写死的 kProcessGroupNCCLDefaultTimeout。先到达 allocate_kv_cache() 末尾那个 barrier 的 rank 等不到 rank 0 发布 ncclUniqueId 便超时，ModelRunner 进程随之死亡，服务从未处理过任何请求。所有开启 KV offload 的并发点都挂起，三个全 GPU 驻留的点则正常跑完。 将预算中分页 KV 的部分减半：dram-utilization 0.534 -> 0.268（每 rank 100 GB）， 0.598 -> 0.343（每 rank 128 GB，拆为 96 GB 分页 KV 加上保持不变的 32 GB state 层）。固定内存总量在并发 8/12 从 1600 GB 降到 800 GB，以上从 1792 GB 降到 1024 GB。 缩小池子的代价很小：CPU 层实测命中率在并发 8/12 为 0.0%，更高并发为 0.3%-2.4%；每 rank 96 GB 仍可容纳约 5300 万 token，约合 530 个 10 万 token 的完整上下文。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d11b64f. Configure here.
| - { tp: 8, dcp-size: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.4.5" }, conc-list: [4, 8, 12], spec-decoding: mtp } | ||
| - dram-utilization: 0.343 | ||
| search-space: | ||
| - { tp: 8, dcp-size: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.4.5" }, conc-list: [32, 40, 56], spec-decoding: mtp } |
There was a problem hiding this comment.
Missing concurrency 16 point
High Severity
Concurrency 16 was removed from the 0.343 DRAM search space. The PR targets the published list including 16, the config comments still describe state offload for concurrency 16 and up, and the launcher already has a dedicated CONC=16 recipe, so that retuned point never runs.
Reviewed by Cursor Bugbot for commit d11b64f. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32857385954 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32857385954 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32912909255 |
|
/reuse-sweep-run |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/semiAnalysisAI/InferenceX/actions/runs/32912909255
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/semiAnalysisAI/InferenceX/actions/runs/32912909255
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook: https://github.com/ROCm/ATOM/blob/main/recipes/Kimi-K3.md
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
Signed: billishyahao
❌❌❌ REJECTED ❌❌❌@billishyahao — the sign-off was filled from a stale copy of the checklist: it is missing the ✅ Check 0 (CODEOWNER): PASS — |
billishyahao
left a comment
billishyahao
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/semiAnalysisAI/InferenceX/actions/runs/32912909255
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/semiAnalysisAI/InferenceX/actions/runs/32912909255
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook: https://github.com/ROCm/ATOM/blob/main/recipes/Kimi-K3.md
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
Signed: billishyahao
✅✅✅ Verdict: PASS ✅✅✅Re-verification of the re-posted sign-off (review). This supersedes the earlier rejection on this same head SHA, which was against the previous stale-template sign-off. ✅ Check 0 (CODEOWNER): PASS — |
|
@cquil11 could you review it? |
4 participants
Mirror of #2723 (branch
amd/agentx_kimik3_atom_0821by @zejunchen-zejun), created on branchamd/agentx_kimik3_atom_0821_zejunat the same head commit1adf729df.Refresh the MI355X Kimi-K3 ATOM AgentX submission onto image kimi_k3_agentic_0821 and replace concurrency [1, 4, 8, 10] with [1, 2, 4, 8, 12, 16, 32, 40, 56], retuning every point.
Concurrency 1-4 is the latency floor: GPU-resident, no decode context parallelism, 7 draft tokens at golden AL 3.84, 8192-token prefill step. From concurrency 8 up decode is KV-bandwidth-bound over 100k+ token agentic contexts, so decode-context-parallel-size 8 shards the KV read across all 8 GPUs and the LMCache DRAM tier backs the paged KV. Concurrency 8 and 12 run 3 draft tokens at golden AL 3.00 with ReplaySSM and 96 in-GPU state checkpoint slots; from concurrency 16 up the KDA recurrent state moves to ATOM's CPU state-offload tier instead. Concurrency 32 and up serve without a draft model.
Both acceptance lengths come from the committed golden curve in golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml (7 -> 3.84, 3 -> 3.00). Evaluations drop the flag and use real acceptance.
LMCACHE_MAX_LOCAL_CPU_SIZE and OFFLOAD_STATE_CPU_SIZE are per rank, so the aggregate TOTAL_CPU_DRAM_GB is divided by TP as the agentic README requires: dram-utilization 0.534 gives 200 GB/rank of paged KV at concurrency 8 and 12, and 0.598 gives 224 GB/rank from concurrency 16 up, split 192 GB paged KV plus 32 GB for the state tier.
Note
Low Risk
Benchmark recipe, container image pin, and published concurrency matrix only; no application runtime or auth changes.
Overview
Refreshes the kimik3-fp4-mi355x-atom-agentic-mtp AgentX submission onto ATOM image
kimi_k3_agentic_0821and replaces concurrency [1, 4, 8, 10] with [1, 2, 4, 8, 12, 16, 32, 40, 56], with per-concurrency retuning inkimik3_fp4_mi355x_atom_mtp.shand matching matrix entries inconfigs/amd-master.yaml.From concurrency 8 up, the recipe enables decode-context-parallel-size 8 (
DCP_SIZE), LMCache DRAM paged-KV offload, and tighter batching; 1–4 stay GPU-resident without DCP. DSpark acceptance follows the golden curve (7 draft / AL 3.84 at low conc, 3 draft / AL 3.00 through 16); 32+ runs without a draft model. ReplaySSM and in-GPU state checkpoints apply at 8 and 12; from 16 up, 32 GiB/rank CPU state offload is carved out of the DRAM budget while paged KV uses the remainder.LMCache setup changes include chunk size 1024 (aligned with DCP×block-size), per-rank split of aggregate DRAM between
LMCACHE_MAX_LOCAL_CPU_SIZEandOFFLOAD_STATE_CPU_SIZE, and smaller paged-KV pools (dram-utilization 0.268 / 0.343) to avoid NCCL timeout during cross-rank CPU pinning.ATOM_MLA_MAX_SPLIT_PER_BATCHis dropped.perf-changelog.yamldocuments the refresh.Reviewed by Cursor Bugbot for commit 8fcfc62. Bugbot is set up for automated code reviews on this repo. Configure here.