feat(agentx): retune Kimi-K3 FP4 MI355X ATOM DSpark recipe on _0821 - #2716
feat(agentx): retune Kimi-K3 FP4 MI355X ATOM DSpark recipe on _0821#2716zejunchen-zejun wants to merge 4 commits into
Conversation
aeb9f0b to
5826bdc
Compare
Refresh the MI355X Kimi-K3 ATOM AgentX submission onto image kimi_k3_agentic_0821 and replace concurrency [1, 4, 8, 10] with [1, 2, 4, 8, 12, 16, 32, 40, 56], retuning every point. Concurrency 1-4 is the latency floor: GPU-resident, no decode context parallelism, 7 draft tokens at golden AL 3.84, 8192-token prefill step. From concurrency 8 up decode is KV-bandwidth-bound over 100k+ token agentic contexts, so decode-context-parallel-size 8 shards the KV read across all 8 GPUs and the LMCache DRAM tier backs the paged KV. Concurrency 8 and 12 run 3 draft tokens at golden AL 3.00 with ReplaySSM and 96 in-GPU state checkpoint slots; from concurrency 16 up the KDA recurrent state moves to ATOM's CPU state-offload tier instead. Concurrency 32 and up serve without a draft model. Both acceptance lengths come from the committed golden curve in golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml (7 -> 3.84, 3 -> 3.00). Evaluations drop the flag and use real acceptance. LMCACHE_MAX_LOCAL_CPU_SIZE and OFFLOAD_STATE_CPU_SIZE are per rank, so the aggregate TOTAL_CPU_DRAM_GB is divided by TP as the agentic README requires: dram-utilization 0.534 gives 200 GB/rank of paged KV at concurrency 8 and 12, and 0.598 gives 224 GB/rank from concurrency 16 up, split 192 GB paged KV plus 32 GB for the state tier. 将 MI355X 上 Kimi-K3 的 ATOM AgentX 提交刷新到 kimi_k3_agentic_0821 镜像， 并把并发点从 [1, 4, 8, 10] 换成 [1, 2, 4, 8, 12, 16, 32, 40, 56]，逐点重调。 并发 1-4 为时延下界：全部驻留 GPU，不开 decode context parallel，草稿 7 token 对应 golden AL 3.84，prefill 步长 8192。并发 8 起，10 万 token 以上的 agentic 上下文使 decode 受 KV 带宽约束，因此用 decode-context-parallel-size 8 把 KV 读取 分摊到 8 张卡，并由 LMCache DRAM 层承接分页 KV。并发 8 与 12 使用草稿 3 token （golden AL 3.00）、开启 ReplaySSM 并保留 96 个 GPU 内 state checkpoint 槽位； 并发 16 起，KDA 循环状态改由 ATOM 的 CPU state-offload 层承载。并发 32 及以上 不加载草稿模型。 两个接受长度均取自仓库内已提交的 golden 曲线（7 -> 3.84，3 -> 3.00）；评测 （eval）不传该参数，使用真实接受率。 LMCACHE_MAX_LOCAL_CPU_SIZE 与 OFFLOAD_STATE_CPU_SIZE 均为每 rank 设置，因此按 agentic README 的要求将聚合预算 TOTAL_CPU_DRAM_GB 除以 TP：dram-utilization 0.534 在并发 8/12 下给到每 rank 200 GB 分页 KV；0.598 在并发 16 起给到每 rank 224 GB，拆为 192 GB 分页 KV + 32 GB state 层。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5826bdc to
c8cd524
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d17b64f. Configure here.
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
|
The new PR is here: #2723 |
2 participants
Refresh the MI355X Kimi-K3 ATOM AgentX submission onto image kimi_k3_agentic_0821 and replace concurrency [1, 4, 8, 10] with [1, 2, 4, 8, 12, 16, 32, 40, 56], retuning every point.
Concurrency 1-4 is the latency floor: GPU-resident, no decode context parallelism, 7 draft tokens at golden AL 3.84, 8192-token prefill step. From concurrency 8 up decode is KV-bandwidth-bound over 100k+ token agentic contexts, so decode-context-parallel-size 8 shards the KV read across all 8 GPUs and the LMCache DRAM tier backs the paged KV. Concurrency 8 and 12 run 3 draft tokens at golden AL 3.00 with ReplaySSM and 96 in-GPU state checkpoint slots; from concurrency 16 up the KDA recurrent state moves to ATOM's CPU state-offload tier instead. Concurrency 32 and up serve without a draft model.
Both acceptance lengths come from the committed golden curve in golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml (7 -> 3.84, 3 -> 3.00). Evaluations drop the flag and use real acceptance.
LMCACHE_MAX_LOCAL_CPU_SIZE and OFFLOAD_STATE_CPU_SIZE are per rank, so the aggregate TOTAL_CPU_DRAM_GB is divided by TP as the agentic README requires: dram-utilization 0.534 gives 200 GB/rank of paged KV at concurrency 8 and 12, and 0.598 gives 224 GB/rank from concurrency 16 up, split 192 GB paged KV plus 32 GB for the state tier.
Note
Low Risk
Changes are limited to benchmark recipes, YAML matrix entries, and changelog text; no application runtime or security-sensitive code paths.
Overview
Refreshes the kimik3-fp4-mi355x-atom-agentic-mtp AgentX submission on ATOM image
kimi_k3_agentic_0821and expands the published concurrency sweep from [1, 4, 8, 10] to [1, 2, 4, 8, 12, 16, 32, 40, 56], with per-point retuning inkimik3_fp4_mi355x_atom_mtp.shand matching search-space /dram-utilizationblocks inconfigs/amd-master.yaml.Concurrency tiers: Low conc (1–4) stays GPU-resident with DCP off, 7 DSpark draft tokens, and golden AL 3.84. From conc 8, the server adds
--decode-context-parallel-size8, LMCache DRAM KV offload, and 3 draft tokens at AL 3.00; 8/12 enable ReplaySSM and 96 in-GPU state-checkpoint slots. From 16, KDA recurrent state uses the CPU state-offload tier (32 GiB/rank carved from the per-rank DRAM budget); 32+ runs without speculative decoding.LMCache / hybrid offload: Per-rank CPU DRAM is split between
LMCACHE_MAX_LOCAL_CPU_SIZEandOFFLOAD_STATE_CPU_SIZE;LMCACHE_CHUNK_SIZEmoves to 1024 (aligned with DCP8 × 128-token blocks); addsOFFLOAD_KV_FOR_HYBRID, enablesOFFLOAD_PROFILE=1, and conditionally sets state-offload staging (OFFLOAD_GPU_STAGING_CHUNKS=32). DropsATOM_MLA_MAX_SPLIT_PER_BATCH.Documents the refresh in
perf-changelog.yaml.Reviewed by Cursor Bugbot for commit b0134e1. Bugbot is set up for automated code reviews on this repo. Configure here.