Skip to content

Latest commit

 

History

History
73 lines (51 loc) · 9.74 KB

appendix-b.md

File metadata and controls

73 lines (51 loc) · 9.74 KB

Appendix B: Inference Hardware-Software Co-Design and References


Inference hardware-software co-design checklist

Evaluate model architecture and serving configuration jointly at your target prompt lengths, output lengths, batch concurrency, and latency SLAs. Compare candidate designs on quality and goodput (completed requests per second meeting both TTFT and TPOT targets on an identical chip allocation). The table below summarizes supporting co-design checks across the serving stack:

Co-design area Guidance and mechanism What to measure
Attention, KV format, and kernel layout Pair head_dim = 256 with proportionally fewer Q and KV heads, preserving total projection widths, logical KV bytes, and iso-quality. Inspect physical KV memory allocation after sharding, paging, and padding. Tune KV page sizes (block-size = 256) and attention compute blocks separately; prefer fused KV updates and phase-matched kernels. Ensure quantized KV saves physical HBM bytes while holding long-context quality (TPU Ragged Paged Attention). Physical allocated KV bytes/token, attention and KV-update kernel time, tile padding waste, layout conversions, long-context task quality.
Attention parallelism vs. expert parallelism Select attention TP/DP independently from expert EP. A large routed expert pool does not require equally wide attention TP; account for KV head replication when TP > num_kv_heads. Compare wider sharding against running more independent replicas at a fixed chip budget, since replica count governs the per-shard batch available to amortize weight reads. The Qwen3.5-397B on Ironwood playbook pairs attention DP with expert EP to avoid duplicating KV heads. Per-replica concurrency, KV replication factor, collective latency, net goodput/chip, P99 TPOT.
MoE batch occupancy and load balance Size expert width, count, and active top-$K$ for real decode batch sizes. Per-token active parameters do not capture the union of expert weights streamed across a batch. Profile expert routing skew alongside mean tokens/expert, since the busiest rank sets step latency. Rebalance expert placement or replicate hot experts in HBM without altering router math (DeepSeek-V3). Distinct experts touched per step, tokens/expert skew, busiest-rank latency, dispatch/combine and weight-read bytes.
Batching and prefill/decode placement Prewarm a compact set of compiled token and batch-slot buckets. Tune continuous batching and prefill chunk size jointly so long prefills do not stall ongoing decode while avoiding tiny compute-starved chunks. Benchmark colocated chunked prefill (Sarathi-Serve) against disaggregated prefill/decode pools (DistServe) with separately tuned pool sizes and shardings, including KV transfer, layout conversion, and queueing delays. TTFT and TPOT percentiles (P50/P99), queue wait time, KV handoff bytes and latency, prefill/decode pool utilization, net goodput/chip.
Prefix reuse and locality-aware routing Reuse exact matching prefix KV blocks where supported (vLLM automatic prefix caching) and route related requests using both KV cache locality and queue load (llm-d routing) so queueing hotspots do not erase saved prefill compute. Note that prefix caching skips repeated prefill FLOPs, not decode KV reads over that cached context. Reused prefix token fraction, end-to-end TTFT including queueing, cache eviction/recompute rate, active KV occupancy, rank load skew.
MTP / speculative decoding Tune draft length across the complete draft, verify, and KV commit/rollback cycle at target concurrency (Principle 8.2). Acceptance rate alone is insufficient once drafter latency, verification FLOPs, MoE touched-expert expansion, and rollback overhead are included. Validate rejection-sampling correctness so the output distribution matches the target model (speculative decoding). Mean accepted prefix length, net committed tokens/s, full cycle latency, sampler distribution parity, net goodput/chip.

Two useful serving budget checks

  1. Logical KV cache footprint: For standard multi-head / grouped-query attention with equal K and V head dimensions and element byte size $s_{\text{KV}}$, logical KV memory across active requests $r$ and attention layers $l$ is:

    $$M_{\text{KV}} = 2 \cdot s_{\text{KV}} \sum_{r, l} S_{r,l} \cdot n_{\text{KV},l} \cdot d_{\text{head},l}$$

    where $S_{r,l}$ is the cached token length for request $r$ at layer $l$. Halving $n_{\text{KV},l}$ when doubling $d_{\text{head},l}: 128 \to 256$ preserves logical $M_{\text{KV}}$ identically. Physical HBM allocation additionally includes page fragmentation, padding, quantization scale metadata, and any TP replication; latent-compressed KV (MLA) and recurrent linear-attention states require their own state accounting.

  2. Expected MoE batch expert occupancy and LatentMoE dispatch parity: Under uniform independent routing across $E$ experts with $K$ distinct experts per token and $B$ concurrent decode tokens, the expected number of distinct experts touched in a single MoE layer is:

    $$E_{\text{touched}}(B) = E \cdot \left[1 - \left(1 - \frac{K}{E}\right)^B\right]$$

    Use this expression only as a back-of-the-envelope sanity check, and rely on profiled expert occupancy and HBM weight reads on real traffic. Separately, LatentMoE's save-and-reinvest condition:

    $$K_{\text{new}} \cdot \ell \le K_{\text{old}} \cdot d_{\text{model}}$$

    guarantees that routed activation dispatch volume does not increase at equal precision; expert weight-read bytes and MXU compute FLOPs remain distinct budgets to check alongside it. Note that the iso-quality head_dim = 256 results in Principle 1 hold total projection widths constant via proportional head-count reduction; the fixed-head-count head_dim sweep in the TPU Ragged Paged Attention paper evaluates a different scaling experiment.


References

Cloud TPU, JAX, and reference stacks

Frontier model architecture reports

Scaling laws, serving systems, and profiling tools