InfiniTensor · goog00 · Jul 6, 2026
diff --git a/bench/qwen3_moe/README.md b/bench/qwen3_moe/README.md
@@ -0,0 +1,111 @@
+# Qwen3-MoE 并发吞吐基准（MetaX）
+
+T2-1-3 的服务吞吐基准，对齐赛题「测试风格参考长文本优化赛题（T2-1-2）」的要求：
+起 InfiniLM 服务 + `vllm bench serve` 压测 **concurrency × input-len × output-len** 矩阵，
+产出 **base vs this** 的输出吞吐 / 总吞吐对比（赛题「性能」维度的打分依据）。
+
+平台：沐曦 MetaX C500，TP=2。
+
+## 文件
+
+| 文件 | 作用 |
+|---|---|
+| `serve.sh` | 起 OpenAI 兼容服务（Qwen3-30B-A3B-Thinking，metax，TP=2，paged-attn + flash-attn + ignore-eos）|
+| `run_bench.sh` | 客户端 `vllm bench serve` 遍历矩阵（**需装 vllm**）|
+| `load_client.py` | **零依赖**并发压测客户端（纯标准库，无需 vllm/aiohttp）——vllm 装不上时用这个 |
+| `summarize.py` | 解析结果文件成表格；给两个目录则输出 base-vs-this 的 Δ% |
+
+> 两个客户端产出**同一格式**的结果文件（`bench_results/<TAG>/bsX_inY_outZ.txt`），
+> summarize.py 都能解析。**MetaX 容器一般没有 vllm**（且与 maca torch 易冲突），推荐用 `load_client.py`。
+
+## 服务依赖
+
+服务端只需轻量依赖：
+
+```bash
+pip install fastapi uvicorn
+```
+
+## 用 load_client.py（推荐，无 vllm）
+
+```bash
+# 终端 A：起服务（见下）
+# 终端 B：
+python3 bench/qwen3_moe/load_client.py --tag this      # 或 --tag base
+python3 bench/qwen3_moe/summarize.py bench_results/this
+```
+
+矩阵/并发/模型名均可用 `--batch-sizes/--input-lens/--output-lens/--model/--base-url` 覆盖。
+`load_client.py` 用非流式请求测**输出吞吐 / 总吞吐**（赛题打分项）；不测 TTFT/TPOT（需流式，表里显示 `-`）。
+输入长度用 CJK 填充字近似（~1 token/字）；要精确可加 `--tokenizer <模型路径>`（需 transformers）。
+
+## 默认压测矩阵
+
+- concurrency：`1 8 32`
+- input-len：`32 256 2048`
+- output-len：`256 1024`
+
+（均可用环境变量覆盖，见各脚本头部注释。）
+
+## base vs this 怎么跑
+
+「this」= T2-1-3 分支（grouped_gemm 批化 MoE）。「base」= 改造前的**朴素逐 token×逐 expert** MoE
+（`main` 分支 / grouped_gemm 之前的实现）。两者用**同一套客户端脚本**分别压测，只是换部署的 `.so`。
+
+### 1) 跑 this（当前 T2-1-3）
+
+```bash
+# 终端 A：起服务（保持前台）
+cd /data/InfiniLM
+MODEL=/data/huggingface_home/Qwen3-30B-A3B-Thinking-2507 \
+  bash bench/qwen3_moe/serve.sh
+# 等打印 “load weights over” + uvicorn 起监听
+
+# 终端 B：等服务 ready 后压测
+cd /data/InfiniLM
+curl -s http://127.0.0.1:8102/health && echo " <- server ready"
+TAG=this bash bench/qwen3_moe/run_bench.sh
+```
+
+### 2) 跑 base（朴素 MoE 基线）
+
+```bash
+# 切到基线实现并重建 + 部署 .so（示例，按你的基线 commit 调整）
+cd /data/InfiniLM && git stash && git checkout main
+cd /data/InfiniCore && xmake install
+cd /data/InfiniLM && xmake build _infinilm
+cp build/linux/x86_64/release/_infinilm.cpython-310-x86_64-linux-gnu.so python/infinilm/lib/
+
+# 起服务 + 压测（同上，改 TAG=base）
+MODEL=/data/huggingface_home/Qwen3-30B-A3B-Thinking-2507 bash bench/qwen3_moe/serve.sh   # 终端 A
+TAG=base bash bench/qwen3_moe/run_bench.sh                                                # 终端 B
+```
+
+> 注：base 与 this 必须用**完全相同的客户端参数**（矩阵、seed 策略、ignore-eos）才可比。
+> 脚本已固定这些；两次只改 `TAG` 和部署的 `.so`。
+
+### 3) 汇总对比
+
+```bash
+python3 bench/qwen3_moe/summarize.py bench_results/base bench_results/this
+```
+
+输出每个 (bs,in,out) 的 `out_tok/s` 与 `total_tok/s` 的 base→this 及 Δ%。
+只看单侧：`python3 bench/qwen3_moe/summarize.py bench_results/this`。
+
+## 显存 / 参数提示
+
+- `NUM_BLOCKS` 默认 1024（≈ 26 万 token 的 KV，覆盖 concurrency 32 × (2048+1024) 有余）。
+  **OOM 就调小**（`NUM_BLOCKS=512`），**报 KV 不足就调大**。
+- `MAX_CACHE`（--max-cache-len）需 ≥ 最大 input+output（默认 4096 覆盖 2048+1024）。
+- **注意力后端 `ATTN` 默认 `paged-attn`**：MetaX build **未编 flash-attn**（用 `--attn flash-attn`
+  会报 "FlashAttention is not enabled in this build" 并崩 forward）。若 paged-attn 在你的 build 也不可用，
+  退回 static cache：`PAGED=0 ATTN=default bash bench/qwen3_moe/serve.sh`（并发批处理能力会下降）。
+- `--enable-graph` 默认**关**：MoE 前向含数据相关的 host dispatch + syncStream，graph capture 可能不兼容。
+  想试开：`GRAPH=1 bash bench/qwen3_moe/serve.sh`，若起不来或结果异常就关掉。
+- 采样固定 `top-k 1`（greedy）+ `ignore-eos`，保证每请求生成满 output-len、结果可比。
+
+## 关注指标
+
+赛题打分看 **Output token throughput (tok/s)** 与 **Total Token throughput (tok/s)**；
+小规模（bs=1）作为「不回退」门槛。summarize.py 已抽取这两项 + req/s + TTFT + TPOT。
diff --git a/bench/qwen3_moe/load_client.py b/bench/qwen3_moe/load_client.py
@@ -0,0 +1,176 @@
+#!/usr/bin/env python3
+"""Dependency-free concurrent load client for the InfiniLM OpenAI server.
+
+A stdlib-only alternative to `vllm bench serve` (no vllm / aiohttp needed), for
+environments where vllm is not installed (e.g. MetaX containers). Sweeps the
+concurrency × input-len × output-len matrix and writes one result file per
+config in the exact label format `summarize.py` parses.
+
+Because the server runs with `--ignore-eos`, each request generates exactly
+`output_len` tokens, so output-token throughput is measured against the nominal
+count (also reconciled with the response `usage` field when present).
+
+Usage:
+    python3 load_client.py --tag this
+    python3 load_client.py --tag base --base-url http://127.0.0.1:8102 \
+        --batch-sizes 1,8,32 --input-lens 32,256,2048 --output-lens 256,1024
+
+Input length is approximated by repeating a CJK filler char (~1 token each);
+this is fine for relative base-vs-this comparison. Pass --tokenizer to count
+prompt tokens exactly if `transformers` is available.
+"""
+import argparse
+import json
+import os
+import time
+import urllib.request
+from concurrent.futures import ThreadPoolExecutor, as_completed
+
+FILLER = "的"  # ~1 Qwen token each; used to build an approximately input_len prompt.
+
+
+def build_prompt(input_len, tokenizer=None):
+    """Return a user message string of approximately `input_len` tokens."""
+    if tokenizer is not None:
+        # Grow the filler until the tokenizer reports >= input_len tokens.
+        s = FILLER * input_len
+        while len(tokenizer(s)["input_ids"]) < input_len:
+            s += FILLER * 8
+        return s
+    return FILLER * input_len
+
+
+def one_request(base_url, model, prompt, output_len, timeout):
+    """Send one non-streaming chat request; return (ok, latency_s, completion_tokens)."""
+    payload = json.dumps({
+        "model": model,
+        "messages": [{"role": "user", "content": prompt}],
+        "max_tokens": output_len,
+        "ignore_eos": True,
+        "stream": False,
+        "temperature": 1.0,
+    }).encode("utf-8")
+    req = urllib.request.Request(
+        base_url.rstrip("/") + "/v1/chat/completions",
+        data=payload, headers={"Content-Type": "application/json"}, method="POST")
+    t0 = time.perf_counter()
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
+            body = json.loads(resp.read().decode("utf-8"))
+    except Exception as e:  # noqa: BLE001 - report and keep going
+        return False, time.perf_counter() - t0, 0, str(e)
+    lat = time.perf_counter() - t0
+    # Prefer server-reported usage; fall back to nominal output_len.
+    comp = output_len
+    usage = body.get("usage") or {}
+    if isinstance(usage.get("completion_tokens"), int) and usage["completion_tokens"] > 0:
+        comp = usage["completion_tokens"]
+    return True, lat, comp, None
+
+
+def run_config(base_url, model, bs, il, ol, repeat, timeout, tokenizer):
+    num_prompts = bs * repeat
+    prompt = build_prompt(il, tokenizer)
+    results = []
+    err_sample = None
+
+    t_start = time.perf_counter()
+    with ThreadPoolExecutor(max_workers=bs) as pool:
+        futs = [pool.submit(one_request, base_url, model, prompt, ol, timeout)
+                for _ in range(num_prompts)]
+        for f in as_completed(futs):
+            ok, lat, comp, err = f.result()
+            results.append((ok, lat, comp))
+            if not ok and err_sample is None:
+                err_sample = err
+    wall = time.perf_counter() - t_start
+
+    ok_n = sum(1 for ok, _, _ in results if ok)
+    out_tokens = sum(c for ok, _, c in results if ok)
+    in_tokens = ok_n * il  # approximate (see build_prompt)
+    lat_ok = [lat for ok, lat, _ in results if ok]
+    mean_lat_ms = 1000.0 * sum(lat_ok) / len(lat_ok) if lat_ok else 0.0
+
+    metrics = {
+        "successful": ok_n,
+        "num_prompts": num_prompts,
+        "duration_s": wall,
+        "req_s": ok_n / wall if wall > 0 else 0.0,
+        "out_tok_s": out_tokens / wall if wall > 0 else 0.0,
+        "total_tok_s": (in_tokens + out_tokens) / wall if wall > 0 else 0.0,
+        "mean_latency_ms": mean_lat_ms,
+        "err_sample": err_sample,
+    }
+    return metrics
+
+
+def write_result(path, bs, il, ol, m):
+    # Labels chosen to match summarize.py's regexes.
+    lines = [
+        "============ Serving Benchmark Result ============",
+        f"config:                                  bs={bs} in={il} out={ol}",
+        f"Successful requests:                     {m['successful']}/{m['num_prompts']}",
+        f"Benchmark duration (s):                  {m['duration_s']:.2f}",
+        f"Request throughput (req/s):              {m['req_s']:.4f}",
+        f"Output token throughput (tok/s):         {m['out_tok_s']:.2f}",
+        f"Total Token throughput (tok/s):          {m['total_tok_s']:.2f}",
+        f"Mean E2E latency (ms):                   {m['mean_latency_ms']:.2f}",
+    ]
+    if m["err_sample"]:
+        lines.append(f"error_sample:                            {m['err_sample']}")
+    open(path, "w", encoding="utf-8").write("\n".join(lines) + "\n")
+
+
+def parse_list(s):
+    return [int(x) for x in s.replace(",", " ").split()]
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--base-url", default="http://127.0.0.1:8102")
+    ap.add_argument("--model", default="Qwen3-30B-A3B-Thinking-2507")
+    ap.add_argument("--tag", default="this", help="result subdir under --outdir-root")
+    ap.add_argument("--outdir-root", default="bench_results")
+    ap.add_argument("--batch-sizes", type=parse_list, default=[1, 8, 32])
+    ap.add_argument("--input-lens", type=parse_list, default=[32, 256, 2048])
+    ap.add_argument("--output-lens", type=parse_list, default=[256, 1024])
+    ap.add_argument("--repeat", type=int, default=3,
+                    help="requests per config = concurrency * repeat")
+    ap.add_argument("--timeout", type=float, default=1200.0)
+    ap.add_argument("--tokenizer", default=None,
+                    help="optional HF tokenizer path for exact prompt token counts")
+    args = ap.parse_args()
+
+    tok = None
+    if args.tokenizer:
+        try:
+            from transformers import AutoTokenizer
+            tok = AutoTokenizer.from_pretrained(args.tokenizer, trust_remote_code=True)
+        except Exception as e:  # noqa: BLE001
+            print(f"(tokenizer load failed, using approximate prompt: {e})")
+
+    outdir = os.path.join(args.outdir_root, args.tag)
+    os.makedirs(outdir, exist_ok=True)
+    print(f">>> tag={args.tag} url={args.base_url} model={args.model} -> {outdir}")
+    print(f">>> concurrency={args.batch_sizes} input={args.input_lens} output={args.output_lens}")
+
+    for bs in args.batch_sizes:
+        for il in args.input_lens:
+            for ol in args.output_lens:
+                path = os.path.join(outdir, f"bs{bs}_in{il}_out{ol}.txt")
+                print(f">>> bs={bs} in={il} out={ol} ...", end="", flush=True)
+                m = run_config(args.base_url, args.model, bs, il, ol,
+                               args.repeat, args.timeout, tok)
+                write_result(path, bs, il, ol, m)
+                if m["successful"] == 0:
+                    print(f" FAIL ({m['err_sample']})")
+                else:
+                    print(f" ok  out={m['out_tok_s']:.1f} tok/s  total={m['total_tok_s']:.1f} tok/s"
+                          f"  ({m['successful']}/{m['num_prompts']})")
+
+    print(f">>> done. 汇总: python3 {os.path.join(os.path.dirname(__file__), 'summarize.py')} {outdir}")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/bench/qwen3_moe/run_bench.sh b/bench/qwen3_moe/run_bench.sh
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+# ---------------------------------------------------------------------------
+# 客户端并发吞吐压测（vllm bench serve），对齐 T2-1-2 的服务测法。
+# 遍历 concurrency × input-len × output-len 矩阵，每组结果落一个文件。
+#
+# 前置：serve.sh 已在同机 PORT 上把服务起好。
+#
+# 用法：
+#   TAG=base ./run_bench.sh        # 跑基线版（如 main 的 naive MoE），结果存 bench_results/base
+#   TAG=this ./run_bench.sh        # 跑本方案（T2-1-3 grouped_gemm），结果存 bench_results/this
+#
+# 环境变量：
+#   TAG          结果子目录名（base / this），用于 base-vs-this 对比
+#   PORT         服务端口（默认 8102）
+#   MODEL_NAME   请求里的 model 字段（服务单模型，一般被忽略；默认见下）
+#   TOKENIZER    分词器路径，用于准确统计 token 数
+#   BATCH_SIZES  concurrency 列表（默认 "1 8 32"）
+#   INPUT_LENS   输入长度列表（默认 "32 256 2048"）
+#   OUTPUT_LENS  输出长度列表（默认 "256 1024"）
+#   REPEAT       每组的请求数 = concurrency × REPEAT（默认 3）
+# ---------------------------------------------------------------------------
+set -euo pipefail
+
+TAG="${TAG:-this}"
+PORT="${PORT:-8102}"
+MODEL_NAME="${MODEL_NAME:-Qwen3-30B-A3B-Thinking-2507}"
+TOKENIZER="${TOKENIZER:-/data/huggingface_home/Qwen3-30B-A3B-Thinking-2507}"
+REPEAT="${REPEAT:-3}"
+
+read -r -a BATCH_SIZES <<< "${BATCH_SIZES:-1 8 32}"
+read -r -a INPUT_LENS  <<< "${INPUT_LENS:-32 256 2048}"
+read -r -a OUTPUT_LENS <<< "${OUTPUT_LENS:-256 1024}"
+
+OUTDIR="${OUTDIR:-bench_results/${TAG}}"
+mkdir -p "${OUTDIR}"
+unset http_proxy https_proxy all_proxy ALL_PROXY 2>/dev/null || true
+
+echo ">>> TAG=${TAG} port=${PORT} -> ${OUTDIR}"
+echo ">>> concurrency=[${BATCH_SIZES[*]}] input=[${INPUT_LENS[*]}] output=[${OUTPUT_LENS[*]}]"
+
+for bs in "${BATCH_SIZES[@]}"; do
+  for il in "${INPUT_LENS[@]}"; do
+    for ol in "${OUTPUT_LENS[@]}"; do
+      sleep 1
+      seed=$(date +%s)
+      num_prompts=$(( bs * REPEAT ))
+      out="${OUTDIR}/bs${bs}_in${il}_out${ol}.txt"
+      echo ">>> bs=${bs} in=${il} out=${ol} num_prompts=${num_prompts} -> ${out}"
+
+      # 用 --extra-body 把 max_tokens/ignore_eos 塞进 chat 请求，保证每请求生成满 output-len，
+      # 吞吐才可比（与 T2-1-2 一致）。--request-rate inf = 爆发模式。
+      vllm bench serve \
+        --backend openai-chat \
+        --model "${MODEL_NAME}" \
+        --tokenizer "${TOKENIZER}" \
+        --endpoint /v1/chat/completions \
+        --port "${PORT}" \
+        --request-rate inf \
+        --seed "${seed}" \
+        --num-prompts "${num_prompts}" \
+        --max-concurrency "${bs}" \
+        --random-input-len "${il}" \
+        --extra-body "{\"max_tokens\": ${ol}, \"ignore_eos\": true}" \
+        > "${out}" 2>&1 || { echo "!! 该组失败，见 ${out}"; continue; }
+    done
+  done
+done
+
+echo ">>> done. 汇总： python3 $(dirname "$0")/summarize.py ${OUTDIR}"