[2026春季][T2-1-1] CearX

[CearX]

Summary

为 InfiniLM 新增 ERNIE-4.5-VL-28B-A3B-Thinking 的文本、图像和视频端到端推理与 TP=4 支持，并通过 Hugging Face 逐 token 对齐、C-Eval、MMLU 和 MMMU 对比评测验证正确性。

• 新增 ERNIE-4.5-VL-28B-A3B-Thinking 的文本、图像和视频推理支持，包括 Vision Tower、spatial/temporal Resampler、二维/三维 RoPE、异构 MoE 和多模态 embedding 替换。
• 扩展 Python、pybind 和 C++ 输入路径，传递 token_type_ids、grid_thw 和 image_type_ids，并接入语言解码器 TP、模型加载和 processor。
• 新增三模态 exact-token 正确性测试，以及 C-Eval、MMLU、MMMU 评测和结果比较工具。

Motivation

2026 年春季启元 AI 大赛 T2-1-1 赛题： ERNIE-4.5-VL-28B-A3B-Thinking 适配。

Closes #

Type of Change

• feat — new feature / new model
• fix — bug fix
• perf — performance improvement (no behavioral change)
• refactor — code restructuring without behavior change
• test — adding or fixing tests only
• docs — documentation only
• build / ci — build system or CI configuration
• chore — tooling, formatting, or other non-code changes
• Breaking change

Test Results of Involved Models on Supported Platforms (Please attach screenshots)

Validation	Result	Environment and boundary
Final stored reference	text 16/16、image 8/8、video 8/8 逐 token 一致，ok=true	4 × RTX 4090 D，BF16，TP=4，static KV，max_cache_len=512；每种模态各 1 个固定请求
Live Hugging Face reference	相同三例均与 HF 新生成 token 一致	同一 latest-main 集成；InfiniLM TP=4，HF device_map=auto；
C-Eval validation	992/1346，73.70%	历史功能分支；1 × RTX 6000 D，TP=1，BF16，cpp backend，GPU gate，max_new_tokens=5；52 subjects
MMLU validation	922/1531，60.22%	与 C-Eval 相同的历史运行口径；57 subjects
MMMU adaptation smoke	InfiniLM TP=4 为 22/50（44%），HF 为 18/50（36%）	历史功能分支；相同 50 个评分 row、official prompt、image_size=224、max_new_tokens=1024 和 parser；两侧另各跳过 2 个候选，17/50 与 16/50 达到生成上限

MMMU 的两条运行使用相同输入和解析口径。在该固定子集上，InfiniLM 正确数未低于 Hugging Face；

测试截图：

• 01_tp4_text_image_video.png

• 02_benchmark_summary.png

赛题附件：

赛题报告.pdf
HONOR_CODE.md
REFERENCE.md

Benchmark / Performance Impact

视觉二维 RoPE、解码器三维 RoPE、多模态 embedding 替换和 router 仍有 CPU 计算或 CPU/GPU 复制。INFINILM_ERNIE_GPU_ROUTER=1 只将纯文本调用的 gate linear 放到 GPU；softmax、top-k、token 分组和 dispatch metadata 仍有 CPU 参与。

Notes for Reviewers

• 运行时 processor 使用官方 AutoProcessor(..., trust_remote_code=True) 生成 tokenizer、图像和视频预处理输入。Transformers 不执行提交模型的运行时 forward；HF model forward 只用于参考对照。
• token_type_ids 驱动专家组，grid_thw 驱动视觉网格和 Resampler；image_type_ids 当前作为元数据传。
• 语言解码器 QKV/O projection 和专家内部 linear 使用 TP；Vision Tower、Resampler、router 和专家集合在各 rank 复制，没有 ERNIE expert parallel。
• 最终输入使用三维 position IDs。视觉二维 RoPE 和解码器三维 RoPE 在 InfiniLM C++ 中实现，当前在 CPU 计算后复制回 GPU；一维/二维 position IDs 的回退路径复用 InfiniCore RoPE。
• 注意 position-ID 布局、相邻偶/奇维 RoPE、图像/视频占位 embedding 替换、专家范围和 gate 权重转置。
• TP=4 是最终提交已验证配置；其他设备数、国产平台、paged/flash backend 和 service 路径未做测试。

CI / ChatOps

• [x]

---

Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat(nvidia): …, fix(cuda/gemm): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
• No stray merge commits from main — the branch is rebased cleanly on top of the current main.
• No fixup! / squash! / wip commits remain.
• Existing PR/branch/commit that followed the legacy issue format.

Scope and Design

• Changes are minimal — nothing unrelated to the stated motivation was added (CONTRIBUTING.md §Code/General).
• No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
• No unrelated formatting churn that would obscure the diff.
• Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene (applies to all languages)

• The code is self-explanatory; comments were added only where the why is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, tab/space mixing, or stray BOMs.
• Identifiers in comments and error messages are wrapped in backticks (e.g. the `seqlens_k` tensor) (CONTRIBUTING.md §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• Code follows the Google C++ Style Guide strictly.
• Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
• Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
• No raw new/delete; RAII / smart pointers / existing allocators are used.
• Changed files are formatted by scripts/format.py.
• No changes/reference to csrc/models/llama_legacy/.

Python Specific (if Python files changed)

• Code is PEP 8 compliant.
• Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
• Docstrings (if any) follow PEP 257 (CONTRIBUTING.md §Python).
• Changed files are formatted by scripts/format.py.
• No changes/reference to python/infinilm/auto_config.py.

Testing

• For any platform that could not be tested, an explicit reason is given in the table and a reviewer with access has been tagged.
• Passed single request test (examples/test_infer.py), or specify the reason for skipping.
• Passed offline performance test (examples/bench.py), or specify the reason for skipping.
• Passed sanity test (test/bench/test_benchmark.py), or specify the reason for skipping.
• Passed service test (python/infinilm/server/inference_server.py + scripts/test_perf.py), or specify the reason for skipping.

Build, CI, and Tooling

• The project builds cleanly from a fresh directory on at least one affected platform.
• CI has been triggered manually (Actions → CI on this branch), or /retest was requested.

Documentation

• README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
• Any user-visible breaking change is called out explicitly under "Motivation" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• Third-party code is license-compatible and attributed.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.