Skip to content

Support PP on tpu_inference #27296

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 26 commits into
base: main
Choose a base branch
from
Draft

Support PP on tpu_inference #27296

wants to merge 26 commits into from

Conversation

@Chenyaaang
Copy link
Contributor

@Chenyaaang Chenyaaang commented Oct 21, 2025 •
edited by github-actions bot
Loading

Purpose

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@mergify mergify bot added llama Related to Llama models v1 tpu Related to Google TPUs labels Oct 21, 2025
njhill and others added 26 commits October 22, 2025 22:58
@njhill @Chenyaaang
[V0 Deprecation] Remove V0 metrics code (vllm-project#27215)
663e7bb 
Signed-off-by: Nick Hill <[email protected]>
@fadara01 @focusunsink @Chenyaaang
[cpu] Dispatch un-quantized linear to oneDNN/ACL by default for AArch…
a4d5557 
…64 (vllm-project#27183)

Signed-off-by: Fadi Arafeh <[email protected]>
Co-authored-by: Michael Yang <[email protected]>
@helunwencser @Chenyaaang
create is_in_the_same_node on cpu (vllm-project#26832)
510c64d 
Co-authored-by: Lunwen He <[email protected]>
@russellb @Isotr0py @Chenyaaang
[Frontend] Enforce tokenize=False when applying chat template (vllm-p…
c98b153 
…roject#27205)

Signed-off-by: Isotr0py <[email protected]>
Co-authored-by: Isotr0py <[email protected]>
@wcwuwc @vangheem
@banjuede @jeejeelee @B-201 @Chenyaaang
[Feature][Kernel]FusedMoE LoRA (vllm-project#21229)
1749b20 
Signed-off-by: wuchen <[email protected]>
Signed-off-by: banjuede <[email protected]>
Signed-off-by: Chen Wu <[email protected]>
Signed-off-by: Danielle Robinson <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: bk-201 <[email protected]>
Co-authored-by: wuchen <[email protected]>
Co-authored-by: Nathan Van Gheem <[email protected]>
Co-authored-by: banjuede <[email protected]>
Co-authored-by: Danielle Robinson <[email protected]>
Co-authored-by: Jee Jee Li <[email protected]>
Co-authored-by: bk-201 <[email protected]>
@nvpohanh @Chenyaaang
[BugFix] GPT-OSS Attention DP + MoE TP weight loading issue (vllm-pro…
2eb7aab 
…ject#24032)

Signed-off-by: Po-Han Huang <[email protected]>
@wenscarl @mgoin @Chenyaaang
[ModelOpt] Load w13/w2_input_scale for all experts, nvfp4 (vllm-proje…
a5c3632 
…ct#26135)

Signed-off-by: Shu Wang <[email protected]>
Signed-off-by: Shu Wang. <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
@varun-sundar-rabindranath @mgoin @Chenyaaang
[Bugfix] Fix gpt-oss w4a8 DP/EP on B200 (vllm-project#26729)
d74d9c0 
Signed-off-by: Varun Sundar Rabindranath <[email protected]>
Co-authored-by: Varun Sundar Rabindranath <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
@benchislett @Chenyaaang
[Bugfix] Fix broken MTP weight loading for FP8 KV Scales (vllm-projec…
5593b0c 
…t#27227)

Signed-off-by: Benjamin Chislett <[email protected]>
@linzebing @Chenyaaang
[Fix][Spec Decode] Fix llama4 draft loading with different quantizati…
7655118 
…on (vllm-project#27136)

Signed-off-by: linzebing <[email protected]>
@NickLucche @Chenyaaang
[Nixl] Minor refactor to handshake related metadata (vllm-project#26410)
f72ff17 
Signed-off-by: NickLucche <[email protected]>
@ywang96 @Chenyaaang
[MM][Core] Decouple ViT backend from LM backend (vllm-project#27061)
c448530 
Signed-off-by: Roger Wang <[email protected]>
@dcampora @Chenyaaang
[Deepseek v3.2] Optimize top_k_per_row (vllm-project#26763)
98123db 
Signed-off-by: Daniel Campora <[email protected]>
@dongbo910220 @Chenyaaang
[Chore] Separate out NCCL utilities from vllm.utils (vllm-project#27197)
94bcb5c 
Signed-off-by: dongbo910220 <[email protected]>
@hmellor @Chenyaaang
[CI] Install pre-release version of apache-tvm-ffi for flashinfer (…
fdd7368 
…vllm-project#27262)

Signed-off-by: Harry Mellor <[email protected]>
@JartX @Chenyaaang
[ROCM] Enable CompressedTensorsWNA16 (vllm-project#27187)
46fcb1a 
Signed-off-by: JartX <[email protected]>
@pavanimajety @Chenyaaang
Add @pavanimajety to .github/codeowners for Flashinfer, ModelOpt rela…
4fb6c8a 
…ted code (vllm-project#27213)

Signed-off-by: Pavani Majety <[email protected]>
@micah-wil @Chenyaaang
[ROCm] Update Triton, Torch, and AITER branches for ROCm base Dockerf…
6e22527 
…ile (vllm-project#27206)

Signed-off-by: Micah Williamson <[email protected]>
@yewentao256 @Chenyaaang
[Feature] Batch Invariant for R1 TP 8 on Blackwell (vllm-project#27229)
a1e5a8c 
Signed-off-by: yewentao256 <[email protected]>
@dagrayvid @Chenyaaang
[Bugfix][P/D] Reduce num_threads used by nixl ucx backend (vllm-proje…
9a8a88d 
…ct#27196)

Signed-off-by: David Whyte-Gray <[email protected]>
@njhill @Chenyaaang
[V0 Deprecation] Remove V0 executors (vllm-project#27142)
fcbd6db 
Signed-off-by: Nick Hill <[email protected]>
@sighingnow @LucasWilkinson @Chenyaaang
[Bugfix] fixes the decoding metadata of dense mla's fp8 kvcache. (vll…
5f8c69a 
…m-project#27144)

Signed-off-by: Tao He <[email protected]>
Signed-off-by: Lucas Wilkinson <[email protected]>
Co-authored-by: Lucas Wilkinson <[email protected]>
@lucianommartins @Chenyaaang
[Model] Revert PR vllm-project#26715: Restore custom PaliGemma and Ge…
df47c7a 
…mma3-MM impl… (vllm-project#27309)

Signed-off-by: Luciano Martins <[email protected]>
Co-authored-by: Luciano Martins <[email protected]>
@Chenyaaang
fix tpu torch compile error
ce43758 
Signed-off-by: Chenyaaang <[email protected]>
@Chenyaaang
PP for single host
1ed270b 
Signed-off-by: Chenyaaang <[email protected]>

llama debug

Signed-off-by: Chenyaaang <[email protected]>

core debug

Signed-off-by: Chenyaaang <[email protected]>

pp for single host

Signed-off-by: Chenyaaang <[email protected]>

pp single host

Signed-off-by: Chenyaaang <[email protected]>

pp single host comment

Signed-off-by: Chenyaaang <[email protected]>

amend single host

Signed-off-by: Chenyaaang <[email protected]>
@Chenyaaang
PP on Ray
f0635ca 
Signed-off-by: Chenyaaang <[email protected]>

amend ray

Signed-off-by: Chenyaaang <[email protected]>
@mergify
Copy link

mergify bot commented Oct 28, 2025

Documentation preview: https://vllm--27296.org.readthedocs.build/en/27296/

@mergify mergify bot added documentation Improvements or additions to documentation ci/build deepseek Related to DeepSeek models frontend multi-modality Related to multi-modality (#4194) new-model Requests to new models qwen Related to Qwen models gpt-oss Related to GPT-OSS models rocm Related to AMD ROCm speculative-decoding labels Oct 28, 2025
@mergify
Copy link

mergify bot commented Oct 28, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Chenyaaang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Oct 28, 2025
@mergify mergify bot added the kv-connector label Oct 28, 2025
@mergify mergify bot added the nvidia label Nov 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@NickLucche NickLucche Awaiting requested review from NickLucche NickLucche will be requested when the pull request is marked ready for review NickLucche is a code owner

@gshtras gshtras Awaiting requested review from gshtras gshtras will be requested when the pull request is marked ready for review gshtras is a code owner

@tjtanaa tjtanaa Awaiting requested review from tjtanaa tjtanaa will be requested when the pull request is marked ready for review tjtanaa is a code owner

@luccafong luccafong Awaiting requested review from luccafong luccafong will be requested when the pull request is marked ready for review luccafong is a code owner

@sighingnow sighingnow Awaiting requested review from sighingnow sighingnow will be requested when the pull request is marked ready for review sighingnow is a code owner

@bigPYJ1151 bigPYJ1151 Awaiting requested review from bigPYJ1151 bigPYJ1151 will be requested when the pull request is marked ready for review bigPYJ1151 is a code owner

@hmellor hmellor Awaiting requested review from hmellor hmellor will be requested when the pull request is marked ready for review hmellor is a code owner

@ApostaC ApostaC Awaiting requested review from ApostaC ApostaC will be requested when the pull request is marked ready for review ApostaC is a code owner

@jeejeelee jeejeelee Awaiting requested review from jeejeelee jeejeelee will be requested when the pull request is marked ready for review jeejeelee is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin will be requested when the pull request is marked ready for review mgoin is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat will be requested when the pull request is marked ready for review robertgshaw2-redhat is a code owner

@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 will be requested when the pull request is marked ready for review yewentao256 is a code owner

@pavanimajety pavanimajety Awaiting requested review from pavanimajety pavanimajety will be requested when the pull request is marked ready for review pavanimajety is a code owner

@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337 DarkLight1337 will be requested when the pull request is marked ready for review DarkLight1337 is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 will be requested when the pull request is marked ready for review ywang96 is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth will be requested when the pull request is marked ready for review tlrmchlsmth is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon will be requested when the pull request is marked ready for review WoosukKwon is a code owner

@aarnphm aarnphm Awaiting requested review from aarnphm aarnphm will be requested when the pull request is marked ready for review aarnphm is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao will be requested when the pull request is marked ready for review youkaichao is a code owner

@houseroad houseroad Awaiting requested review from houseroad houseroad will be requested when the pull request is marked ready for review houseroad is a code owner

@ProExpertProg ProExpertProg Awaiting requested review from ProExpertProg ProExpertProg will be requested when the pull request is marked ready for review ProExpertProg is a code owner

@LucasWilkinson LucasWilkinson Awaiting requested review from LucasWilkinson LucasWilkinson will be requested when the pull request is marked ready for review LucasWilkinson is a code owner

@zou3519 zou3519 Awaiting requested review from zou3519 zou3519 will be requested when the pull request is marked ready for review zou3519 is a code owner

@chaunceyjiang chaunceyjiang Awaiting requested review from chaunceyjiang chaunceyjiang will be requested when the pull request is marked ready for review chaunceyjiang is a code owner

@zhuohan123 zhuohan123 Awaiting requested review from zhuohan123 zhuohan123 will be requested when the pull request is marked ready for review zhuohan123 is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat will be requested when the pull request is marked ready for review alexm-redhat is a code owner

@njhill njhill Awaiting requested review from njhill njhill will be requested when the pull request is marked ready for review njhill is a code owner

@22quinn 22quinn Awaiting requested review from 22quinn 22quinn will be requested when the pull request is marked ready for review 22quinn is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@sangstar sangstar

Labels

ci/build deepseek Related to DeepSeek models documentation Improvements or additions to documentation frontend gpt-oss Related to GPT-OSS models kv-connector llama Related to Llama models multi-modality Related to multi-modality (#4194) needs-rebase new-model Requests to new models nvidia qwen Related to Qwen models rocm Related to AMD ROCm speculative-decoding tpu Related to Google TPUs v1

Projects

NVIDIA
Status: No status
gpt-oss Issues & Enhancements
Status: To Triage

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.