Skip to content

Conversation

hmellor
Copy link
Member

@hmellor hmellor commented Jul 21, 2025

This PR adds an explicit redirect from Emu3ForConditionalGeneration to TransformersForMultimodalLM. This means that:

  • The Transformers backend will be used to load this model
  • Users won't see warnings about the Transformers backend being fallback behaviour

In the docs I have added a paragraph explaining what it means for a model to be officially supported via the Transformers backend.

Closes #11008

hmellor added 3 commits July 21, 2025 13:05
Signed-off-by: Harry Mellor <[email protected]>
Signed-off-by: Harry Mellor <[email protected]>
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added documentation Improvements or additions to documentation new-model Requests to new models labels Jul 21, 2025
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request officially adds support for the Emu3 model by redirecting it to the Transformers backend, and updates the documentation accordingly. The code changes are logical and well-structured. My main feedback is to add a corresponding test case for the newly supported model to ensure its correctness and prevent future regressions, as is standard practice in this repository.

Copy link
Member

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, hope that this would encourage more people to consider Transformers backend before opening their own PRs!

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) July 22, 2025 04:47
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 22, 2025
@DarkLight1337
Copy link
Member

PTAL at multi-modal tests failure

@hmellor
Copy link
Member Author

hmellor commented Jul 22, 2025

It's a weight mapping issue. We've had a TODO in the code for a while to simplify the weight mapping so I've made #21382 to do that ahead of this PR.

Signed-off-by: Harry Mellor <[email protected]>
Copy link

mergify bot commented Jul 23, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@DarkLight1337 DarkLight1337 disabled auto-merge July 23, 2025 09:25
@hmellor
Copy link
Member Author

hmellor commented Jul 23, 2025

Fixing the hf_to_vllm_mapper revealed an issue where the running stats are not being loaded into the nn.BatchNorm3d in Emu3's vision tower. Currently working on a fix.

@mergify mergify bot added the multi-modality Related to multi-modality (#4194) label Jul 23, 2025
@hmellor
Copy link
Member Author

hmellor commented Jul 23, 2025

The failing mapper test was using model.named_parameters() as reference but the Emu3 checkpoint also contains buffers (stats for batch norm).

I have modified the test to ignore buffers that it may find in checkpoints.

@DarkLight1337 DarkLight1337 added this to the v0.10.0 milestone Jul 23, 2025
@vllm-bot vllm-bot merged commit 13abd0e into vllm-project:main Jul 24, 2025
69 of 71 checks passed
@hmellor hmellor deleted the emu3 branch July 24, 2025 10:45
avigny pushed a commit to avigny/vllm that referenced this pull request Jul 31, 2025
wenscarl pushed a commit to wenscarl/vllm that referenced this pull request Aug 4, 2025
x22x22 pushed a commit to x22x22/vllm that referenced this pull request Aug 5, 2025
Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025
npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025
jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025
paulpak58 pushed a commit to paulpak58/vllm that referenced this pull request Aug 13, 2025
taneem-ibrahim pushed a commit to taneem-ibrahim/vllm that referenced this pull request Aug 14, 2025
BoyuanFeng pushed a commit to BoyuanFeng/vllm that referenced this pull request Aug 14, 2025
diegocastanibm pushed a commit to diegocastanibm/vllm that referenced this pull request Aug 15, 2025
epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 28, 2025
googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation multi-modality Related to multi-modality (#4194) new-model Requests to new models ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[New Model]: Emu3
3 participants