[Model] Support deepseek with eagle

[xyang16]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

This PR is to support running eagle speculative decoding on deepseek model. Changed the following file:

• deepseek_eagle.py: deepseek eagle model definition
• registry.py: add the model to registry

Test Plan

Added test in test_spec_decode.py:

pytest -s -v tests/v1/e2e/test_spec_decode.py

vllm serve cmd:

export VLLM_USE_V1=1
export VLLM_MLA_DISABLE=1
vllm serve deepseek-ai/DeepSeek-R1 \
    --port 8080 \
    --tensor-parallel-size 8 \
    --max-model-len 8192 \
    --max-num-seqs 8 \
    --trust-remote-code \
    --speculative-config '{"method": "eagle", "model":"eagle618/eagle-deepseek-r1", "num_speculative_tokens": 3}'

Test Result

The following unit tests passed:

tests/v1/e2e/test_spec_decode.py::test_eagle_correctness[FLASH_ATTN_VLLM_V1-deepseek_eagle]
tests/v1/e2e/test_spec_decode.py::test_eagle_correctness[TREE_ATTN-deepseek_eagle]

serve deepseek-ai/DeepSeek-R1 and benchmarking with llmperf:

INFO 07-16 18:05:38 [metrics.py:87] SpecDecoding metrics: Draft acceptance rate: 58.4%, Mean acceptance length: 2.75, Accepted: 163 tokens, Drafted: 279 tokens, Per-position acceptance rate: 0.882, 0.559, 0.312
INFO 07-16 18:08:09 [metrics.py:87] SpecDecoding metrics: Draft acceptance rate: 63.3%, Mean acceptance length: 2.90, Accepted: 167 tokens, Drafted: 264 tokens, Per-position acceptance rate: 0.886, 0.670, 0.341

(Optional) Documentation Update

[gemini-code-assist]

Code Review

This pull request adds support for Eagle speculative decoding with Deepseek models. I've found a few critical issues in the implementation that will prevent it from working correctly. The model implementation in deepseek_eagle.py incorrectly handles hidden state dimensions and is missing the lm_head layer, which will cause runtime errors. Additionally, the model registry key in registry.py seems to be incorrect, which would prevent the model from being loaded.

[gemini-code-assist]

The implementation of DeepseekV2Model assumes that the draft model and the target model share the same hidden size. For instance, self.hnorm is initialized with the draft model's hidden size (self.config.hidden_size) but is applied to hidden_states from the target model.

This assumption is incorrect for the models used in testing (deepseek-r1 has a hidden size of 4096, while eagle-deepseek-r1 has 1024), and will lead to a runtime error due to shape mismatch.

To fix this, you should explicitly use the hidden sizes from both the draft and target model configurations. You can access the target model's configuration via vllm_config.model_config.

        target_config = vllm_config.model_config.hf_config
        draft_hidden_size = self.config.hidden_size
        target_hidden_size = target_config.hidden_size

        self.fc = nn.Linear(
            draft_hidden_size + target_hidden_size,
            draft_hidden_size,
            bias=False,
        )

        self.enorm = RMSNorm(draft_hidden_size,
                             eps=self.config.rms_norm_eps)
        self.hnorm = RMSNorm(target_hidden_size,
                             eps=target_config.rms_norm_eps)
        self.norm = RMSNorm(draft_hidden_size,
                            eps=self.config.rms_norm_eps)

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[Ja1Zhou]

Hi! I tried installing this pr from source. But got

OSError: eagle618/eagle-deepseek-r1 does not appear to have a file named configuration_deepseek.py. Checkout 'https://huggingface.co/eagle618/eagle-deepseek-r1/tree/main' for available files.

Should the auto_map field of config.json be fixed?

[xyang16]

Hi! I tried installing this pr from source. But got

OSError: eagle618/eagle-deepseek-r1 does not appear to have a file named configuration_deepseek.py. Checkout 'https://huggingface.co/eagle618/eagle-deepseek-r1/tree/main' for available files.

Should the auto_map field of config.json be fixed?

Thanks for your comment, fixed now.

[Ja1Zhou]

Hi! I tried installing this pr from source. But got

OSError: eagle618/eagle-deepseek-r1 does not appear to have a file named configuration_deepseek.py. Checkout 'https://huggingface.co/eagle618/eagle-deepseek-r1/tree/main' for available files.

Should the auto_map field of config.json be fixed?

Thanks for your comment, fixed now.

Hi! I tried installing this pr from source. But got

OSError: eagle618/eagle-deepseek-r1 does not appear to have a file named configuration_deepseek.py. Checkout 'https://huggingface.co/eagle618/eagle-deepseek-r1/tree/main' for available files.

Should the auto_map field of config.json be fixed?

Thanks for your comment, fixed now.

Amazing work!

I wonder if you could share how you got eagle618/eagle-deepseek-r1? As this pr could also improve DS V3 etc. Thank you!

[aarnphm]

Do you have a eagle checkpoint to test with this? If you have some numbers that would be great.

Llama 4 EAGLE was landed recently, so I do think we can probably do the same wrt to tests for this (if you need some references for test cases)

[xyang16]

Do you have a eagle checkpoint to test with this? If you have some numbers that would be great.

Llama 4 EAGLE was landed recently, so I do think we can probably do the same wrt to tests for this (if you need some references for test cases)

Thanks for your review. I have tested with the checkpoint eagle618/eagle-deepseek-r1.

Also added unit test case.

[benchislett]

Does this need to be made compatible with the fused_qkv_a_proj optimization from #21116? I have observed multiple issues with weight loading in MTP not being consistent with the DeepSeek base model weight loading. Will similar issues apply here?

[xyang16]

I have updated the stacked_params_mapping. Thanks!

[benchislett]

There have been many discussions in the community about how to properly handle the rotated input slot, but this does not seem in line with the final state. If I recall correctly, there was concern that overwriting the hidden states to zero will give out-of-distribution results during attention. See the other EAGLE implementations in vLLM (such as llama_eagle.py) for reference.

[xyang16]

Yes, there's discussion this will mess up the attention normalization. I have removed this. Please review. Thanks.

[xyang16]

I see deepseek_mtp.py has also masked the hidden states to 0: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/deepseek_mtp.py#L74

I can remove the line in deepseek_mtp.py together in this PR. But I can leave it there too. Let me know how you think.

[benchislett]

Perhaps best to leave it pending a complete study of impact on AL for MTP. If there isn't a github issue for this task, please create one

[benchislett]

This looks like too many norms being applied. In the Llama_Eagle reference code, the input layernorm to each layer is disabled, and IIRC there is no output layernorm. Here, there are two norms applied to the input (pre-concat and input-layernorm after concat) and two more norms applied after (post_attention_layernorm and self.norm). This does not seem correct.

[xyang16]

I have taken a look at the deepseek_mtp.py at https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/deepseek_mtp.py#L64. The only difference is output self.norm. But in our benchmarking, we found that including the output norm will increase acceptance rate.

[benchislett]

Acceptance rate is around 60% for vanilla eagle head (and better acceptance rate for fine tuned eagle head).

What does this mean? What are the weights of a "vanilla eagle head" in this case?

[xyang16]

Acceptance rate is around 60% for vanilla eagle head (and better acceptance rate for fine tuned eagle head).

What does this mean? What are the weights of a "vanilla eagle head" in this case?

@benchislett Thanks for your review! I mean people can further fine tune the weights and get better acceptance rate. The "vanilla eagle head" in this case is eagle618/eagle-deepseek-r1.

[benchislett]

The implementation now seems more in-line with the MTP implementation. There are still differences between how we handle EAGLE and MTP models (whether norms are applied to input_layernorm or not, normed before output, for example) and this PR blends the two by implementing an EAGLE class in a manner more consistent with MTP.

We should try to find some way to unify implementations and reconcile the differences, but this is probably not the PR to bear that burden. For now, this will suffice and can be extended if future eagle-style MTP modules are released with slight differences in implementation.

[simon-mo]

Stamping given Benjamin approved.

[gyou2021]

Why export VLLM_MLA_DISABLE=1? Is it ok that DeepSeek-R1 inference with VLLM_MLA_DISABLE=0 and deepseek_eagle with VLLM_MLA_DISABLE= 0 ?