[Feature]: Support dense MLP & rope for deepseek architecture

[wavy-jung]

🚀 The feature, motivation and pitch

Huggingface Deepseek-V2 model supports for 1) the case when n_routed_experts is None (which is fully dense model) and 2) the pure RoPE without YaRN.

• https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite/blob/604d5664dddd88a0433dbae533b7fe9472482de0/modeling_deepseek.py#L1207-L1214
• https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite/blob/604d5664dddd88a0433dbae533b7fe9472482de0/modeling_deepseek.py#L754-L760
  These two features should be supported for compatibility with Huggingface modeling code.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[NickLucche]

Would you have the reference of a model employing one of these two architectural changes?

[Isotr0py]

2. the pure RoPE without YaRN.

In fact, the pure RoPE without YaRN has been supported in deepseek_v2 and used by deepseek-vl2 models:

vllm/vllm/model_executor/models/deepseek_v2.py

Lines 249 to 259 in 5d98d56

	if rope_scaling:
	rope_scaling["rope_type"] = 'deepseek_yarn'
	self.use_normal_rope = False
	else:
	self.use_normal_rope = True
	self.rotary_emb = get_rope(qk_rope_head_dim,
	rotary_dim=qk_rope_head_dim,
	max_position=max_position_embeddings,
	base=rope_theta,
	rope_scaling=rope_scaling,
	is_neox_style=False)

vllm/vllm/model_executor/models/deepseek_v2.py

Lines 303 to 312 in 5d98d56

	if self.use_normal_rope:
	seq_len = positions.size(0)
	ori_q_pe_shape, ori_k_pe_shape = q_pe.shape, k_pe.shape
	q_pe = q_pe.reshape(seq_len, -1)
	k_pe = k_pe.reshape(seq_len, -1)
	q_pe, k_pe = self.rotary_emb(positions, q_pe, k_pe)
	if self.use_normal_rope:
	q_pe, k_pe = q_pe.view(ori_q_pe_shape), k_pe.view(ori_k_pe_shape)

[Isotr0py]

FYI, pure RoPE with MLA support will be done in #12729