(WIP) DeepseekV3 (and Multi-Head Latent Attention)

[ysjprojects]

follow up from #1945

More alignment with the huggingface implementation, allowing custom values for q_lora_rank, v_dim, etc.
Cleaned up some lines added for debugging purposes.

Also, the errors in #1945 was simply because pythia simply did not have the necessary parameters for MLA.

[Borda]

let's add a test :)

[ali-alshaar7]

Looks good overall. +1 on the test though, and would be nice if we could share more with the base attention implementation instead of duplicating things. Maybe for a followup. Thank you.

[Borda]

@t-vi @lantiga mind have look, pls :)

[ysjprojects]

@t-vi @lantiga @Borda

DeepseekV3 architecture WIP.

Should we move DeepseekV3MoE and MultiHeadLatentAttention out of model.py into its own file since the architecture is quite unique so it's unlikely that a future model would implement it again?

[Borda]

@ysjprojects are you going to work on this one now? 🦩

[ysjprojects]

@ysjprojects are you going to work on this one now? 🦩

Definitely, but I will first work on much lower hanging fruits that are long overdue (Qwen3 Coder etc.)

[ysjprojects]

Closing this as I plan to break it into multiple components across PRs.

New PR on Multi-Head Latent Attention implementation: #2113