Qwen2.5 spinquant support

[Francis235]

🚀 The feature, motivation and pitch

Hi, I am deploying the Qwen2.5 0.5B model via executorch, I want to use quantized version with spinquant, I want to know, what should I do? Many thanks for considering my request.

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @kimishpatel @jerryzh168 @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng

[shoumikhin]

@kimishpatel can you help with that?

[kimishpatel]

Do you have quantized model that was quantized with spinquant algo?

[Francis235]

Do you have quantized model that was quantized with spinquant algo?

No, I don't have a quantized qwen2.5 0.5b model that was quantized with the spinquant algorithm. I noticed that spiquant does not support qwen2.5 0.5 b.

[GregoryComer]

@kimishpatel Do you have any pointers for how to perform spinquant quantization on the model? I'm tentatively assigning it to you, but feel free to let me know if you're not the right person.

[jackzhxng]

Hi @Francis235, please refer to https://github.com/facebookresearch/SpinQuant on how to prequantize with spin quant. After that is done, you can load the prequantized spin checkpoint with the following instructions - https://github.com/pytorch/executorch/blob/main/examples/models/llama/README.md#option-a-download-and-export-llama32-1b3b-model