#Feature Request# Accelerated Deployment.

[Xingxiangrui]

Accelerated Deployment.. Regarding the current llama, there are many mature acceleration frameworks, such as LMDeploy, vLLM, and AWQ.
After using the MoE architecture, how to use these acceleration frameworks for deployment? Has adaptation been done?

[Spico197]

Hi there, thanks for the question~

It is possible to use these frameworks for LLaMA-MoE inference acceleration, and we are working on it. It may take some time for development and testing. Please hang on tight ❤️

Please don't close this issue until we have further progresses. Thank you for your patience~

[Xingxiangrui]

If I understand correctly, the model structure of llama-MoE is identical to the the model structure of Mixtral-MoE ( negelecting the sliding window attention). You can reuse the model structure of Mixtral-MoE for adaptation. Mixtral-MoE has already been deployed with vLLM and AWQ.