How to Apply Different Quantization Settings Per Layer in ExecuTorch?

[crinex]

Dear @kimishpatel @jerryzh168 @shewu-quic

I want to split a model(eg, Llama-3.2-3B) into multiple layers and apply different quantization settings(qnn_8a8w, qnn_16a4w...) to each layer.
Has such a method been tested in ExecuTorch?
If not, could you suggest how this can be achieved?

Thank you

[kimishpatel]

Is this specific to qnn backend or your question is in general?

[crinex]

@kimishpatel

I meant was about the QNN backend.
I’ve noticed that when I quantize the LLaMA 3.2-3B model using QNN_8A8W, inference doesn’t work properly on the device. It seems that others are reporting similar results in issues as well.
Therefore, I’m considering solving this problem using mixed precision on the QNN backend. Would that be possible?

[iliasslasri]

Hi @crinex, Sorry, I can't answer your question, but I run into same issue and I would like to know why you tried to solve it with this quantization scheme and if this works.

[crinex]

Hi @iliasslasri,
The Llama-3.2-3B model exceeds 4GB in size, even with 16A16W quantization, making it impossible to upload to the QNN DSP.

Therefore, int8 quantization becomes essential. However, ExecuTorch typically only supports static quantization methods for QNN, which cannot sufficiently cover the dynamic range of activations.

As you well know, not all layers in LLM models carry equally significant information for inference.

Therefore, we aim to quantize less inference-sensitive layers to 8-bit or even 4-bit, while int16 or fp16 quantization for the more sensitive layers.

In fact, we have successfully implemented this approach using the ONNX framework instead of ExecuTorch.

[kimishpatel]

The Llama-3.2-3B model exceeds 4GB in size, even with 16A16W quantization, making it impossible to upload to the QNN DSP.

WHy do you say that? Also typically dynamic activation quantizatio is needed if you have int8 only. But with int16 you can live with static quant. This is what has been my understanding. Does that not work?

In fact, we have successfully implemented this approach using the ONNX framework instead of ExecuTorch

Was that for QNN DSP?

cc @cccclai. Chen can you also tag appropriate POCs from qualcomm side?

[crinex]

@kimishpatel

WHy do you say that? Also typically dynamic activation quantizatio is needed if you have int8 only. But with int16 you can live with static quant. This is what has been my understanding. Does that not work?

As you mentioned, we used qnn_8a8w for quantization to int8. However, the inference did not work properly after quantization, even with calibration-based quantization. Moreover, from an application perspective, mixed precision is a very commonly used method.

Was that for QNN DSP?

That's right, we managed to make it work on the QNN DSP. We extracted the mixed-precision model using ONNX and utilized the runtime provided by QNN. Additionally, they also apply mixed precision to almost all LLM models. They don't lower all blocks to int16 or int8. You can check this on the Qualcomm AI Hub.

[kimishpatel]

As you mentioned, we used qnn_8a8w for quantization to int8. However, the inference did not work properly after quantization, even with calibration-based quantization. Moreover, from an application perspective, mixed precision is a very commonly used method.

AFAIK qnn doesnt support dynamic quantization of activations

That's right, we managed to make it work on the QNN DSP. We extracted the mixed-precision model using ONNX and utilized the runtime provided by QNN. Additionally, they also apply mixed precision to almost all LLM models. They don't lower all blocks to int16 or int8. You can check this on the Qualcomm AI Hub

ANy pointers?

[crinex]

AFAIK qnn doesnt support dynamic quantization of activations

That's right. Since QNN does not provide dynamic quantization, the dynamic changes in activations cannot be addressed solely through calibration. Therefore, we used Mixed Precision.

ANy pointers?

https://aihub.qualcomm.com/mobile/models/llama_v3_2_3b_chat_quantized?domain=Generative+AI&useCase=Text+Generation
Check out the Technical Details in the link provided.

[cccclai]

It sounds like you're trying to apply different precision on different layers. It is still static quantization, instead of dynamic quantization.

We did some analysis on the mix precision, but haven't tried it on qnn backend. To use mixed precision in executorch, we'd need to annotate the linear layer differently for each layer. By inspecting node.meta, we should be able to find out which layer this node belongs to and apply the appropriate quantization schema accordingly.