feat: added support for Mistral models in Pytorch workflow and HF quantization script #3843
Conversation
…ion script Signed-off-by: jdebache <[email protected]>
|
Hello @litaotju do you know who is the best person to review modelling changes in the Pytorch workflow? |
|
|
||
| ## Quantizing from the HuggingFace format to the HuggingFace format | ||
| It is useful to be able to quantize models from HuggingFace, without changing their format, for example if you plan to use them in TensorRT-LLM's Pytorch-based workflow. The `quantize_hf_to_hf.py` script serves that purpose. It is a reduced version of ModelOpt's [example post-training quantization script](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/examples/llm_ptq/hf_ptq.py). Please refer to the original for a more up-to-date version. For example, it can be used to quantize a model to `fp8` for tensor-parallelism 4: | ||
| ``` |
There was a problem hiding this comment.
Hi @hypdeb , I don't think it's a good idea to copy scripts from ModelOpt to TRT-LLM. Can we add some links to ModelOpt instead? like this one: https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/torch.md#quantization
There was a problem hiding this comment.
I agree that in general copying sources is not a good idea. I did it here as this is an example usage script, so it is basically made to be copied and adjusted by users, or at least that's how I see it. What do you think?
There was a problem hiding this comment.
Could you please create a PR for Mistral models only? We can merge it first. For the HF quantization script, I think it's still under debate about the user workflow.
There was a problem hiding this comment.
Here is the MR with only the added Mistral modelling code: #3845
There was a problem hiding this comment.
For the HF quantization script, I think it's still under debate about the user workflow.
Could you maybe CC me on these discussions / ping me on Slack on this? I am interested as I would like to add a more lightweight and TRTLLM independent flow for quantization to my automation on the long term.
No description provided.