feat: output-format arg for synthetic data generation script and improved support for Mistral models in the Pytorch workflow

[hypdeb]

• Added an output-format argument to prepare_dataset.py allowing the generation of datasets compatible with tllm-bench without having to go through stdout
• Added explicit support for Mistral models in the Pytorch modelling code, as their configurations are slightly different from the Llama models (even though the architecture is mostly similar)
• Added an example script on how to quantize a HuggingFace checkpoint to a HuggingFace checkpoint for usage in the Pytorch workflow

[hypdeb]

/bot run

[hypdeb]

@kaiyux @FrankD412 I would like to reopen the discussion on the output format of the synthetic dataset generation scripts.

Personally, I think it makes sense for the script to be able to generate different output formats. This set of changes should add support for the tllm-bench format without breaking the current behaviour as the default value for the new output-format argument is gptManagerBenchmark, which is what the script is generating currently.

[hypdeb]

That being said, of course, it would be preferrable if, in the future, tools that ingest datasets could conform to an existing format.

[tensorrt-cicd]

PR_Github #3302 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #3302 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #2300 completed with status: 'FAILURE'

[FrankD412]

That being said, of course, it would be preferrable if, in the future, tools that ingest datasets could conform to an existing format.

I agree, the whole reason I introduced the second format was because gptManagerBenchmark was limited to input ids because it was in C++ (and didn't have access to the transformers library in Python). When I started trtllm-bench, an initial goal was to allow it to take either a prompt or the input ids and be able to take in requests from stdin. The existing JSON didn't lend itself to the latter, but the latter ended up becoming less important (though it might be coming back up again).

Also, maybe this is personal preference, but this PR now addresses two unrelated things which makes it harder to roll back or track changes in trtllm-bench. Mind if we split these sorts of things in the future?

[hypdeb]

I will split the Mistral stuff out.

[hypdeb]

Here is the Mistral changes: #3843. I'll rebase this one on top of it once merged, which should leave only the dataset changes.

[hypdeb]

Moved the dataset generation stuff to #3866, and expanded it. Closing this one as all changes are now represented on other branches.