[FT] Custom model to TransformersModel

[Giuseppe5]

Hello everyone,

First of all, thanks for the amazing work.

Issue encountered

I have been trying to use lighteval but I'm facing an issue.

From my current understanding, it is only possible to pass a pretrained string to TransformersModelConfig, which means the model has to be already present on the hub and can't be modified in any way before using it for eval.

I tried passing model directly to Pipeline but I get the following error:

AttributeError: 'TransformersModel' object has no attribute 'generation_config_dict'

Solution/Feature

Instead of passing a pretrained string to TransformersModelConfig, I was wondering if it's possible to pass a torch.nn.Module and use that for evaluation purpose.
The idea is to pass to lighteval a transformed HF model (e.g., after applying quantization through third-party libraries).

Please let me know if I'm using the library wrong or misunderstanding something.

Many thanks,
Giuseppe

[clefourrier]

Hi! Thanks for the kind words!

Easy way is simply to upload your quantized model to the hub, or save it locally and load from there.
You can also use lighteval programmatically (https://huggingface.co/docs/lighteval/using-the-python-api) and load a model directly in Python - you might need to inherit the LightevalModel abstract class :)

[Giuseppe5]

Yes I have been playing with it through the python API, passing model to Pipeline but I get the error below:

AttributeError: 'TransformersModel' object has no attribute 'generation_config_dict'

This happens also when loading a model from HF and then passing it to pipeline (no quantization or any other modification really to a normal AutoModelForCasualLM)

[Giuseppe5]

After some experimenting, the issue seems to be due to a mismatch in how TransformersModel gets initalized when calling init compared to what appens with from_model.
There are a few configuration missing in from_model that cause the error and some numerical discrepancies once that error is fixed.

Also, unrelated to this, I noticed some smaller issues around some hyper-params like override_batch_size. If I just remove it, it goes to None and then there's an error when checking if override_batch_size > 0.

A similar issue seems to exist around the accelerator parameter. Related to this, in the API example above, the accelerator variable is created but never used.

I can post a script to reproduce but I basically copy-pasted the API tutorial you linked with some minor modifications here and there.

[sasha-hailo]

I second @Giuseppe5 's observations.
Encountered the same exception, when trying to initialize TransformersModel via from_model path (when provided a HF transformers module).
The root cause seems missing treatment of several aspects, one of which is generation_config_dict.
But I'm afraid there are other issues here, as I keep stumbling on additional exceptions.

@clefourrier, I definitely think it's a bug.

[Giuseppe5]

I ended up doing a very ugly looking fix in a personal fork, but I would not recommend that for anything if not to understand what's missing to get custom models to "work", but a lot of clean-up is needed starting from there.

I could try to put that into words if there's the chance to get a PR for that.

[sasha-hailo]

@Giuseppe5 ,
Did you manage to eventually produce a running pipeline that actually outputs correct results?
I did clean up the bugs to make it run, but the outputs do not match those obtained by a sanity run via the CLI (e.g., lighteval vllm ...).