Enable loading model from hub that has already been converted

[echarlaix]

• Enable loading model from the hub that have already been converted to executorch like this model

• Infer whether the model should be exported or not by checking whether a .pte file is present when loading the model

from optimum.executorch import ExecuTorchModelForCausalLM

model_id = "optimum-internal-testing/tiny-random-llama"
model = ExecuTorchModelForCausalLM.from_pretrained(model_id, revision="executorch")

[guangy10]

@echarlaix thank you for adding support to load converted and cached ExecuTorch model from hub. We are thinking of supporting same use cases. I'd like to get your insights how we can collaborate on it together. To make the discussion easier I opened a new GitHub Issue here: #15

[guangy10]

nit: It can still be a path to a local directory, right? Isn't the name confusing?

[echarlaix]

Yes! I used model_id to match what we currently have for OptimizedModel

https://github.com/huggingface/optimum/blob/856b25267ca686f05706cd44f0ebb669618cf302/optimum/modeling_base.py#L336

and ORTModel

https://github.com/huggingface/optimum/blob/856b25267ca686f05706cd44f0ebb669618cf302/optimum/onnxruntime/modeling_ort.py#L142

but if you think this is not clear we can think about an alternative

[guangy10]

I think the _save_pretrained is implemented already, when we export to ExecuTorch and save the pte to a local fs. I think we can just move the logic to this API, to provide HF users a consistent experience when working with ExecuTorch backend

[echarlaix]

we need to save the resulting pte file in _save_pretrained

optimum-executorch/optimum/executorch/modeling.py

Line 253 in 38c9782

raise NotImplementedError

can be done in a following PR

[guangy10]

IIUC it will try loading from a local directory only if the model_id is a local path, i.e. from_pretrained(model_id=my/local/path/to/executorch/model/). If the model_id is a hub id meta-llama/Llama-3.2-1B, it will download the cached one from hub. There is a case when the pte is already cached locally when calling from_pretrained(model_id=meta-llama/Llama-3.2-1B), shouldn't it prioritize to return the local cached one instead?

[echarlaix]

There is a case when the pte is already cached locally when calling from_pretrained(model_id=meta-llama/Llama-3.2-1B), shouldn't it prioritize to return the local cached one instead?

This will happen only for cases where the pte file is already present on the hub model repo (for example for https://huggingface.co/optimum-internal-testing/tiny-random-llama/tree/executorch) so for

ExecuTorchModelForCausalLM.from_pretrained("optimum-internal-testing/tiny-random-llama", revision="executorch")

but this won't be the case for https://huggingface.co/meta-llama/Llama-3.2-1B/tree/main as currently not pte files are present.

Is your point that the pte file should be saved in the cache after export ? not sure this is something that we want to do as it could results in conflicts

[guangy10]

I'm confused what are the difference w/ and w/o revision="executorch"? I guess the underlying question is what does the "revision" parameter do?

Oh I see what it is. https://huggingface.co/optimum-internal-testing/tiny-random-llama/tree/executorch. Here are the follow up question: When revision="main", the pte doesn't exist there, what would happen?

[echarlaix]

if no pte files detected then export will be set to True and the model will be converted to ExecuTorch on-the-fly, after this the user can either save the resulting model locally with .save_pretrained(save_dir) or directly push it on the hub with .push_to_hub(repo_id) (both method needs to be implemented, let me know if you're interested by tackling this in a following PR)

[guangy10]

Look through the PR but it's still unclear to me where exactly the cached model is fetched from hub. Let's taking the "meta-llama/Llama-3.2-1B" for example.

Is it fetching the the cached model from https://huggingface.co/executorch-community/Llama-3.2-1B-Instruct assuming this is an entry like that?

Or it's fetching the cached model from https://huggingface.co/meta-llama/Llama-3.2-1B under "Files and versions" tab?

[guangy10]

The logic that publishes pte to hub doesn't exist yet, which will be enabled in a separate PR, right?

[echarlaix]

Look through the PR but it's still unclear to me where exactly the cached model is fetched from hub. Let's taking the "meta-llama/Llama-3.2-1B" for example.

Is it fetching the the cached model from https://huggingface.co/executorch-community/Llama-3.2-1B-Instruct assuming this is an entry like that?

Or it's fetching the cached model from https://huggingface.co/meta-llama/Llama-3.2-1B under "Files and versions" tab?

If the model_id is set to meta-llama/Llama-3.2-1B then https://huggingface.co/meta-llama/Llama-3.2-1B

The logic that publishes pte to hub doesn't exist yet, which will be enabled in a separate PR, right?

can be integrated via a push_to_hub method in #13 (comment)

[echarlaix]

Merging and we can add more features (saving exported model locally, pushing on the hub, docs) in following PRs cc @guangy10