Skip to content

Conversation

@rkazants
Copy link
Collaborator

@rkazants rkazants commented Nov 7, 2025

What does this PR do?

Example of conversion cmd-line for ibm-granite/granite-4.0-micro:

optimum-cli export openvino -m ibm-granite/granite-4.0-micro granite-4.0-micro

Example of inference for ibm-granite/granite-4.0-micro using OpenVINO backend:

from transformers import AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM

model_path = "ibm-granite/granite-4.0-micro"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = OVModelForCausalLM.from_pretrained(model_path)

# change input text as desired
input_text = "The capital of France is"
# tokenize the text
input_tokens = tokenizer(input_text, return_tensors="pt").to(device)
# generate output tokens
output = model.generate(**input_tokens, max_length=10)
# decode output tokens into text
output = tokenizer.batch_decode(output)
print(output[0])

Before submitting

  • [N/A] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@rkazants rkazants requested a review from echarlaix November 7, 2025 16:49
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@rkazants rkazants changed the title [OpenVINO] Add support for Granite-4.0 family (incl. models with MoE and Mamba as well) [OpenVINO] Add support for Granite-4.0 family (incl. models with MoE and Mamba2 as well) Nov 7, 2025
@rkazants rkazants changed the title [OpenVINO] Add support for Granite-4.0 family (incl. models with MoE and Mamba2 as well) [OpenVINO] Add support for Granite-4.0 family (incl. models w. MoE and Mamba2) Nov 7, 2025
@rkazants rkazants changed the title [OpenVINO] Add support for Granite-4.0 family (incl. models w. MoE and Mamba2) [OpenVINO] Add support for Granite-4.0 family (incl. models with MoE and Mamba2) Nov 7, 2025
@rkazants rkazants added the openvino-slow Runs OpenVINO slow tests with different versions of transformers label Nov 9, 2025
@rkazants rkazants removed the openvino-slow Runs OpenVINO slow tests with different versions of transformers label Nov 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants