[Bug]: `get_input_embeddings` not auto‑handled for Qwen3OmniMoeForConditionalGeneration

[Steven0236]

⚙️ Your current environment

The output of python collect_env.py

### Environment Information ###
Operating System: `Linux-6.16.5-100.fc41.x86_64-x86_64-with-glibc2.40`
Python Version: `3.12.11 (main, Jun 12 2025, 00:00:00) [GCC 14.3.1 20250523 (Red Hat 14.3.1-1)]`
llm-compressor Version: `0.7.1`
compressed-tensors Version: `0.11.0`
transformers Version: `4.57.0.dev0`
torch Version: `2.8.0`
CUDA Devices: `['NVIDIA GeForce RTX 5070 Ti', 'NVIDIA GeForce RTX 5070 Ti', 'NVIDIA GeForce RTX 5070 Ti', 'NVIDIA GeForce RTX 5070 Ti']`

🐛 Describe the bug

Trying to quantize the new Qwen3-Omni model and getting the following error. Not sure if the root cause is in the transformers library code or in llm-compressor.

  File ".../quantize.py", line 62, in quantize_model
    oneshot(model=model, recipe=recipe) # Applies the quantization recipe
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...

  File ".../llmcompressor/entrypoints/utils.py", line 65, in pre_process
    untie_word_embeddings(model_args.model)
  File ".../llmcompressor/transformers/sparsification/compressed_tensors_utils.py", line 131, in untie_word_embeddings
    input_embed = model.get_input_embeddings()
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../transformers/modeling_utils.py", line 1802, in get_input_embeddings

NotImplementedError: `get_input_embeddings` not auto‑handled for Qwen3OmniMoeForConditionalGeneration; please override in the subclass.

Can anyone help out?

🛠️ Steps to reproduce

...
model = Qwen3OmniMoeForConditionalGeneration.from_pretrained(modelpath, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(modelpath)
ignore_list = 'lm_head'.split()
recipe = QuantizationModifier(targets="Linear", scheme='FP8_DYNAMIC', ignore=ignore_list)
oneshot(model=model, recipe=recipe)

[brian-dellabetta]

Same as error reported in #1868 , will look into this

[brian-dellabetta]

Hi @Steven0236 , I have found the bug. I can open a PR tomorrow, one way to fix until it lands is to set your oneshot call to

oneshot(model=model, recipe=recipe, tie_word_embeddings=True)

lmk if that still errors out for you

[Steven0236]

Many thanks @brian-dellabetta. That workaround seems to work.
I'll leave this open so that others can see the workaround until the PR is included.

[juju812]

tie_word_embeddings=True workaround works for quant scene without calibration data.

If the quant scene changed to awq, another error will happen due to the absence of forward method in Qwen3OmniMoeForConditionalGeneration or Qwen2_5OmniForConditionalGeneration.

I've tried to quant Qwen2_5OmniThinkerForConditionalGeneration only, as per #1673 (comment) did. But quanted thinker can't be loaded by vllm.

@brian-dellabetta will this modelling adaption issue be fixed? Otherwise maybe one complete workaround is need for Qwen Omni models.

[brian-dellabetta]

Hi @juju812 , you should be targeting the Thinker portion of the model, I don't think the full omni models will pass through the entire oneshot.

Are you saying you can load the Thinker model up in vllm, but the quantized version is failing? Are you using the instructions here?

[brian-dellabetta]

Note that the resolution for this get_input_embeddings issue is in progress, you can track it here

• Fixing untie to be used only as needed and automatic #1963

Issues with compressing and serving the qwen omni models might persist downstream.

[juju812]

Hi, @brian-dellabetta,

Sorry for the late reply. The actual problem is the model config generated by llm-compressor after quantizing Qwen2_5OmniThinkerForConditionalGeneration can't be recognized by vllm/transformers. Detailed error message as below.

I'm trying to migrate thinker config and weight to original Qwen2_5OmniForConditionalGeneration model. And before that, I choose llm-compressor to generate FP8_DYNAMIC model, and autoawq for AWQ one.

error while loading `Qwen2_5OmniThinkerForConditionalGeneration`

(APIServer pid=37312) Traceback (most recent call last):
(APIServer pid=37312) File "/usr/local/bin/vllm", line 10, in
(APIServer pid=37312) sys.exit(main())
(APIServer pid=37312) ^^^^^^
(APIServer pid=37312) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 54, in main
(APIServer pid=37312) args.dispatch_function(args)
(APIServer pid=37312) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 50, in cmd
(APIServer pid=37312) uvloop.run(run_server(args))
(APIServer pid=37312) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 109, in run
(APIServer pid=37312) return __asyncio.run(
(APIServer pid=37312) ^^^^^^^^^^^^^^
(APIServer pid=37312) File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=37312) return runner.run(main)
(APIServer pid=37312) ^^^^^^^^^^^^^^^^
(APIServer pid=37312) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=37312) return self._loop.run_until_complete(task)
(APIServer pid=37312) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37312) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=37312) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 61, in wrapper
(APIServer pid=37312) return await main
(APIServer pid=37312) ^^^^^^^^^^
(APIServer pid=37312) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1941, in run_server
(APIServer pid=37312) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=37312) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1961, in run_server_worker
(APIServer pid=37312) async with build_async_engine_client(
(APIServer pid=37312) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37312) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=37312) return await anext(self.gen)
(APIServer pid=37312) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37312) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 179, in build_async_engine_client
(APIServer pid=37312) async with build_async_engine_client_from_engine_args(
(APIServer pid=37312) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37312) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=37312) return await anext(self.gen)
(APIServer pid=37312) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37312) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 205, in build_async_engine_client_from_engine_args
(APIServer pid=37312) vllm_config = engine_args.create_engine_config(usage_context=usage_context)
(APIServer pid=37312) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37312) File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1119, in create_engine_config
(APIServer pid=37312) model_config = self.create_model_config()
(APIServer pid=37312) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=37312) File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 963, in create_model_config
(APIServer pid=37312) return ModelConfig(
(APIServer pid=37312) ^^^^^^^^^^^^
(APIServer pid=37312) File "/usr/local/lib/python3.12/dist-packages/pydantic/_internal/_dataclasses.py", line 123, in init
(APIServer pid=37312) s.pydantic_validator.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
(APIServer pid=37312) pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
(APIServer pid=37312) Value error, The checkpoint you are trying to load has model type qwen2_5_omni_thinker but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
(APIServer pid=37312)
(APIServer pid=37312) You can update Transformers with the command pip install --upgrade transformers. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command pip install git+https://github.com/huggingface/transformers.git [type=value_error, input_value=ArgsKwargs((), {'model': ...rocessor_plugin': None}), input_type=ArgsKwargs]
(APIServer pid=37312) For further information visit https://errors.pydantic.dev/2.11/v/value_error

[brian-dellabetta]

Hi @juju812 , does the AutoAWQ model load and run in vllm?

[juju812]

Hi @juju812 , does the AutoAWQ model load and run in vllm?

Yes, for autoawq is running based on the entire model Qwen2_5OmniForConditionalGeneration, it can be loaded by vllm.

[brian-dellabetta]

Thanks @juju812 for the information, I will try this out tomorrow and ask around.

[HDCharles]

#1997 should have fixed this issue but i think there may be further issues with Omni

[brian-dellabetta]

With tie_word_embeddings=True, this is erroring out during subgraph tracing for me.

File "/home/brian-dellabetta/projects/llm-compressor/src/llmcompressor/pipelines/sequential/pipeline.py", line 70, in __call__
    subgraphs = trace_subgraphs(model, sample_input, sequential_targets, ignore)

I am going to tag this with the tracing label, we are refactoring the current implementation to be more robust. Will ping here with status updates. cc @kylesayrs