首先给项目点赞！这是一个很好的全模态能力尝试项目，我将模型量化，以及测试情况做个汇报，还有一些报错问题跟项目团队请教，谢谢！

[wikeeyang]

测试环境：Ubuntu 22.04，128GB RAM，Python 3.11，CUDA 12.4，Torch 2.6.0，RTX 4090 24GB 单显卡。
测试模型：Ming-Lite-Omni-1.5，量化方式：NF4 与 bf16 相结合。

首先分析了模型结构、权重和精度，个人认为应该可以通过量化，把项目在单卡 24GB 显存内跑起来，但不确定量化对输出质量的影响有多大。
在测试过程中，除了视频内容识别 OOM 外，全部模型加载后，现在基本在 20-23GB 左右。
在测试过程中，发现 repo 中的代码，很多还是 v1.0 时候的代码，好像还没有完全根据 v1.5 的升级，将相应代码全部调整完成。

模型刚加载完成后，显存占用 20GB 左右，如下：

基础文字对话功能正常：

图像识别对话功能正常：

视频识别对话功能 OOM：

语音识别对话功能正常：

文生图功能正常：

文字输入，语音输出，或者语音对话功能报错，好像是 talker 部分模型加载问题：

报错日志：

history: [(('/home/tkadm/Ming/temp/e8b1237d57a5922949fe61c6bca802ae2ddd7d63c4159763d08acaa7aaef4683/speechQA_sample.wav',), None)]
[{'role': 'HUMAN', 'content': [{'type': 'audio', 'audio': '/home/tkadm/Ming/temp/e8b1237d57a5922949fe61c6bca802ae2ddd7d63c4159763d08acaa7aaef4683/speechQA_sample.wav'}]}]
Traceback (most recent call last):
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/queueing.py", line 715, in process_events
response = await route_utils.call_process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/route_utils.py", line 322, in call_process_api
output = await app.get_blocks().process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/blocks.py", line 2220, in process_api
result = await self.call_function(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/blocks.py", line 1743, in call_function
prediction = await utils.async_iteration(iterator)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/utils.py", line 739, in async_iteration
return await anext(iterator)
^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/utils.py", line 733, in anext
return await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/utils.py", line 716, in run_sync_iterator_async
return next(iterator)
^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/utils.py", line 877, in gen_wrapper
response = next(iterator)
^^^^^^^^^^^^^^
File "/home/tkadm/Ming/gradio_demo_me-old.py", line 344, in chat_predict
text, audio_path, image_path = generate(model, processor, messages, state, use_audio_response=use_audio_response)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/Ming/gradio_demo_me-old.py", line 252, in generate
audio_path = text_to_speach(model, text, outputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/Ming/gradio_demo_me-old.py", line 151, in text_to_speach
audio_detokenizer = AudioDetokenizer(
^^^^^^^^^^^^^^^^^
File "/home/tkadm/Ming/modeling_bailing_talker.py", line 661, in init
self.model.load(flow_model_path, hifigan_model_path)
File "/home/tkadm/Ming/audio_detokenizer/cli/flow_stream_model.py", line 39, in load
self.flow.load_state_dict(torch.load(flow_model, map_location=self.device), strict=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/torch/serialization.py", line 1470, in load
raise pickle.UnpicklingError(_get_wo_message(str(e))) from None
_pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, do those steps only if you trust the source of the checkpoint.
(1) In PyTorch 2.6, we changed the default value of the weights_only argument in torch.load from False to True. Re-running torch.load with weights_only set to False will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source.
(2) Alternatively, to load with weights_only=True please check the recommended steps in the following error message.
WeightsUnpickler error: Unsupported global: GLOBAL pathlib.PosixPath was not an allowed global by default. Please use torch.serialization.add_safe_globals([PosixPath]) or the torch.serialization.safe_globals([PosixPath]) context manager to allowlist this global if you trust this class/function.

Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html.

图像编辑功能报错，好像是图片尺寸问题：

错误日志如下：

history: [(('/home/tkadm/Ming/temp/16da8d87c405652ce67ca4fe9eb661562eb47afb9ff1b0a3cd872bd3a9a5a1d0/cake.jpg',), None), ('Add a candle on top of the cake', None)]
[{'role': 'HUMAN', 'content': [{'type': 'image', 'image': '/home/tkadm/Ming/temp/16da8d87c405652ce67ca4fe9eb661562eb47afb9ff1b0a3cd872bd3a9a5a1d0/cake.jpg'}, {'type': 'text', 'text': 'Add a candle on top of the cake'}]}]
Traceback (most recent call last):
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/queueing.py", line 715, in process_events
response = await route_utils.call_process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/route_utils.py", line 322, in call_process_api
output = await app.get_blocks().process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/blocks.py", line 2220, in process_api
result = await self.call_function(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/blocks.py", line 1743, in call_function
prediction = await utils.async_iteration(iterator)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/utils.py", line 739, in async_iteration
return await anext(iterator)
^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/utils.py", line 733, in anext
return await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/utils.py", line 716, in run_sync_iterator_async
return next(iterator)
^^^^^^^^^^^^^^
File "/home/tkadm/miniconda3/envs/video/lib/python3.11/site-packages/gradio/utils.py", line 877, in gen_wrapper
response = next(iterator)
^^^^^^^^^^^^^^
File "/home/tkadm/Ming/gradio_demo_me-old.py", line 344, in chat_predict
text, audio_path, image_path = generate(model, processor, messages, state, use_audio_response=use_audio_response)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/Ming/gradio_demo_me-old.py", line 256, in generate
image_path = generate_image(model, processor, messages, has_audio=has_audio)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tkadm/Ming/gradio_demo_me-old.py", line 133, in generate_image
image = model.generate(
^^^^^^^^^^^^^^^
TypeError: modeling_bailingmm.BailingMMNativeForConditionalGeneration.generate() got multiple values for keyword argument 'image_gen_width'

烦请项目团队帮忙分析一下错误原因，谢谢！
由于时间关系，还没来得及详细阅读分析代码，可能很多理解上还不够到位，请项目团队指点。
待代码全部测试通过后，我将把量化后的模型，上传到 https://huggingface.co/wikeeyang 本人的模型目录下分享。

[wikeeyang]

经分析 cookbook.ipynb 代码，比对 test_audio_tasks.py 代码是适合 v1.5 版本的，gradio_demo.py 部分代码好像暂时未更新。
结合前面的报错提示，由于我环境的 torch 版本较高，为2.6.0，须修改：audio_detokenizer/cli/flow_stream_model.py 中第 41 行：
self.flow.load_state_dict(torch.load(flow_model, map_location=self.device)["model"], strict=True)
改为：
self.flow.load_state_dict(torch.load(flow_model, map_location=self.device, weights_only=False)["model"], strict=True)
运行 test_audio_tasks.py 测试正常，日志如下：

Loading checkpoint shards: 100%|███████████████████████████████████| 3/3 [00:23<00:00, 7.97s/it]
Multiple distributions found for package optimum. Picked distribution: optimum-quanto
2025-07-28 09:31:36,633 INFO input frame rate=50
2025-07-28 09:31:38,518 INFO input frame rate=50
Testing ASR...
The seen_tokens attribute is deprecated and will be removed in v4.41. Use the cache_position model input instead.
2025-07-28 09:31:41,880 WARNING The input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in torch.bfloat16.
Chinese 现在是不是也该长点心了吧
Generate time: 2.85s
Testing Speech QA...
From v4.47 onwards, when a model cache is to be returned, generate will return a Cache instance instead by default (as opposed to the legacy tuple of tuples format). If you want to keep returning the legacy format, please set return_legacy_cache=True.
Starting from v4.46, the logits model output will have the same type as the model (except at train time, where it will always be FP32)
We detected that you are passing past_key_values as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate Cache class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)
Generate time: 21.93s
SPeech QA 文本响应：你好！我是一个多模态模型，由蚂蚁集团研发。我可以回答你的问题，帮助你解决疑惑。请问有什么我可以帮助你的吗？
SPeech QA 音频响应已保存到 out.wav
Testing TTS...
2025-07-28 09:32:07,383 INFO for last sentence
Generate time: 4.84s
TTS 音频响应已保存到 out_tts.wav

[wikeeyang]

经分析图像编辑部分的报错原因，以及 generate_image() 函数的代码，猜测是因为传递了多余的图像尺寸参数所致，因此，修改 gradio_demo.py 脚本的 generate_image() 函数中传递图像生成参数部分的代码：
image_gen_param = {
"image_gen_cfg": 6.0,
"image_gen_steps": 20,
"image_gen_width": 512,
"image_gen_height": 512
}
将 "image_gen_width": 512, 和 "image_gen_height": 512 两行注释掉，重新跑这部分的测试，结果正常，图片如下：

输出日志如下：

history: [(('/home/tkadm/Ming/temp/16da8d87c405652ce67ca4fe9eb661562eb47afb9ff1b0a3cd872bd3a9a5a1d0/cake.jpg',), None), ('Add a candle on top of the cake', None)]
[{'role': 'HUMAN', 'content': [{'type': 'image', 'image': '/home/tkadm/Ming/temp/16da8d87c405652ce67ca4fe9eb661562eb47afb9ff1b0a3cd872bd3a9a5a1d0/cake.jpg'}, {'type': 'text', 'text': 'Add a candle on top of the cake'}]}]
[Hyper-Param]:
do_img_cfg: False
guidance_scale: 6.0
img_cfg: 1.0
cfg_mode: 1
100%|███████████████████████████████████████████████████████| 20/20 [00:01<00:00, 10.40it/s]

至此，按项目的 example 示例，对量化模型的全部功能测试基本跑通，量化模型会尽快上传至：

https://huggingface.co/wikeeyang/Ming-Lite-Omni-v1.5-NF4

该项目的确是不错的项目，再次点赞 👍👍👍，欢迎有兴趣的朋友，大家一起测试研究，交流学习。。。

[mingyi456]

Oh no, even a 4 bit NF4 quantization of the model takes up 20GB of VRAM? I was hoping 8 bit precision would be enough to squeeze into 24GB of VRAM.

I see on other issues the audio output module takes up 19GB of VRAM at 16 bits, and can easily be disabled according to the developers. I guess the 20GB VRAM usage with 4 bit quantization is with the talker module enabled? And it should be possible to run an 8 bit quant in 24GB of VRAM as long as the talker module is disabled?

Regarding the OOM issue with video analysis, I suspect there might not be an easy way around it. When I tried running Qwen-2.5-VL-7b at full precision with the transformers library, it also OOMs very easily with videos, unless you heavily restrict the video length and resolution.

Finally, have you tried other methods of quantization? It seems for image generation models like Flux, even a naive Q4_0 GGUF quant seems to produce outputs closer to 16 bit compared to NF4. and GGUF has significantly better K and IQ quants as well.

[wikeeyang]

@mingyi456 NF4 只是针对主模型部分，connector 和 transformer 部分并未量化，24GB 能跑(不含 Video 解析)是全部模型加载，能在一个 Web 页面上测试全部功能。文中曾经提到的不加载部分模型，可以节省 19GB 之类的说法，适用于 v1.0 版本模型，v1.5 版本模型已经有了调整。另外，针对 gguf 之类的量化方式，根据我的经验，llama.cpp 应该暂时不支持此类新的 moe 模型。另外，video analysis 需要显存加大，OOM 可以理解，可以通过上传较短的 1-2 秒的视频，估计能跑起来。

@orchidmalevolence 麻烦项目团队，能否抽空更新一下 gradio_demo.py 中，与 text_to_speech 部分相关的代码，我大致比较了一下，这部分代码，v1.0 与 v1.5 在调用函数上好像有较大差异，由于本人代码能力有限，这部分功能，在 gradio_demo.py 还是未能实现，谢谢！

[davidjohnstone]

@wikeeyang Is your Hugging Face repository missing the safetensors files?

@mingyi456 I suspect it would be profitable to run the MoE part on CPU because this is using Ling-Lite, which is most of the size of the model and it runs pretty fast just on CPU. (I haven't used this yet, but the audio generation part of Qwen 2.5 Omni takes up most of the time in generation and I assume this is similar.)

[wikeeyang]

@davidjohnstone 感谢您的快速回复！🤣，是的，很不好意思，上班时间公司连外网有限制，无法上传大文件，我晚上会上传上去。

另外，整个 gradio_demo 现在只剩下 “文本-语音” 与“语音-语音” 对话部分未能跑通了，我看了一下，v1.5版本的 modeling_bailing_talker.py 与 v1.0 版本的 modeling_bailing_talker.py 在 token2wav() 函数上很大区别，这个暂时还没理解透代码，期待您的指点，谢谢！

[mingyi456]

NF4 只是针对主模型部分，connector 和 transformer 部分并未量化，24GB 能跑(不含 Video 解析)是全部模型加载，能在一个 Web 页面上测试全部功能。

Sorry if I am asking a stupid question, but what do you mean by not quantizing the transformer part? I thought almost the entire model is a transformer?

文中曾经提到的不加载部分模型，可以节省 19GB 之类的说法，适用于 v1.0 版本模型，v1.5 版本模型已经有了调整。

From what I see, the lines of code referenced in #13 (comment) are no longer relevant, but I believe the code was simply refactored, and it might still be possible to manually disable the talker module.

另外，针对 gguf 之类的量化方式，根据我的经验，llama.cpp 应该暂时不支持此类新的 moe 模型。

I am aware that llama.cpp does not support this model architecture (and even if future support was added, it would likely be limited to image and text input, with text-only output). But I am talking about llama.cpp, I am talking about using GGUF as a quantization format only. This repo allows the use of GGUF as a quantization format for diffusion models in ComfyUI, and there is even some level of support for loading these GGUF quants in the diffusers library. For example, Flux, HiDream, Wan 2.1, and even Bagel, OmniGen2 and Chatterbox have GGUF quantizations available. Although I guess we might need @city96 to implement support manually, and even then it might still need to be merged into the transformers library.

@davidjohnstone From my experience, you really want to use flash attention to reduce the size of the KV cache when utilizing multimodal capabilities, but the flash-attn package only works on Ampere GPUs and newer (and no CPUs are supported at all). So running it on CPU might not even be worth it.

[wikeeyang]

@mingyi456 您好！Ming-Lite-Omni 模型分成几个部分，主模型是模型根目录下面的部分，应该是 Ling-lite-1.5 的一个 MOE 模型，有 LLM、Audio、Vision 等不同层混合，其它子目录如 connector / mlp / talker / transformer / vae 等，太详细的我也说不清楚，如果您需要有更深入的了解，可以麻烦官方解答一下。🤣
至于 GGUF quantization，我研究不深，大致以为 llama.cpp 主要支持各类 LLM，City96 大佬做了 patch，支持 VLM 和 DiT，您说的没错，其实这里面就是模型参数的定义问题，不过我这方面知识有限，不知道如何将模型参数映射过去。🤣

[wikeeyang]

采用未合并的 gradio_demo.py 最新脚本，完成了所有测试，感谢项目团队的快速更新。👍👍👍
NF4 量化 Ling-lite-1.5 主模型后，的确让模型在声音质量和视觉能力方面有较大的损失，但依然能完成基本的功能。
在不加载文生图模型的情况下，18-24GB 的显存，即可处理分斌率 640 以内的短视频内容，但 NF4 量化的确损失较大，可以考虑采用 8bit 的量化方式加载。

从测试的情况来看，基座模型 Ling-lite-1.5 和 Omni 模型的多模态能力还是非常棒的，即便 NF4 量化的情况下，依然能生成将近 2 分钟的语音，而且吐字基本清晰，也基本没有读错。

从整个项目学到不少东西，希望项目团队持续改进优化，如果能开源数据集和训练微调的方法，那就太好了！

最后，再次给优秀的项目及团队点赞。。。 👍👍👍

[LandyGuo]

感谢使用及反馈🙏 PR #43 更新了 gradio demo 以适配 Ming-lite-omni v1.5，修复了gradio demo使用的问题。

为便于开源社区部署模型及使用, 我们近期会开源基于vllm推理优化和量化后的版本，同时也会尽快推出适合在消费级显卡上使用的更轻量的模型。

[wikeeyang]

感谢蚂蚁集团项目团队的辛勤劳动！给我们带来优秀的项目，这是目前全模态模型里面集成功能最丰富的，很不容易。👍👍👍
的确，vLLM 框架在 fp8 精度方面有更好的优化，根据我测试的情况，fp8 精度下，audio 和 vision 部分损失较小，结合部分模型的动态加载、卸载技术，项目目前的架构也完全可以在消费级显卡上平滑运行，从而让更多的社区兴趣爱好者体验全模态模型的魅力。

[dfengpo]

请问能把Audio部分拆分出来，再导出onnx版本吗？让没有强大硬件的普通的凡人也能享受到优秀项目的快乐

[mingyi456]

@dfengpo From my testing, it seems that completely disabling the talker portion of the model only saves a bit of VRAM, around 2GB if I remember correctly, and so converting that portion to onnx (and presumably quantizing it) will save only 1-1.5GB. This is nowhere near enough to allow it to squeeze into a consumer friendly GPU. This applies to the v1.5 model, and it seems that the older v1 version can save up to 19GB this way for some reason.
If you want to save more VRAM, disabling image generation will save you about 6GB of VRAM. From what I see, the transformer component of the model refers to the image generation submodule (since it is a DiT), and it seems to be using SD3 or SD3.5 Medium as the base architecture, while the older v1 model seems to use some form of Sana instead. Basically my point is that the output image quality is unlikely to be groundbreaking, although prompt adherence and understanding might be better, and of course it is still very cool to have a single model cover all these modalities at once.

[wikeeyang]

请问能把Audio部分拆分出来，再导出onnx版本吗？让没有强大硬件的普通的凡人也能享受到优秀项目的快乐

@dfengpo 等 Lite 版本吧，新的 Ming-flash-omni-Preview 有 104B，Ling-Flash 部分太大，再量化也没用，Lite 版本如果有的话，照以前是十几B 的大小，可以玩玩。。。🤣🤣🤣