Would it be possible to support Qwen3-Embedding-0.6B?

[Roba1993]

I'm trying to use model2vec for a project of mine because of the small size and speed. The results unfortunately are not that good yet. Qwen3 is performing really well on my tests. Is there a way to get it supported here?

[Pringled]

Hi @Roba1993, Qwen3 is already supported (e.g. you can distill any of the Qwen models using Model2Vec). However, I did run these myself and saw that indeed the performance is not that great compared to some other models. It could be because they are instruction tuned models, and we don't use those during distillation, or because of the pooling we use. It's something we would need to look into though. For now, I would suggest using models that work well (e.g. the BGE models which we use for our own Potion models).

[jpzhangvincent]

Same.. I also found Qwen3 embedding 0.6B also doesn't work well.

[sarim-zafar]

Just putting this out there to make testing faster: https://huggingface.co/1sarim/Qwen3-Embedding-0.6B-m2v

[puppetm4st3r]

same here, performance is degradated, is there an option to apply an instruct template on destilation process?

[stephantul]

Hello!

We're working on this. We are currently able to improve performance of modernBERT massively due to tokenizer improvements. But the same improvements unfortunately do not lead to Qwen being improved.

My bet is not the instruction tuning, but the fact that we perform mean pooling, while Qwen uses last token pooling, as indicated in their README: https://huggingface.co/Qwen/Qwen3-Embedding-0.6B

I think we could add last token pooling, but unfortunately we don't have a mechanism to detect whether a given model uses last token pooling or not. So I'm not really sure how we should proceed.

[puppetm4st3r]

Hello!

We're working on this. We are currently able to improve performance of modernBERT massively due to tokenizer improvements. But the same improvements unfortunately do not lead to Qwen being improved.

My bet is not the instruction tuning, but the fact that we perform mean pooling, while Qwen uses last token pooling, as indicated in their README: https://huggingface.co/Qwen/Qwen3-Embedding-0.6B

I think we could add last token pooling, but unfortunately we don't have a mechanism to detect whether a given model uses last token pooling or not. So I'm not really sure how we should proceed.

why not a setting in the constructor with default value to mean pool, for models like qwen could be changed to last, but it is always responsability of the user to kwnow what pool is best for that model, for the destiled result maybe a key in the config.json to kwno what pool must be use when destil version will be instanciated.

[stephantul]

@puppetm4st3r I agree that this is the only way to go. I'm still hesitant because in my experience most users do not really read the docs and then open an issue or are disappointed. But having the option is better than not having it, I suppose. I'll take a stab it when I have time.

[Pringled]

Hi @Roba1993 @jpzhangvincent @puppetm4st3r ,

I've just added support for configurable pooling, see #288. This is currently on main, but I'll release it next week. If you want, you can already try it from main. Based on my testing, this fixes the performance issues with Qwen3, and most likely other decoder-style based embedding models as well. You can do the following:

model = distill(model_name="Qwen/Qwen3-Embedding-0.6B", pooling="last")

And this will lead to the following performance gains:

| Model                  | Average (All) | Average (MTEB) | Classification | Clustering | PairClassification | Reranking | Retrieval | STS   | Summarization | PEARL | WordSim |
|------------------------|---------------|----------------|----------------|------------|--------------------|-----------|-----------|-------|---------------|-------|---------|
| qwen3_last_token_pooling | 43.56         | 42.39          | 58.46          | 27.03      | 68.50              | 45.47     | 19.58     | 66.34 | 30.56         | 48.63 | 46.43   |
| qwen3_mean_pooling       | 20.81         | 21.16          | 32.30          | 14.47      | 32.98              | 33.20     | 1.90      | 34.94 | 28.70         | 27.76 | 9.00    |

[puppetm4st3r]

nice! would be awesome if you can upload the qwen3 model to the huggingface repo!!!!

[Pringled]

@puppetm4st3r Sure! I pushed a base distill here: https://huggingface.co/Pringled/m2v-Qwen3-Embedding-0.6B. Let me know if you have any feedback!

[Pringled]

Closing since this is now supported.