Benchmark uses unsupported temperature parameter with reasoning models

[messimm]

Description

The benchmark currently passes the temperature parameter to all models by default.

However, reasoning models such as gpt-5 do not support custom temperature values and only allow the default value. As a result, evaluation fails with errors like:

Error code: 400 - Unsupported value: 'temperature' does not support 0.0 with this model.

gpt-5 is used as model by default.

Expected behavior

Do not send temperature (and similar sampling parameters) to reasoning models.

[123sio]

In our setup, temperature can be passed this way on the API platform we use, and the benchmark simply follows that interface.
You can find the specific API details here: https://yunwu.apifox.cn/api-232421952.

[messimm]

In our setup, temperature can be passed this way on the API platform we use, and the benchmark simply follows that interface.
You can find the specific API details here: https://yunwu.apifox.cn/api-232421952.

Thanks for the clarification!

I think there are two important points worth highlighting (possibly also documenting in the issue or benchmark description):

We are using different APIs. Even for the same underlying model and task, different API platforms (and their parameter handling) can lead to noticeably different results. This difference in API interfaces and implementations can affect benchmark comparability, so it might be useful to explicitly note this on GitHub.

Not everyone has access to Chinese API platforms. For this reason, it would be very interesting to see the results of Kandinsky 5.0 (2B and 19B) in the main leaderboard using your API setup, so that the results are directly comparable with other models evaluated through the same interface. Kandinsky 5.0 is currently the strongest open-source model on LMArena for video generation, so such a comparison could be quite valuable for the community.

I’d be very happy to collaborate on this. I can either provide pre-generated videos or share a link to the model along with detailed instructions for running it locally, depending on what works best for you.

Looking forward to your thoughts!

[123sio]

Thanks for the detailed follow-up — these are all fair points.

You’re right that some configuration parameters are not supported in a uniform way across all API platforms. We agree this is a real compatibility issue. We’ll update the code to better handle this case so that the benchmark can work more smoothly across different APIs.

Regarding Kandinsky, we really appreciate your interest in the benchmark and your willingness to collaborate. We do think this could be valuable. That said, for us to evaluate it in a consistent and comparable way, we would prefer either access to an API interface for the model or detailed instructions for running it locally. This would make integration into our existing evaluation pipeline much more straightforward.

[messimm]

Hi!

Please refer specifically to the I2V inference section in the README — it contains detailed guidance on how to run image-to-video generation.

There are two model versions available:

Lite — ~2B parameters

Pro — ~19B parameters

For the Pro model, we recommend running it with the HD configuration (see the corresponding config name in the repository). This setup provides the best quality.

If GPU resources are limited, you can instead use the sd configuration, which is more memory-efficient.

The exact configuration names and paths to the config files are listed in the tables in the README.

Let us know if you need any additional details 🙂

[messimm]

Sorry, I missed the most important bit — here’s the link to the model:
https://github.com/kandinskylab/kandinsky-5

[123sio]

We have run evaluation for Kandinsky 5.0 I2V Pro HD 5s on our benchmark using a single RTX PRO 6000 GPU for generation. The corresponding results have now been added to the Scoreboard.

[messimm]

Thank you very much for running the evaluation and for adding the results to the Scoreboard — really appreciate the effort.

Just to clarify one detail: in your message you mention Kandinsky 5.0 I2V Pro HD 5s, but in the leaderboard the model is listed as Lite. Could you please confirm which version was actually used for the evaluation?

[123sio]

Sorry about that — we mentioned the wrong model version in our previous reply. The model we actually evaluated was Kandinsky 5.0 I2V Lite 5s, using k5_lite_i2v_5s_sft_sd.yaml.

The reply contained the wrong version name, but the Scoreboard is already correct.

[messimm]

What about the larger 19B Pro model? As far as I understand, the RTX PRO 6000 has 96 GB of memory and uses the Blackwell architecture, so it seems like it should be sufficient to run the larger model as well.

[123sio]

We’ve now run evaluation for the larger Kandinsky 5.0 I2V Pro model as well, and the results have been updated in the Scoreboard.

[messimm]

Great Work! Thanks!

[123sio]

Glad it’s helpful! If you find the benchmark useful in your work, feel free to cite our paper