New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A problem about mmlu score #1

Open

Haruka1307 opened this issue Dec 24, 2024 · 1 comment

Haruka1307 commented Dec 24, 2024

Hi!

I noticed that mmlu on official report achieves 66.7 higher than whole data sft and selected data sft,how to explain this ?

Collaborator

fannie1208 commented Jan 2, 2025

Hi,

Thank you for your question.
In their report, they use macro_avg/acc_char as evaluation metric while we use em as metric in our paper.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment