Distillation
#104
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
-
Hi! Does anyone have any details on how DeepSeek distilled R1 into smaller models? In the technical report, they provide just no information except for saying that they used SFT and the dataset of 800k examples used to train R1.
If they just do SFT of Qwen on these examples, it's not distillation. Distillation would be if they used R1 to generate scores and then finetuned Qwen to simulate these scores.
Who knows something?
Beta Was this translation helpful? Give feedback.
All reactions