How to set generation sampling parameters?

[rawsh]

Need to use deepseek reference settings of temperature=0.6, top_p=0.95.

Greedy sampling does poorly on AIME:

r1-1.5B

• AIME24: 23.33%

Tried to refer to lighteval docs and ran into issues using model config:

model: # Model specific parameters
  base_params:
    model_args: "pretrained=Qwen/Qwen2.5-7B-Instruct,dtype=bfloat16,max_model_length=768,gpu_memory_utilisation=0.7" # Model args that you would pass in the command line
  generation: # Generation specific parameters
    temperature: 1.0
    stop_tokens: null
    truncate_prompt: false

run with:

TASK=aime24 lighteval vllm \
    "config.yaml" \
    "custom|$TASK|0|0" \
    --custom-tasks tasks.py \
    --use-chat-template \
    --output-dir ./results/

hitting:

TypeError: expected str, bytes or os.PathLike object, not dict

ref

[rawsh]

Ok figured this out, would a PR for a python pipeline + optional generation params be helpful?

[rawsh]

r1-1.5B (temp=0.6, top_p=0.95)

AVG: 44.98%

• AIME25: 20%
• AIMO-AIME: 21.11%
• AIMO-AMC: 67.47%
• MATH_500: 82%
• GPQA Diamond: 34.34%

[oasis-0927]

So how did you manage to set those parameters? I tried to change MODEL_ARGS in the evaluation script but it didn't work. Thx.

[oasis-0927]

So how did you manage to set those parameters? I tried to change MODEL_ARGS in the evaluation script but it didn't work. Thx.

NVM I found that using the d4e6f59bd106d4cf2fda0f8fae3dadafe557515b commit version works fine.