Skip to content

Pull requests: huggingface/lighteval

New pull request New
131 Open 676 Closed
131 Open 676 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Add a regression test for truncated-then-full sample-cache runs
#1291 opened Jul 8, 2026 by shipbehaves Contributor Loading…
1
fix(normalizations): exclude -1 padding from token-norm length + guard mismatch (#1170)
#1290 opened Jul 8, 2026 by WatchTree-19 Loading…
Add Armenian language evaluation suite (ArmBench-LLM)
#1289 opened Jul 8, 2026 by bugdaryan Loading…
Add unit tests for VLLMModel sampling params and config wiring
#1287 opened Jul 7, 2026 by jayzuccarelli Loading…
1
Batch tokenization in the transformers backend's scoring loop
#1286 opened Jul 6, 2026 by ErenAta16 Loading…
1
Add sample count to the results summary table
#1285 opened Jul 6, 2026 by ErenAta16 Loading…
1
Add support for selecting few-shot examples by id
#1284 opened Jul 6, 2026 by ErenAta16 Loading…
1
Make doc builds faster
#1283 opened Jul 6, 2026 by mishig25 Loading…
1
add missing docstrings to three public functions in utils/imports.py
#1282 opened Jul 4, 2026 by RavSinghChandan Loading…
feat: add sample count column to results table
#1280 opened Jul 2, 2026 by SnehaDeshmukh28 Loading…
1
Honor the model's generation_config.eos_token_id in the transformers backend
#1279 opened Jul 2, 2026 by waqaskhan137 Loading…
1
Fix reversed gold index in boolq:contrastset prompt
#1277 opened Jun 29, 2026 by vineethsaivs Loading…
Fix CorpusLevelF1Score(None) to report positive-class F1, not max-per-class
#1275 opened Jun 25, 2026 by iamsharduld Loading…
Fix MajAtN IndexError when the gold is not the first choice
#1274 opened Jun 25, 2026 by iamsharduld Loading…
Fix helm_normalizer mis-scoring numbers (homogeneize before remove_punc)
#1273 opened Jun 25, 2026 by iamsharduld Loading…
Add script-aware language tags for FLORES 200
#1272 opened Jun 23, 2026 by goktugozkanmd Loading…
1
fix(mmlu_pro): correct answer letters in prompt and choices
#1269 opened Jun 19, 2026 by vineethsaivs Loading…
1
feat: add static linter command for custom tasks
#1268 opened Jun 19, 2026 by Kesav2k04 Loading…
1
feat: add ID-based few-shot example selection (#634)
#1263 opened Jun 16, 2026 by sohammmmm10 Loading…
Fix KeyError: 'default' for bigbench tasks with nonexistent split
#1262 opened Jun 14, 2026 by nuthalapativarun Loading…
1
2
Document unconditioned_query requirement for PMI normalization
#1261 opened Jun 14, 2026 by nuthalapativarun Loading…
1
2
Fix non-deterministic cache hash for MetricGrouping tasks
#1260 opened Jun 14, 2026 by nuthalapativarun Loading…
1
2
Sync LightevalTaskConfig docstring with current fields
#1259 opened Jun 14, 2026 by nuthalapativarun Loading…
1
2
Fix broken default_tasks.py link in quicktour docs
#1258 opened Jun 14, 2026 by nuthalapativarun Loading…
2
Fix LCB grader failing solutions that read sys.stdin.buffer
#1257 opened Jun 13, 2026 by vineethsaivs Loading…
1
ProTip! Add no:assignee to see everything that’s not assigned.