Skip to content

Issues: huggingface/lighteval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

83 Open 122 Closed
83 Open 122 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[BUG] error setting tokenizer with custom generation params for vllm bug Something isn't working
#563 opened Feb 14, 2025 by rawsh
Issues with torch version other than 2.2.2
#562 opened Feb 14, 2025 by Nevermetyou65
Weird cuda OutOfMemoryError error
#561 opened Feb 14, 2025 by Nevermetyou65
Lighteval endpoint openai feature request New feature/request
#560 opened Feb 13, 2025 by zhaohan-alan
[BUG] Use sampling for greedy mode bug Something isn't working
#559 opened Feb 13, 2025 by tonysy
Handle errors during sympy conversion
#557 opened Feb 13, 2025 by hynky1999
@hynky1999
[BUG] Nanotron runner imports non-existant bug Something isn't working
#555 opened Feb 13, 2025 by jquesnelle
2
how can i use this "community|alghafa:meta_ar_dialects " as a task
#554 opened Feb 12, 2025 by pratim808
[BUG] tiktoken is not optional dependency bug Something isn't working
#546 opened Feb 8, 2025 by hynky1999
couldn't find it in the cached files and it looks like Elron/bleurt-tiny-512, how to set the model path?
#545 opened Feb 8, 2025 by bannima
1
More flexibility in parameters for OpenAI / LiteLLM feature request New feature/request
#544 opened Feb 7, 2025 by satpalsr
2
Better Docs feature request New feature/request
#543 opened Feb 7, 2025 by satpalsr
1
[EVAL] new task
#542 opened Feb 7, 2025 by suraj169Pi
[FT] Faster generation with TransformersModel by using less padding feature request New feature/request
#531 opened Feb 3, 2025 by rolshoven
Question about Request Count and Generative Sampling
#528 opened Jan 31, 2025 by rolshoven
3
updated pypi package with torch>=2.0,<3.0
#526 opened Jan 30, 2025 by zia1138
lighteval with llama3.2 [RuntimeError: No executable batch size found, reached zero.]
#525 opened Jan 29, 2025 by Nevermetyou65
4
[BUG] ImportError: cannot import name 'ExprExtractionConfig' from 'lighteval.metrics.dynamic_metrics' bug Something isn't working
#523 opened Jan 28, 2025 by hlzhang109
5
<think> tags for thinking models
#513 opened Jan 24, 2025 by JoelNiklaus
1
[FT] Enable lazy model initialization feature request New feature/request
#496 opened Jan 11, 2025 by JoelNiklaus
[BUG] support local API? bug Something isn't working
#492 opened Jan 9, 2025 by Jalen-Zhong
2
[FT] Custom model to TransformersModel feature request New feature/request
#489 opened Jan 7, 2025 by Giuseppe5
3
[BUG] By default pip install lighteval is installing the cpu only torch version , its killing dependencies. bug Something isn't working
#487 opened Jan 6, 2025 by kzos
[FT] Add and test multinode runs back feature request New feature/request
#482 opened Jan 2, 2025 by clefourrier
[FT] Enhancing CorpusLevelTranslationMetric with Asian Language Support feature request New feature/request
#478 opened Dec 27, 2024 by ryan-minato
ProTip! What’s not been updated in a month: updated:<2025-01-14.