-
Notifications
You must be signed in to change notification settings - Fork 310
Add Filipino Benchmark Community Tasks #852
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
started reviewing, i'm launching to evals to make sure all is good so i will come back to you once it's ready. Otherwise very great job, the PR looks great !
Thanks so much for the review! 👍 I have already updated and fixed the errors :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR ports Filipino benchmark tasks into the HuggingFace community tasks, adding language-specific literals, a task manifest, and the full set of Filipino task configurations.
- Added custom translation literals for Cebuano and Tagalog
- Introduced an example manifest listing all Filipino community tasks
- Added a new
filipino_evals.py
module defining all Filipino task configs
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
File | Description |
---|---|
src/lighteval/tasks/templates/utils/translation_literals.py | Added TranslationLiterals entries for Cebuano and Tagalog |
examples/tasks/all_filipino_tasks.txt | New file listing Filipino community tasks |
community_tasks/filipino_evals.py | Added Filipino task definitions across various categories |
Comments suppressed due to low confidence (2)
examples/tasks/all_filipino_tasks.txt:1
- The file appears to include literal line numbers and '+' prefixes in each line, which will break downstream parsers. Remove the leading numbers and '+' characters so each line only contains the task identifier (e.g.
community|readability_ceb_mcf|0|0
).
community|readability_ceb_mcf|0|0
community_tasks/filipino_evals.py:1
- [nitpick] This module is very large (~800 lines) and houses many task definitions. Consider splitting tasks into category-specific submodules or separate files to improve readability and maintainability.
# MIT License
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Hi! Here's our working PR for porting our changes from our own fork filbench/lighteval into HuggingFace's official community tasks :)