Skip to content

Add Armenian language evaluation suite (ArmBench-LLM)#1289

Open
bugdaryan wants to merge 18 commits into
huggingface:mainfrom
Metric-AI-Lab:main
Open

Add Armenian language evaluation suite (ArmBench-LLM)#1289
bugdaryan wants to merge 18 commits into
huggingface:mainfrom
Metric-AI-Lab:main

Conversation

@bugdaryan

Copy link
Copy Markdown

What does this PR do?

This PR adds a comprehensive Armenian (hy) language evaluation suite to LightEval,
covering 24 tasks across classification, extraction, QA, reading comprehension,
summarization, translation, and knowledge/reasoning. All tasks are backed by the
public Metric-AI/ArmBench-LLM-data
dataset on the Hugging Face Hub and are registered under the armenian:* namespace.

Armenian is a low-resource language with almost no standardized LLM evaluation
coverage. This suite gives the community a reproducible way to benchmark models on
native Armenian tasks, with prompts and gold labels written in Armenian.

Context / related reading

The methodology, dataset, and results behind this suite are described in our blog post:
ArmBench-LLM 1.0: Benchmarking LLMs on Armenian Language Tasks.

Tasks added

All tasks are exposed as armenian:<task>:

Category Tasks
Text classification topic-14class (SIB-200 topics), sentiment
Orthography / correction space_fix, punctuation
Named entity recognition finer, pioner
POS tagging pos (UD Armenian-ArmTDP)
Open / in-context QA arak, ms_marco, squad
Reading comprehension & MCQA belebele, scientific, syndarin, dream, include, hartak
Summarization email, conversation
Paraphrasing paraphrase
Machine translation short_sentences_translation
Knowledge & reasoning mmlu_pro, exam_history, exam_literature, exam_math

Metrics added

Armenian tasks need scoring robust to Armenian script and answer formatting, so this
PR introduces:

  • Armenian-specific metrics (metrics/armenian_metrics.py): exam accuracy, MCQA
    accuracy, MMLU-Pro accuracy, NER span-overlap F1, POS accuracy, and an
    Armenian-tuned BERTScore.
  • Text-tagging metric (TextTaggingMetric + text_tagging_metric): character
    n-gram F1 matching for keyword/keyphrase extraction tasks.
  • Extraction helpers (metrics/utils/armenian_eval_utils.py): regex-based answer
    extractors for letter/numeric multiple-choice and free-form numeric answers.

Other changes

  • transformers_model.py: wrap batch[0].stop_sequences in list() before
    concatenating with the EOS token, so generation works when stop_sequences is a
    tuple rather than a list.

How to run

lighteval accelerate \
  "model_name=<your-model>" \
  "lighteval|armenian:mmlu_pro|0|0"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants