Hi FastChat team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets (Apache 2.0, TestCase/Suite schema with a real JSON Schema validator). It has a benchmarks/ directory converting well-known public benchmarks into validated EvalPort suites, each with attribution and license preserved, so someone using a different harness can still run the same questions without a bespoke converter.
I read the actual data files rather than guessing at fields. fastchat/llm_judge/data/mt_bench/question.jsonl:
{"question_id": 81, "category": "writing", "turns": ["Compose an engaging travel blog post about a recent trip to Hawaii...", "Rewrite your previous response. Start every sentence with the letter A."]}plus fastchat/llm_judge/data/mt_bench/reference_answer/gpt-4.jsonl (reference answers keyed by question_id), and fastchat/llm_judge/data/judge_prompts.jsonl for the LLM-judge rubric.
This maps onto EvalPort cleanly: turns → a multi-turn TestCase.input list, question_id/category → metadata, the matching reference_answer entry → expected_output where present, and the judge prompt → an EvalPort llm_judge grader definition (MT-Bench's pairwise/single-answer grading is exactly the kind of grader EvalPort's spec has a slot for).
Would your maintainers be open to an EvalPort-format export of MT-Bench (and/or Vicuna-Bench, same structure) living in EvalPort's benchmarks/ directory, with attribution to this repo/paper and the Apache-2.0 license preserved? I'd do the conversion work and validate it against EvalPort's real schema — filing this as a heads-up issue first rather than sending a PR out of nowhere. Also happy to link back to FastChat from EvalPort's benchmark listing either way. Let me know if this is of interest, or not a priority right now — no worries either way.
Hi FastChat team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets (Apache 2.0,
TestCase/Suiteschema with a real JSON Schema validator). It has abenchmarks/directory converting well-known public benchmarks into validated EvalPort suites, each with attribution and license preserved, so someone using a different harness can still run the same questions without a bespoke converter.I read the actual data files rather than guessing at fields.
fastchat/llm_judge/data/mt_bench/question.jsonl:{"question_id": 81, "category": "writing", "turns": ["Compose an engaging travel blog post about a recent trip to Hawaii...", "Rewrite your previous response. Start every sentence with the letter A."]}plus
fastchat/llm_judge/data/mt_bench/reference_answer/gpt-4.jsonl(reference answers keyed byquestion_id), andfastchat/llm_judge/data/judge_prompts.jsonlfor the LLM-judge rubric.This maps onto EvalPort cleanly:
turns→ a multi-turnTestCase.inputlist,question_id/category→metadata, the matchingreference_answerentry →expected_outputwhere present, and the judge prompt → an EvalPortllm_judgegrader definition (MT-Bench's pairwise/single-answer grading is exactly the kind of grader EvalPort's spec has a slot for).Would your maintainers be open to an EvalPort-format export of MT-Bench (and/or Vicuna-Bench, same structure) living in EvalPort's
benchmarks/directory, with attribution to this repo/paper and the Apache-2.0 license preserved? I'd do the conversion work and validate it against EvalPort's real schema — filing this as a heads-up issue first rather than sending a PR out of nowhere. Also happy to link back to FastChat from EvalPort's benchmark listing either way. Let me know if this is of interest, or not a priority right now — no worries either way.