Benchmark Design
Benchmark design creates standardised task sets, metrics, and evaluation protocols to measure and compare LLM capabilities fairly across models and over time.
What is Benchmark Design?
Benchmark design creates standardised task sets, metrics, and evaluation protocols to measure and compare LLM capabilities fairly across models and over time.
Benchmark design creates standardised task sets, metrics, and evaluation protocols to measure and compare LLM capabilities fairly across models and over time.
Where is it used?
MMLU (57 subjects), HumanEval (164 code problems), GSM8K (math), and BIG-Bench are designed benchmarks; HELM and Open LLM Leaderboard standardise their evaluation.
How to build it
Collect task instances with gold answers, define a metric (exact-match, pass@k), write an `lm-eval-harness` task YAML with `doc_to_text` and `doc_to_target`, and validate on 2 models.