Model Evaluation
Model evaluation is the systematic process of measuring an LLM's quality, capability, and safety across benchmarks, held-out datasets, and human judgement to compare models and guide development.
What is Model Evaluation?
Model evaluation is the systematic process of measuring an LLM's quality, capability, and safety across benchmarks, held-out datasets, and human judgement to compare models and guide development.
Model evaluation is the systematic process of measuring an LLM's quality, capability, and safety across benchmarks, held-out datasets, and human judgement to compare models and guide development.
Where is it used?
Llama-3, GPT-4, and Claude report MMLU, HumanEval, and MT-Bench scores; the `lm-evaluation-harness` from EleutherAI is the standard open framework for running benchmarks.
How to build it
Install `lm-eval`, run `lm_eval --model hf --model_args pretrained=meta-llama/Meta-Llama-3-8B --tasks mmlu,hellaswag,arc_challenge --batch_size 8` to get benchmark scores.