Accuracy
Accuracy is the fraction of predictions that exactly match the ground-truth answer, used for multiple-choice and classification benchmarks like MMLU and ARC.
What is Accuracy?
Accuracy is the fraction of predictions that exactly match the ground-truth answer, used for multiple-choice and classification benchmarks like MMLU and ARC.
Accuracy is the fraction of predictions that exactly match the ground-truth answer, used for multiple-choice and classification benchmarks like MMLU and ARC.
Where is it used?
MMLU, ARC, and HellaSwag report accuracy; `lm-eval-harness` computes `acc` and `acc_norm`; it is the headline metric for most LLM capability comparisons.
How to build it
Run `lm_eval --tasks mmlu --model hf --model_args pretrained=...` and read the `acc` column; for custom MCQ, compare `argmax(logits)` to the gold label and average the match.