Human Evaluation
Human evaluation collects human judgements of model outputs (pairwise preference, rating, or ranking) to assess qualities like helpfulness and safety that automated metrics miss.
What is Human Evaluation?
Human evaluation collects human judgements of model outputs (pairwise preference, rating, or ranking) to assess qualities like helpfulness and safety that automated metrics miss.
Human evaluation collects human judgements of model outputs (pairwise preference, rating, or ranking) to assess qualities like helpfulness and safety that automated metrics miss.
Where is it used?
LMSYS Chatbot Arena uses crowdsourced pairwise votes for Elo scores; Anthropic and OpenAI use human raters for alignment eval; MT-Bench uses GPT-4 as a human-proxy judge.
How to build it
Use `lm-eval-harness --tasks mt_bench` for GPT-4 judged scores, or deploy a pairwise voting UI with `fastapi` + `gradio` to collect human preferences and compute Elo with `elo.py`.