Perplexity
Perplexity is the exponential of the average cross-entropy loss, interpretable as the model's effective vocabulary size at each prediction; lower perplexity means the model is more confident and accurate.
What is Perplexity?
Perplexity is the exponential of the average cross-entropy loss, interpretable as the model's effective vocabulary size at each prediction; lower perplexity means the model is more confident and accurate.
Perplexity is the exponential of the average cross-entropy loss, interpretable as the model's effective vocabulary size at each prediction; lower perplexity means the model is more confident and accurate.
Where is it used?
GPT-3, Llama-2, and Mistral report perplexity on WikiText and The Pile; `evaluate` library and `lm-eval-harness` compute it; it is the standard intrinsic language-modelling metric.
How to build it
Compute `perplexity = torch.exp(torch.mean(loss))` over a held-out dataset, or use `evaluate.load("perplexity")` with `model_id` and `input_texts` to get the score in one call.