Validation
Validation evaluates the model on a held-out set of data not used for training, measuring generalization. For LLMs, this typically means computing perplexity or loss on a validation split and running benchmark tasks periodically.
What is Validation?
Validation evaluates the model on a held-out set of data not used for training, measuring generalization. For LLMs, this typically means computing perplexity or loss on a validation split and running benchmark tasks periodically.
Validation evaluates the model on a held-out set of data not used for training, measuring generalization. For LLMs, this typically means computing perplexity or loss on a validation split and running benchmark tasks periodically.
Where is it used?
During LLaMA-2 and GPT-3 pretraining, validation loss is tracked every N steps to detect overfitting or divergence. Downstream benchmarks (MMLU, HumanEval, GSM8K) are run on intermediate checkpoints to guide training decisions and model selection.
How to build it
Split data: hold out 1% as validation. Periodically: `model.eval(); with torch.no_grad(): val_loss = compute_loss(model, val_loader)`. Track in Weights & Biases or TensorBoard. Run benchmarks with `lm-evaluation-harness` on saved checkpoints.