Learning Rate Scheduling
Learning rate scheduling varies the LR over training, typically starting low (warmup), peaking, then decaying (cosine or linear). This stabilizes early training, allows fast learning in the middle, and fine-tunes convergence at the end.
What is Learning Rate Scheduling?
Learning rate scheduling varies the LR over training, typically starting low (warmup), peaking, then decaying (cosine or linear). This stabilizes early training, allows fast learning in the middle, and fine-tunes convergence at the end.
Learning rate scheduling varies the LR over training, typically starting low (warmup), peaking, then decaying (cosine or linear). This stabilizes early training, allows fast learning in the middle, and fine-tunes convergence at the end.
Where is it used?
Cosine decay with warmup is the standard for GPT-3, LLaMA, and Mistral pretraining. GPT-2 used linear warmup + linear decay. LLaMA-2 decays to 10% of peak LR. Warmup steps are typically 1-2% of total steps to avoid early instability.
How to build it
Use HuggingFace: `scheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=2000, num_training_steps=total_steps)`. Call `scheduler.step()` after each optimizer step. For custom: `lr = peak * 0.5 * (1 + cos(pi * step / total))` after warmup.