Learning Rate
The learning rate controls the size of each weight update. Too high and training diverges; too low and training is slow or stalls. It is the single most important hyperparameter in LLM training, typically in the range 1e-5 to 3e-4.
What is Learning Rate?
The learning rate controls the size of each weight update. Too high and training diverges; too low and training is slow or stalls. It is the single most important hyperparameter in LLM training, typically in the range 1e-5 to 3e-4.
The learning rate controls the size of each weight update. Too high and training diverges; too low and training is slow or stalls. It is the single most important hyperparameter in LLM training, typically in the range 1e-5 to 3e-4.
Where is it used?
LLaMA-2 7B uses a peak LR of 3e-4 with cosine decay; GPT-3 175B uses 6e-5. Fine-tuning uses much lower rates (1e-5 to 5e-5). LoRA fine-tuning uses higher rates (1e-4 to 3e-4) because only adapter weights are updated.
How to build it
Set in the optimizer: `AdamW(model.parameters(), lr=3e-4)`. Use a warmup phase: linearly increase LR from 0 to peak over the first 2000 steps, then cosine decay to 10% of peak. HuggingFace: `get_cosine_schedule_with_warmup(optimizer, num_warmup_steps, num_training_steps)`.