Adam / AdamW
Adam combines momentum (running gradient average) and RMSprop (running squared gradient average) for adaptive per-parameter learning rates. AdamW decouples weight decay from the gradient update, improving regularization.
What is Adam / AdamW?
Adam combines momentum (running gradient average) and RMSprop (running squared gradient average) for adaptive per-parameter learning rates. AdamW decouples weight decay from the gradient update, improving regularization.
Adam combines momentum (running gradient average) and RMSprop (running squared gradient average) for adaptive per-parameter learning rates. AdamW decouples weight decay from the gradient update, improving regularization.
Where is it used?
AdamW is the default optimizer for training GPT, Llama, and virtually all modern transformers; it converges faster and more stably than SGD on large language models.
How to build it
Use `torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)`, train a small transformer, and compare loss curves against vanilla Adam to see weight decay's effect.