SGD
Stochastic Gradient Descent updates weights using the gradient computed on a small mini-batch, adding noise from sampling but enabling scalable training on large datasets.
What is SGD?
Stochastic Gradient Descent updates weights using the gradient computed on a small mini-batch, adding noise from sampling but enabling scalable training on large datasets.
Stochastic Gradient Descent updates weights using the gradient computed on a small mini-batch, adding noise from sampling but enabling scalable training on large datasets.
Where is it used?
SGD with momentum is used in vision models (ResNet) and some LLM pretraining recipes; its simplicity makes it a baseline optimizer, though Adam dominates transformers.
How to build it
Use `torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)` in a training loop, compare convergence speed with plain SGD (momentum=0) on a loss curve.
Code
A practical example: