Gradient Descent
Gradient descent is the optimization algorithm that iteratively updates parameters in the direction of the negative gradient, reducing the loss step by step. The step size is the learning rate.
What is Gradient Descent?
Gradient descent is the optimization algorithm that iteratively updates parameters in the direction of the negative gradient, reducing the loss step by step. The step size is the learning rate.
Gradient descent is the optimization algorithm that iteratively updates parameters in the direction of the negative gradient, reducing the loss step by step. The step size is the learning rate.
Where is it used?
SGD, Adam, and AdamW are all variants of gradient descent used to train every LLM; GPT-4 pretraining ran gradient descent on trillions of tokens over thousands of steps.
How to build it
Implement vanilla GD in PyTorch: initialize `w`, compute loss, call `.backward()`, and update `w.data -= lr * w.grad` in a loop, then compare with `torch.optim.SGD`.
Code
A practical example:
Further reading
Want to go deeper? These resources cover Gradient Descent in more detail: