Gradients
A gradient is the vector of partial derivatives of a scalar function with respect to all its inputs — it points in the direction of steepest ascent. Training moves weights in the negative gradient direction.
What is Gradients?
A gradient is the vector of partial derivatives of a scalar function with respect to all its inputs — it points in the direction of steepest ascent. Training moves weights in the negative gradient direction.
A gradient is the vector of partial derivatives of a scalar function with respect to all its inputs — it points in the direction of steepest ascent. Training moves weights in the negative gradient direction.
Where is it used?
Every parameter in GPT and Llama has a gradient computed during backprop; the optimizer uses these gradients to update weights and minimize the loss.
How to build it
Create `w = torch.randn(3, requires_grad=True)`, compute `loss = (w**2).sum()`, call `loss.backward()`, and print `w.grad` to see the gradient vector.
Code
A practical example: