Backpropagation
Backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule backward through the network. It makes training deep networks computationally feasible.
What is Backpropagation?
Backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule backward through the network. It makes training deep networks computationally feasible.
Backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule backward through the network. It makes training deep networks computationally feasible.
Where is it used?
PyTorch's `loss.backward()` runs backprop through every transformer layer; without it, training GPT-scale models would be intractable.
How to build it
Set `requires_grad=True` on inputs, compute a loss, call `.backward()`, then inspect `param.grad` for each parameter — register a backward hook with `register_full_backward_hook` to trace it.
Code
A practical example:
Further reading
Want to go deeper? These resources cover Backpropagation in more detail: