Backpropagation
Backpropagation computes the gradient of the loss with respect to every parameter by applying the chain rule from the output layer back to the input. These gradients tell the optimizer how to adjust each weight to reduce the loss.
What is Backpropagation?
Backpropagation computes the gradient of the loss with respect to every parameter by applying the chain rule from the output layer back to the input. These gradients tell the optimizer how to adjust each weight to reduce the loss.
Backpropagation computes the gradient of the loss with respect to every parameter by applying the chain rule from the output layer back to the input. These gradients tell the optimizer how to adjust each weight to reduce the loss.
Where is it used?
Backpropagation runs after every forward pass during training of all neural models (GPT, LLaMA, BERT). PyTorch's autograd engine handles it automatically. ZeRO and pipeline parallelism distribute the backward pass across GPUs for large models.
How to build it
Call `loss.backward()` after computing the loss. PyTorch builds the computation graph during the forward pass and traverses it in reverse. Inspect gradients: `for name, p in model.named_parameters(): print(name, p.grad)`. Clear with `optimizer.zero_grad()` before each step.