Chain Rule
The chain rule states that the derivative of a composition of functions is the product of their individual derivatives. It enables computing gradients through deep nested computations.
What is Chain Rule?
The chain rule states that the derivative of a composition of functions is the product of their individual derivatives. It enables computing gradients through deep nested computations.
The chain rule states that the derivative of a composition of functions is the product of their individual derivatives. It enables computing gradients through deep nested computations.
Where is it used?
Backpropagation is literally the chain rule applied layer-by-layer through a neural network; a 100-layer transformer's gradients are products of 100 Jacobians.
How to build it
Define `y = f(g(x))` with PyTorch tensors requiring grad, call `.backward()`, and verify `x.grad` equals the product of the local derivatives you compute by hand.
Code
A practical example: