Gradient Memory
Gradient memory is the VRAM used to store partial derivatives of the loss with respect to every trainable parameter during backpropagation. Each parameter has a matching gradient tensor of the same shape.
What is Gradient Memory?
Gradient memory is the VRAM used to store partial derivatives of the loss with respect to every trainable parameter during backpropagation. Each parameter has a matching gradient tensor of the same shape.
Gradient memory is the VRAM used to store partial derivatives of the loss with respect to every trainable parameter during backpropagation. Each parameter has a matching gradient tensor of the same shape.
Where is it used?
In PyTorch training loops, `.backward()` populates `.grad` on every leaf tensor; mixed-precision training keeps master gradients in fp32, while DeepSpeed ZeRO can shard them across GPUs.
How to build it
Run a forward+backward pass on a small model and print `torch.cuda.memory_allocated()` before and after `.backward()`; inspect `param.grad.shape` for each parameter in `model.parameters()`.