GPU & VRAM
VRAM (Video RAM) is the high-bandwidth memory on a GPU that stores model weights, activations, gradients, and the optimizer state. It is the primary bottleneck for LLM size and batch size.
What is GPU & VRAM?
VRAM (Video RAM) is the high-bandwidth memory on a GPU that stores model weights, activations, gradients, and the optimizer state. It is the primary bottleneck for LLM size and batch size.
VRAM (Video RAM) is the high-bandwidth memory on a GPU that stores model weights, activations, gradients, and the optimizer state. It is the primary bottleneck for LLM size and batch size.
Where is it used?
An 8B-parameter model in fp16 needs ~16GB VRAM just for weights; training it with Adam adds ~32GB for optimizer state. H100s have 80GB HBM3 to fit large batch sizes for GPT-scale training.
How to build it
Inspect VRAM usage with `torch.cuda.memory_allocated()` and `nvidia-smi`, load a model, run a forward pass, and observe how activation memory scales with batch size.
Code
A practical example: