MDSection 21 of 38 · 7 subtopics
GPU & Memory
VRAM budgeting: weights, activations, gradients, optimizer state, KV cache.
Learning path
Work through each subtopic in order. Click any file below to open its lesson. Track your progress with the checkbox at the bottom.
Subtopics
Each subtopic includes a concise explanation, code examples, and references.
01
VRAM
GPU memory meter.
vram
02
Model Memory
Model-size calculator.
model-memory
03
Weight Memory
Weight-storage visualization.
weight-memory
04
Activation Memory
Activation tensor timeline.
activation-memory
05
Gradient Memory
Training-memory diagram.
gradient-memory
06
Optimizer Memory
Optimizer-state allocation diagram.
optimizer-memory
07
KV Cache Memory
Cache-growth visualization.
kv-cache-memory