Optimizer Memory
Optimizer memory is VRAM consumed by optimizer state such as momentum and variance buffers. AdamW stores two extra tensors per parameter (first and second moments), tripling memory beyond weights and gradients.
What is Optimizer Memory?
Optimizer memory is VRAM consumed by optimizer state such as momentum and variance buffers. AdamW stores two extra tensors per parameter (first and second moments), tripling memory beyond weights and gradients.
Optimizer memory is VRAM consumed by optimizer state such as momentum and variance buffers. AdamW stores two extra tensors per parameter (first and second moments), tripling memory beyond weights and gradients.
Where is it used?
Adam/AdamW in Hugging Face Trainer and every GPT/Llama training script allocates 2x param-size optimizer state; memory-optimised optimizers like Adafactor or 8-bit Adam in bitsandbytes reduce this.
How to build it
Create `torch.optim.AdamW(model.parameters(), lr=1e-4)`, run one optimisation step, then sum the bytes of `state['exp_avg']` and `state['exp_avg_sq']` across all params to measure optimizer memory.