Activation Memory
Activation memory is the VRAM used to store intermediate tensors (attention scores, FFN intermediates, residual streams) during the forward and backward passes. It scales with batch size, sequence length, and model width, and is the main memory bottleneck during training.
What is Activation Memory?
Activation memory is the VRAM used to store intermediate tensors (attention scores, FFN intermediates, residual streams) during the forward and backward passes. It scales with batch size, sequence length, and model width, and is the main memory bottleneck during training.
Activation memory is the VRAM used to store intermediate tensors (attention scores, FFN intermediates, residual streams) during the forward and backward passes. It scales with batch size, sequence length, and model width, and is the main memory bottleneck during training.
Where is it used?
During LLM training, activation memory dominates and grows with sequence length. FlashAttention reduces activation memory from O(nĀ²) to O(n) by avoiding materializing the full attention matrix. Gradient checkpointing (activation recomputation) trades compute for memory by recomputing activations during backward.
How to build it
Estimate: `act_mem ā batch_size * seq_len * d_model * n_layers * bytes_per_param * factor` (factor ~10-20 for attention + FFN intermediates). Reduce: use `torch.utils.checkpoint.checkpoint(layer, x)` to recompute activations in backward. Use FlashAttention: `from flash_attn import flash_attn_func`. Monitor: `torch.cuda.memory_allocated()` during forward pass.