VRAM
VRAM (Video RAM) is the memory on a GPU used to store model weights, activations, KV cache, and intermediate tensors during inference. Its size determines the maximum model that can run; its bandwidth determines inference speed.
What is VRAM?
VRAM (Video RAM) is the memory on a GPU used to store model weights, activations, KV cache, and intermediate tensors during inference. Its size determines the maximum model that can run; its bandwidth determines inference speed.
VRAM (Video RAM) is the memory on a GPU used to store model weights, activations, KV cache, and intermediate tensors during inference. Its size determines the maximum model that can run; its bandwidth determines inference speed.
Where is it used?
An A100 has 40 or 80 GB of VRAM; an H100 has 80 GB; consumer RTX 4090 has 24 GB. A 7B model in FP16 needs ~14 GB of VRAM, fitting on a 4090. A 70B model in FP16 needs ~140 GB, requiring multiple GPUs or quantization. VRAM is the primary constraint for LLM deployment.
How to build it
Check VRAM: `torch.cuda.get_device_properties(0).total_memory` (bytes). Monitor usage: `torch.cuda.memory_allocated() / 1e9` (GB). Free cache: `torch.cuda.empty_cache()`. Use `nvidia-smi` in terminal to see real-time VRAM usage. Calculate needs: `n_params * bytes_per_param + kv_cache + activations`.
Code
A practical example: