Model Memory
Model memory is the VRAM required to load all of a model's parameters. It equals `parameter_count * bytes_per_parameter`, determined by the precision format (FP32=4B, FP16/BF16=2B, INT8=1B, INT4=0.5B per parameter).
What is Model Memory?
Model memory is the VRAM required to load all of a model's parameters. It equals `parameter_count * bytes_per_parameter`, determined by the precision format (FP32=4B, FP16/BF16=2B, INT8=1B, INT4=0.5B per parameter).
Model memory is the VRAM required to load all of a model's parameters. It equals `parameter_count * bytes_per_parameter`, determined by the precision format (FP32=4B, FP16/BF16=2B, INT8=1B, INT4=0.5B per parameter).
Where is it used?
A 7B model needs 28 GB in FP32, 14 GB in FP16, 7 GB in INT8, and 3.5 GB in INT4. LLaMA-2 70B needs 140 GB in FP16 (2x A100 80GB) or 35 GB in INT4 (fits on 2x 24GB GPUs). Model memory is the baseline; KV cache and activations add more during inference.
How to build it
Calculate: `model_mem = sum(p.numel() for p in model.parameters()) * p.element_size()`. In PyTorch: `mem = sum(p.nelement() * p.element_size() for p in model.parameters()) / 1e9` (GB). Choose precision based on available VRAM. Use `device_map='auto'` to split across GPUs.