Weight Memory
Weight memory is the portion of VRAM occupied specifically by the model's weight matrices (embeddings, attention, FFN, LayerNorm). It is the largest component of model memory and the primary target for quantization to reduce footprint.
What is Weight Memory?
Weight memory is the portion of VRAM occupied specifically by the model's weight matrices (embeddings, attention, FFN, LayerNorm). It is the largest component of model memory and the primary target for quantization to reduce footprint.
Weight memory is the portion of VRAM occupied specifically by the model's weight matrices (embeddings, attention, FFN, LayerNorm). It is the largest component of model memory and the primary target for quantization to reduce footprint.
Where is it used?
In a 7B LLaMA model, weight memory is ~14 GB in FP16, accounting for the vast majority of VRAM usage. During QLoRA fine-tuning, base weights are quantized to INT4 (3.5 GB) while LoRA adapters (~1% of params) stay in BF16. The LM head and embeddings often share weights to save memory.
How to build it
Measure: `weight_mem = sum(p.nelement() * p.element_size() for n, p in model.named_parameters() if 'weight' in n)`. Identify largest layers: `sorted([(n, p.nelement()*p.element_size()/1e6) for n,p in model.named_parameters()], key=lambda x: -x[1])[:10]`. Reduce via quantization or weight tying.