Memory Optimization
Memory optimisation for inference reduces VRAM consumption via KV-cache quantisation, weight quantisation (int8/int4), PagedAttention, and activation offloading to fit larger models or batches.
What is Memory Optimization?
Memory optimisation for inference reduces VRAM consumption via KV-cache quantisation, weight quantisation (int8/int4), PagedAttention, and activation offloading to fit larger models or batches.
Memory optimisation for inference reduces VRAM consumption via KV-cache quantisation, weight quantisation (int8/int4), PagedAttention, and activation offloading to fit larger models or batches.
Where is it used?
vLLM PagedAttention, `bitsandbytes` int4/int8 loading, and `llama.cpp` GGUF quantisation are standard; TensorRT-LLM uses INT8 KV cache for Llama-3-70B on a single H100.
How to build it
Load a model with `BitsAndBytesConfig(load_in_4bit=True)`, compare `torch.cuda.memory_allocated()` vs fp16, and run generation to verify quality and memory savings.