Weight Quantization
Weight quantization converts only the model's weight matrices to lower precision (INT8/INT4), while activations remain in floating-point. This is the simplest form of quantization and primarily reduces model size and memory bandwidth.
What is Weight Quantization?
Weight quantization converts only the model's weight matrices to lower precision (INT8/INT4), while activations remain in floating-point. This is the simplest form of quantization and primarily reduces model size and memory bandwidth.
Weight quantization converts only the model's weight matrices to lower precision (INT8/INT4), while activations remain in floating-point. This is the simplest form of quantization and primarily reduces model size and memory bandwidth.
Where is it used?
GPTQ, AWQ, and GGUF quantize weights of LLaMA, Mistral, and Falcon models. Weight-only quantization is popular because it does not require calibration data for activations and dequantizes weights on-the-fly during matmul. It is the default for consumer-GPU LLM inference.
How to build it
Per-tensor: `scale = w.abs().max() / 127; q_w = torch.round(w / scale).to(torch.int8)`. In the forward pass: `out = x @ (q_w.float() * scale)`. Per-channel: compute scale per output channel. With AWQ: `from awq import AutoAWQForCausalLM; model.quantize(...)`.