Quantization Error
Quantization error is the difference between the original floating-point value and the dequantized approximation. It arises because multiple float values map to the same integer level, introducing noise that can degrade model accuracy.
What is Quantization Error?
Quantization error is the difference between the original floating-point value and the dequantized approximation. It arises because multiple float values map to the same integer level, introducing noise that can degrade model accuracy.
Quantization error is the difference between the original floating-point value and the dequantized approximation. It arises because multiple float values map to the same integer level, introducing noise that can degrade model accuracy.
Where is it used?
GPTQ and AWQ are designed to minimize quantization error by choosing scales that preserve important weights. LLM.int8() detects outlier features and keeps them in FP16 to reduce error. Quantization error is measured by perplexity increase or benchmark accuracy drop.
How to build it
Compute: `error = (dequantized - original).abs().mean()` (MAE) or `((deq - orig) ** 2).mean()` (MSE). Compare across quantization methods. To reduce error: use per-channel scales, group quantization (e.g., 128-element groups in GGUF), or mixed-precision (outliers in FP16).