Quantization-Aware Training
Quantization-aware training (QAT) simulates quantization effects during training by inserting fake-quantization operations in the forward pass, allowing the model to adapt to quantization noise. This typically yields better accuracy than PTQ, especially at low bit widths.
What is Quantization-Aware Training?
Quantization-aware training (QAT) simulates quantization effects during training by inserting fake-quantization operations in the forward pass, allowing the model to adapt to quantization noise. This typically yields better accuracy than PTQ, especially at low bit widths.
Quantization-aware training (QAT) simulates quantization effects during training by inserting fake-quantization operations in the forward pass, allowing the model to adapt to quantization noise. This typically yields better accuracy than PTQ, especially at low bit widths.
Where is it used?
QAT is used in mobile/edge deployment (TensorFlow Lite, PyTorch Mobile) where INT8 accuracy matters. It is less common for LLMs due to training cost, but QLoRA is a related approach that quantizes base weights while training low-rank adapters. QAT is used for INT4 and lower precision where PTQ accuracy is unacceptable.
How to build it
In PyTorch: `model.qconfig = torch.ao.quantization.get_default_qconfig('fbgemm'); model = torch.ao.quantization.prepare_qat(model, inplace=True)`. Train normally; quantization noise is simulated in forward. Convert: `model_int8 = torch.ao.quantization.convert(model)`. For LLMs: use QLoRA with NF4 quantization during fine-tuning.