Post-Training Quantization
Post-training quantization (PTQ) quantizes a model after training is complete, without retraining. It is fast and requires no training infrastructure, making it the most practical quantization approach for deploying existing models.
What is Post-Training Quantization?
Post-training quantization (PTQ) quantizes a model after training is complete, without retraining. It is fast and requires no training infrastructure, making it the most practical quantization approach for deploying existing models.
Post-training quantization (PTQ) quantizes a model after training is complete, without retraining. It is fast and requires no training infrastructure, making it the most practical quantization approach for deploying existing models.
Where is it used?
GPTQ, AWQ, and BitsAndBytes are all PTQ methods used on LLaMA, Mistral, and GPT models. GGUF conversion in llama.cpp is PTQ. PTQ is the go-to for users who want to run models on consumer hardware without access to training GPUs or the original training data.
How to build it
GPTQ: `from auto_gptq import AutoGPTQForCausalLM; model = AutoGPTQForCausalLM.from_pretrained(base); model.quantize(calibration_data); model.save_quantized('out')`. BitsAndBytes: `from_pretrained(..., load_in_4bit=True)`. AWQ: `AutoAWQForCausalLM.from_pretrained(base); model.quantize(...)`. Calibrate with ~128 samples.