Quantized Inference
Quantized inference runs the model using quantized weights and/or activations, performing matrix multiplications in integer arithmetic where possible. This reduces memory footprint and leverages hardware INT8/INT4 acceleration for faster inference.
What is Quantized Inference?
Quantized inference runs the model using quantized weights and/or activations, performing matrix multiplications in integer arithmetic where possible. This reduces memory footprint and leverages hardware INT8/INT4 acceleration for faster inference.
Quantized inference runs the model using quantized weights and/or activations, performing matrix multiplications in integer arithmetic where possible. This reduces memory footprint and leverages hardware INT8/INT4 acceleration for faster inference.
Where is it used?
llama.cpp runs quantized GGUF models (INT4/INT8) on CPU and GPU. TensorRT and vLLM support INT8 inference on NVIDIA GPUs. ONNX Runtime and TFLite run quantized models on mobile. Quantized inference is the standard for deploying LLMs on resource-constrained hardware.
How to build it
With llama.cpp: `./main -m model-Q4_K_M.gguf -p 'prompt'`. With HuggingFace + BitsAndBytes: `model = AutoModelForCausalLM.from_pretrained(name, load_in_8bit=True); outputs = model.generate(**inputs)`. Measure: compare tokens/second and perplexity vs FP16 baseline.