Quantization
Also known as: model quantization, neural network quantization, INT8 quantization, 4-bit quantization, GGUF
Quantization is a model compression technique that reduces the size and memory requirements of a neural network by representing its weights and activations in lower-precision numerical formats.
A model trained with 32-bit floating-point numbers (FP32) can be quantized to 16-bit (FP16), 8-bit integers (INT8), or even 4-bit formats, shrinking the model's memory footprint by 2-8x and typically increasing inference speed significantly. Quantization is one of the primary techniques for making large AI models practical for on-device deployment or high-throughput serving at lower cost.
Quantization, explained at three levels
At home
Quantization is about using lower-quality numbers to represent an AI model's knowledge. Imagine writing a recipe that calls for '2.34817 cups of flour'. You could simplify it to '2.3 cups' and the cake would still taste nearly the same. Quantization does the same thing to the numbers that represent an AI model's 'knowledge': instead of storing very precise numbers, it stores rounded versions. The model is smaller and faster, and usually still performs almost as well.
At work
Quantization is one of the primary reasons it is now possible to run capable language models on consumer laptops. The Llama 3 8B model in its full-precision (FP16) format requires about 16GB of GPU memory. Quantized to 4-bit (using GGUF format), it requires about 5GB and can run on most modern computers with a GPU. The performance trade-off is modest for many tasks, making 4-bit quantized models the default for local AI deployment.
At home
Quantization-aware training (QAT) and post-training quantization (PTQ) are the two main approaches. Post-training quantization converts a trained model's weights to lower precision after training, without retraining: faster but may reduce quality more. Quantization-aware training incorporates the quantization into the training process itself, simulating the effects of lower precision during training so the model learns to compensate: more accurate but requires retraining or fine-tuning. For LLMs, PTQ is most common because retraining these models is extremely expensive.
At work
The accuracy impact of quantization varies by bit depth. FP16 (16-bit float) quantization from FP32 has minimal accuracy impact. INT8 (8-bit integer) quantization typically loses less than 1% accuracy on most tasks. INT4 (4-bit integer) quantization can lose 2-5% accuracy on complex reasoning tasks but may be acceptable for applications where lower precision is sufficient. Mixed-precision quantization (quantizing different layers to different bit depths based on sensitivity) can maintain accuracy better than uniform quantization.
At home
Quantization affects different model components differently. Attention weights in transformer models are generally more sensitive to quantization than feed-forward weights. Activation quantization (quantizing the values flowing through the model during inference, not just the stored weights) can cause additional quality degradation because activations have more variable distributions than weights. Many production quantization approaches quantize weights only (W8A16 or W4A16: 8-bit or 4-bit weights, 16-bit activations) as a compromise between compression and quality.
At work
From a practitioner standpoint, quantization decisions should be driven by the specific deployment target and quality requirements. If deploying to a GPU cluster with ample memory, FP16 is often the right choice (maximum quality, modest memory reduction). For on-device deployment on consumer hardware, 4-bit quantization is usually necessary. For each deployment target, evaluate quantized model quality on your specific task before committing: general benchmark scores may not reflect quality on your specialized use case, and the accuracy loss from quantization may be higher for specialized or technical domains than for general language tasks.
Real-world examples
At home
Jerome wanted to run a capable AI language model locally on his laptop for privacy. He downloaded a 4-bit quantized version of a 7B parameter model using the GGUF format, which required only 4GB of RAM. The model ran at about 15 tokens per second on his CPU (usable for writing assistance tasks) without any data leaving his computer.
At work
An IoT company needed to run an image classification model on microcontrollers with 256KB of flash storage and 32KB of RAM: tiny devices with no GPU. Their ML engineers quantized the model from FP32 to INT8, reducing it from 12MB to 3MB, then further optimized for their target processor using INT4 quantization for the most memory-intensive layers, producing a 1.5MB model that fit within the device constraints with acceptable accuracy on their quality inspection task.
Common confusions
- Quantization is not the same as model distillation. Distillation creates a new, structurally smaller model by training a student on a teacher's outputs. Quantization reduces the numerical precision of an existing model's weights without changing the model's structure. A distilled model has fewer parameters; a quantized model has the same number of parameters represented in a smaller numerical format. Both reduce model size and inference cost but through different mechanisms.
- Lower bit depth does not always mean unacceptably lower quality. For many common tasks (question answering, summarization, translation, writing assistance) the quality difference between a 16-bit and a 4-bit quantized version of the same model is small. For specialized technical tasks requiring precise reasoning (complex mathematics, code generation, scientific reasoning) quantization quality loss may be more significant. Always evaluate on your specific task.
- Quantization precision and inference precision are different concepts. A model can be stored in 4-bit precision (quantized weights) but perform some calculations in 16-bit precision during inference (dequantizing for the computation). This mixed-precision approach (common in modern quantization frameworks) provides memory efficiency without the accuracy loss of fully quantized computation.
For analysts
In BA artefacts, quantization surfaces as a technical implementation constraint for AI features with specific memory, cost, or on-device requirements. When requirements specify that a model must run within a specific memory budget or on-device without cloud connectivity, the ML engineering team will likely use quantization. The BA's role is to specify the accuracy threshold on the relevant task that any quantized model must meet: this becomes the acceptance criterion for the quantization approach chosen. Useful acceptance criterion: 'The deployed model, in its quantized configuration for the target device, must achieve a minimum accuracy of [X]% on the domain-specific evaluation set, measured against the full-precision model's output as the ground truth.'
History & origin
Quantization has been used in signal processing for decades to reduce storage requirements for audio and image data. Its application to neural network weights was explored from the late 2010s as deep learning models grew larger. INT8 quantization became standard in production inference frameworks (TensorFlow Lite, ONNX Runtime, TensorRT) by 2019, enabling efficient mobile and edge deployment. The rise of very large language models (70B+ parameter LLMs) from 2022 made quantization central to practical LLM deployment: without quantization, running a 70B model requires hundreds of gigabytes of GPU memory. GPTQ and GGUF quantization formats for LLMs became widely used open-source tools from 2023.