Model quantization reduces the numeric precision used to store a model's weights — from standard 32-bit floating point down to 16, 8, or even 4 bits — trading a small amount of numerical accuracy for a large reduction in memory and compute cost.
Why Quantization Works at All
Neural network weights don't need the full precision of 32-bit floating point to be useful — the specific 20th decimal digit of a weight rarely matters for the network's overall behavior. Quantization exploits this by representing weights with fewer bits, accepting a small, carefully-controlled amount of rounding error in exchange for major memory and speed benefits.
The Quantization Formula
A range of real-valued weights \([x_{\min}, x_{\max}]\) is mapped onto a fixed number of discrete integer levels (\(2^b\) levels for \(b\)-bit quantization). Dequantizing (converting back to an approximate real value) reverses this: \(\hat{x} = q \times \text{scale} + x_{\min}\).
Numerical Example
Quantizing weights in range \([-1, 1]\) to 8-bit (\(2^8=256\) levels): \(\text{scale} = \frac{1-(-1)}{255} \approx 0.00784\). A weight of \(0.3\): \(q = \text{round}\left(\frac{0.3-(-1)}{0.00784}\right) = \text{round}(165.8) = 166\). Dequantized: \(\hat{x} = 166 \times 0.00784 + (-1) \approx 0.302\) — very close to the original \(0.3\), a small, acceptable rounding error.
The Precision/Efficiency Tradeoff
| Precision | Memory per Parameter | Typical Accuracy Impact |
|---|---|---|
| FP32 (standard) | 4 bytes | None (baseline) |
| FP16 / BF16 | 2 bytes | Minimal, widely used as a training default |
| INT8 | 1 byte | Small, often negligible with careful quantization |
| INT4 / NF4 | 0.5 bytes | Larger, but often still acceptable, especially for inference or frozen weights (see QLoRA) |
Code
import torch
def quantize(x, bits=8):
x_min, x_max = x.min(), x.max()
scale = (x_max - x_min) / (2**bits - 1)
q = torch.round((x - x_min) / scale)
return q, scale, x_min
def dequantize(q, scale, x_min):
return q * scale + x_min
weights = torch.tensor([-1.0, 0.3, 0.7, -0.5])
q, scale, x_min = quantize(weights, bits=8)
reconstructed = dequantize(q, scale, x_min)
print(weights)
print(reconstructed) # very close to the original, small rounding errorCommon Mistakes
- Applying aggressive quantization to weights that are still being actively trained — as noted in QLoRA, quantization error can compound across training steps for weights receiving gradient updates; it's generally safer for frozen weights or inference-only deployment.
- Assuming quantization always requires retraining or fine-tuning afterward — many "post-training quantization" techniques can be applied directly to an already-trained model with reasonable accuracy retention, without further training.
Interview Relevance
Q: "Why does reducing weight precision from 32-bit to 8-bit typically cause only a small accuracy drop, rather than a proportional one?" Neural network weights don't require full 32-bit precision to be functionally useful — small rounding errors from quantization tend to average out or get absorbed by the network's inherent robustness to minor perturbations, rather than compounding into large output errors, especially when quantization is applied carefully (e.g. per-channel scaling, or specialized formats like NF4).
Practice Question
Quantize the value \(0.6\) to 4-bit precision, given a range of \([-1, 1]\). (Hint: \(2^4=16\) levels, scale \(=\frac{2}{15}\).)
