What quantization actually does
Quantization maps continuous floating-point values onto a small grid of integers. A weight matrix in 16-bit floats becomes a matrix of low-bit integers plus a handful of floating-point scales that say how to stretch the grid back to real numbers. The standard affine scheme stores, per group of weights, a scale and a zero-point; at run time the integer is multiplied by the scale to recover an approximation of the original value.
# affine quantize / dequantize for one group scale = (w.max() - w.min()) / (qmax - qmin) zero = round(qmin - w.min() / scale) q = clamp(round(w / scale) + zero, qmin, qmax) # store: int w_hat = (q - zero) * scale # reconstruct
Affine quantization: each value is rounded onto an integer grid with a per-group scale s and zero-point z, then reconstructed as s(q−z).
The size win is mechanical: 4-bit weights are one quarter the bytes of 16-bit, so a 7B model drops from ~14 GB to ~3.5 GB. Because decoding is memory-bound, that is also roughly a 4× bandwidth win and a real speedup — provided you have a kernel that does the matrix multiply directly on the packed integers.
The hard part: outliers
If everything were uniformly distributed, quantization would be trivial. The trouble is that large language models develop a few activation outliers — channels whose magnitudes are 10× to 100× the rest. A single huge value forces a coarse scale that wastes precision on every other value in the group. Naively quantizing activations to 8 bits can collapse a model's accuracy entirely because of these spikes.
SmoothQuant tackles this with a beautiful algebraic trick: since a linear layer computes activations times weights, you can divide the activations by a per-channel factor and multiply the weights by the same factor without changing the output. Choose that factor to migrate the outlier difficulty out of the hard-to-quantize activations and into the weights, which tolerate it better. After smoothing, both sides quantize cleanly to INT8.
SmoothQuant's algebraic trick: divide activations by a per-channel factor and multiply the weights by it, migrating outlier difficulty from X into W.
One-shot post-training quantization: GPTQ and AWQ
Post-training quantization (PTQ) compresses an already-trained model with only a small calibration set — a few hundred unlabeled samples — and no gradient updates. Two methods dominate the 4-bit weight-only regime. GPTQ quantizes weights one column at a time, and after fixing each column it uses approximate second-order (Hessian) information to update the not-yet-quantized weights so they compensate for the error just introduced. The result is far more accurate than rounding each weight independently.
The PTQ objective behind GPTQ: quantize layer-by-layer to minimize the squared error between original and quantized outputs on a small calibration set X.
AWQ (activation-aware weight quantization) starts from a different observation: not all weights matter equally. The weights that multiply large-magnitude activation channels are salient — quantizing them poorly hurts a lot. AWQ measures activation statistics on the calibration set, identifies the salient weight channels, and scales them up before quantizing so they keep more effective precision. It is simpler than GPTQ, has no backward pass, and tends to be more robust across model families.
When PTQ is not enough: QAT, ternary, and the KV cache
Below 4 bits, post-training methods start to crack and you need the model to learn to be robust to quantization. Quantization-aware training (QAT) inserts fake-quantize operations into the forward pass during training, so the weights drift toward values that survive rounding; gradients flow through the non-differentiable rounding via a straight-through estimator. QAT costs a training run but recovers accuracy that PTQ cannot.
The extreme of this idea is BitNet, which trains transformers from scratch with weights constrained to three values — roughly 1.58 bits each. A ternary weight matrix turns matrix multiplication into matrix addition and subtraction, removing the multipliers entirely and slashing energy. The lesson is general: the lowest-precision models are not quantized after the fact, they are born low-precision.
Finally, do not forget the other big memory consumer in long-context decoding: the attention cache. KV-cache quantization stores keys and values in 4 or 8 bits, which can matter more than weight precision once contexts grow into the tens of thousands of tokens, because the cache grows linearly with sequence length.
Diagram of attention as a query–key–value lookup, highlighting the keys and values that are held in the KV cache.