Quantization: Doing More with Fewer Bits

Why Fewer Bits Win

Every weight and activation in a neural network is, by default, a 32-bit floating-point number (FP32). That is a generous amount of precision — about seven decimal digits — and for inference most of it is wasted. Model quantization is the craft of storing those numbers in far fewer bits, most commonly 8-bit integers (int8 inference). The payoff is immediate: an INT8 value takes one byte instead of four, so the model is roughly 4x smaller on disk and in memory.

Smaller is only the start. Recall from guide 1 that inference is often memory-bound: the chip spends more time hauling numbers in from memory than actually computing. Quarter the bytes and you quarter the memory traffic, which often translates directly into speed. Integers are also cheaper to compute with — an 8-bit integer multiply-add burns a fraction of the energy and silicon area of a 32-bit float one, which is why a hardware accelerator (a GPU tensor core, an NPU, a phone's DSP) can run INT8 math several times faster than FP32.

So fewer bits win on size, bandwidth, energy, and raw throughput all at once. The catch — and the subject of the rest of this guide — is doing it without wrecking accuracy. Round too crudely and the model's predictions degrade; round wisely and you can often keep accuracy within a fraction of a percent of the original. The next sections give you the exact math and the recipes that make aggressive quantization safe.

The Math of Quantization: Scale and Zero-Point

At its core, quantization is a straight-line map from a continuous range of real numbers onto a small, evenly spaced grid of integers. Two numbers fully describe that line: the scale (how big one integer step is, in real units) and the zero-point (which integer lands exactly on real zero). This is called affine or uniform quantization because the steps are all the same size and the map is a simple linear shift-and-scale.

Left: quantize (real → integer). Middle: dequantize (integer → real). Right: how the scale is chosen from the real and integer ranges.

Let us name every symbol. r is the real (FP32) value we want to store; q is the integer code we actually store; s is the scale, the real-world size of one integer step; and z is the zero-point, the specific integer that decodes back to real 0. The constants q_min and q_max are the smallest and largest integers the format allows — for signed int8 that is −128 and 127. The first formula quantizes (real → integer): divide by the step size, round to the nearest integer, then shift by the zero-point. The second dequantizes (integer → real): undo the shift and multiply back by the step. That little ≈ is the whole story of this guide — the round() throws away a sliver of information, so r comes back almost but not exactly equal.

1. Pick the real range to cover. Say a tensor's values run from r_min = −2.5 to r_max = 4.0, and we target signed int8, so q_min = −128 and q_max = 127.
2. Compute the scale: s = (r_max − r_min)/(q_max − q_min) = (4.0 − (−2.5))/(127 − (−128)) = 6.5 / 255 ≈ 0.02549. Each integer step is worth about 0.0255 in real units.
3. Compute the zero-point: z = round(q_min − r_min/s) = round(−128 − (−2.5/0.02549)) = round(−128 + 98.08) = round(−29.92) = −30. Sanity check: decoding q = −30 gives s·(−30 − (−30)) = 0, exactly real zero.
4. Quantize the sample value r = 1.0: q = round(1.0 / 0.02549) + (−30) = round(39.23) − 30 = 39 − 30 = 9. We store the single byte 9.
5. Dequantize to see what we recover: r̂ = s·(q − z) = 0.02549·(9 − (−30)) = 0.02549·39 ≈ 0.994.
6. Read off the quantization error: |r − r̂| = |1.0 − 0.994| = 0.006. That sits comfortably under half a step (s/2 ≈ 0.0127), which is the worst rounding can ever cost you.

Post-Training Quantization (PTQ)

Post-training quantization (PTQ) is the easy door into the room. You take a model already trained in FP32 and convert it to integers without any retraining — no gradients, no labels, no GPU-hours, often just a few minutes of processing. For weights this is straightforward: their values are fixed, so you simply scan each weight tensor, read off its min and max, and compute a scale and zero-point exactly as we did above. This turns a normal model into a quantized one for int8 inference.

Activations are trickier, because their ranges depend on the input and you do not know them until data flows through. The fix is calibration: run a small, representative batch of real inputs through the model and watch the running min/max (or a smarter percentile) at every layer's output. Those observed ranges become the scales for the activations. A few hundred unlabeled images is usually plenty.

One more lever decides a lot of accuracy: granularity. Per-tensor quantization uses a single scale for an entire weight tensor. That is simple but brutal when one output channel has much larger weights than its neighbours — the big channel forces a large scale, and every small channel is then rounded coarsely. Per-channel quantization gives each output channel (each convolution filter) its own scale, so a loud channel no longer drowns out the quiet ones. For weights this is nearly free and is the standard choice; activations are usually left per-tensor because per-channel activation scales are harder for hardware to exploit.

# Post-training quantization with calibration
model_fp32 = load_trained_model()
model_fp32.eval()

# 1. Insert observers that watch activation ranges
qmodel = prepare_ptq(model_fp32)    # adds min/max observers per tensor

# 2. Calibrate: feed a few hundred REPRESENTATIVE samples
for images in calibration_loader:   # ~100-500 images, NO labels needed
    qmodel(images)                  # observers record running min/max

# 3. Freeze observed ranges into scale + zero-point, convert to int8
int8_model = convert_ptq(qmodel)    # per-channel weights, per-tensor activations

Because it is so cheap and frequently costs well under a percent of accuracy, PTQ is the first thing you should try. Reach for the heavier machinery only if PTQ's accuracy drop turns out to be unacceptable.

Quantization-Aware Training (QAT)

When PTQ's accuracy loss is too large — common at very low bit-widths or on delicate models — you move to quantization-aware training (QAT). The idea is to simulate quantization during training so the network can adapt its weights to the rounding it will face at inference time, instead of being ambushed by it afterwards.

QAT works by inserting 'fake quantize' nodes into the forward pass. At each such node the tensor is quantized and then immediately dequantized — round it to the integer grid, then map it back to a (now slightly lossy) float. The network therefore sees the exact rounding errors it will hit in production and learns weights that are robust to them. Crucially, a full-precision FP32 master copy of the weights is kept and updated by the optimizer; the fake-quant only shapes what the forward pass experiences. This is still quantization — we just rehearse it while we can still learn.

The straight-through estimator: pass the gradient through the rounding as if it were the identity.

Here is the snag the equation solves. The forward pass uses x_q, the rounded value, but round() is a staircase: flat between steps, with vertical jumps at the boundaries. On every flat tread its slope is exactly zero, and at the jumps it is undefined. So its true derivative ∂x_q/∂x is zero almost everywhere. If we backpropagated honestly through it, the gradient reaching the weights would be multiplied by zero — the network would get no learning signal and training would stall completely. The straight-through estimator (STE) sidesteps this with a deliberate lie: on the backward pass it pretends the rounding was the identity function, setting ∂x_q/∂x := 1 for values inside the representable range (and 0 for values clipped outside it). In symbols, ∂L/∂x ≈ ∂L/∂x_q — the gradient flows straight through the fake-quant node untouched. The forward pass still feels the rounding, but the backward pass can still learn.

def fake_quantize(x, s, z, q_min, q_max):
    # forward: quantize, then immediately dequantize
    q   = clamp(round(x / s) + z, q_min, q_max)
    x_q = s * (q - z)
    return x_q

# Straight-through estimator (conceptual autograd):
#   forward:  return x_q        (the rounded/clamped value)
#   backward: grad_in = grad_out  # pass straight through, as if identity
#   (gradient is zeroed only where x was clipped outside [r_min, r_max])

QAT costs real training time and a working training pipeline, so spend it deliberately. It earns its keep for low-bit targets (INT4 and below), for accuracy-critical models where even a half-percent drop matters, and for architectures PTQ handles badly. If plain PTQ already lands within tolerance, QAT is effort you do not need.

INT8 Inference in Practice

Quantization error is not spread evenly across a network — a handful of layers cause most of the damage. The usual suspects: the first convolution (it sees raw pixels with wide, uneven ranges), the final classifier (small logit differences decide the answer, so coarse rounding flips predictions), attention and softmax blocks (exponentials are sensitive to input scale), and depthwise convolutions (each channel is thin, so a per-tensor scale fits them poorly). Knowing these in advance tells you where to look first when int8 inference disappoints.

The practical response is mixed precision: quantize the bulk of the network to INT8, but keep the few fragile layers in FP16 or even FP32. You surrender a little of the speed and memory win on those layers and buy back most of the lost accuracy. A model that is 95% INT8 still captures almost all of quantization's benefit.

How do you find the guilty layers? Per-layer error analysis. Run the FP32 and quantized models side by side on the same inputs and compare each layer's output — with a metric like signal-to-quantization-noise ratio or plain mean-squared error — to see where the divergence spikes. This is the same profiling mindset from guide 1, now aimed at accuracy rather than latency: measure, do not guess. Tools that report per-layer error let you target exactly which layers to bump back up to higher precision (see model profiling and PTQ tooling).

1. Quantize, then measure end-to-end accuracy on a held-out set — never trust the conversion blindly.
2. If accuracy dropped, run per-layer error analysis to rank the worst offenders.
3. Keep the top offenders (often first conv, last classifier, attention) in FP16/FP32.
4. Switch weights to per-channel before reaching for anything heavier.
5. Tame activation outliers with percentile-based range clipping.
6. Re-measure after each change so you know which fix actually paid off.

Choosing Your Path: PTQ vs QAT, INT8 vs FP16

Pulling it together, here is a ladder to climb only as high as you must. Each rung costs more effort than the last, so stop the moment your accuracy and latency targets are met.

1. Try FP16 first. Halving from 32 to 16 bits is almost free, rarely dents accuracy, and many accelerators run it at full speed. Often this alone is enough.
2. Move to PTQ INT8 with good calibration. Feed a representative calibration set and measure. This is where the big 4x size win and integer-hardware speedup really land.
3. Switch weights to per-channel. If per-tensor INT8 lost too much, per-channel weight scales usually recover most of it at almost no cost.
4. Add mixed precision. Keep the few sensitive layers in FP16/FP32, guided by per-layer error analysis.
5. Reach for QAT last. Only if accuracy still falls short, or you need INT4 and below, pay for quantization-aware retraining.

Notice the through-line: each rung trades more of your effort for more of the model's accuracy. Post-training quantization gives you most of the prize for almost none of the work, which is why it sits near the bottom of the ladder; quantization-aware training buys back the last few tenths of a percent but demands a full training run, which is why it sits at the top.

Let hardware steer every choice. A precision is only fast if the chip accelerates it — some NPUs are INT8-only and ignore FP16; some GPUs love FP16 and BF16 but gain little from INT8; the very newest parts add INT4 or FP8. There is no point shipping int8 inference to a board that runs it no faster than float, nor paying for QAT to hit a precision the hardware accelerator cannot even execute. Always match your target precision to what the deployment chip actually rewards — exactly the accelerator point from guide 1.

One thing this guide has quietly assumed is that something actually executes your quantized model fast. That something is a runtime or inference engine — and the same INT8 graph can be twice as fast on one engine as on another. The next guide opens that box: ONNX, TensorRT, and the hardware stack that turns these quantized models into real-world speed.