model quantization

Neural networks normally store and compute every weight and intermediate value as a 32-bit floating-point number (FP32), which spans a huge range with fine precision. Quantization is the idea of using far fewer bits — typically 8-bit integers (int8), sometimes 4-bit — to represent those numbers. The everyday analogy: instead of recording every price to the exact cent, you round to the nearest dollar. You lose a little precision, but the data is much smaller and the arithmetic is far cheaper — exactly what you want when shipping a vision model to a phone, a camera, or a loaded GPU server.

Concretely, uniform affine quantization maps a real value r to an integer q by q = round(r / s) + z. Here s (the "scale") is a positive floating-point number equal to the size of one quantization step, and z (the "zero-point") is an integer chosen so that the real value 0 maps exactly to an integer (important so zero-padding stays exact). To use the value again you dequantize: r ≈ s·(q − z). A bit-width of b gives 2^b distinct levels, so int8 gives 256. When z is forced to 0 the scheme is "symmetric," otherwise "asymmetric." The scale can be shared across a whole tensor (per-tensor) or chosen separately for each output channel of a weight (per-channel), the latter being far more accurate for conv and linear layers.

The payoffs are large and concrete. Going FP32→int8 cuts model size roughly 4× and, more importantly, cuts the memory bandwidth needed to stream weights and activations — the true bottleneck for most inference. Integer matrix-multiply units (SIMD on CPUs, int8 tensor cores on NVIDIA GPUs, NPUs on phones) run several times faster than FP32. The cost is quantization error: rounding plus clipping of values that fall outside the chosen range. Vision CNNs (ResNet, MobileNet, EfficientNet) are typically very robust to int8, often losing well under 1% top-1 accuracy; transformers and ViTs are harder because they produce large activation outliers that stretch the scale and crush precision for the typical values.

Distinguish several flavors: weight-only (compress storage, dequantize on the fly) vs weight-and-activation (run the actual matmuls in integer); static (activation scales fixed via calibration) vs dynamic (computed per input at runtime); and post-training quantization (PTQ, no retraining) vs quantization-aware training (QAT, simulate quantization while training). As of the mid-2020s, int8 is the workhorse for deployed vision, FP8 (E4M3/E5M2) is standard on Hopper/Ada-class GPUs for both training and inference, and int4 weight-only (GPTQ, AWQ) is common for large language and multimodal models where memory dominates.