int8 inference

Int8 inference is the most common concrete instance of quantization that actually ships: run the model's heavy matrix multiplies using 8-bit integers (values −128…127 for signed, or 0…255 for unsigned) instead of 32-bit floats. Modern CPUs, GPUs, and phone NPUs all have dedicated int8 arithmetic units that do this several times faster while moving a quarter of the bytes — so it improves both speed and memory at once.

Mechanically: weights and activations are quantized to int8 with per-channel or per-tensor scales; the matmul multiplies int8 by int8 but accumulates the products in int32 (so the running sum cannot overflow); then a "requantization" step rescales that int32 result back to int8 for the next layer, by multiplying with the combined scale — usually implemented efficiently as a fixed-point integer multiply plus a bit-shift. Biases are kept in int32. A common, hardware-friendly convention is symmetric int8 weights with asymmetric (zero-point) int8 activations, and either static (calibrated) or dynamic activation ranges.

Accuracy in practice is excellent for CNN classification and detection — ResNet, YOLO, EfficientDet typically lose under 1% — when scales are well calibrated and per-channel. ViTs and attention/normalization-heavy networks need careful outlier handling (or quantization-aware training) to match. The typical payoff over FP32 is roughly 2–4× lower latency and 4× less memory in the matmul-bound parts of the network, delivered through backends like TensorRT, ONNX Runtime, TFLite, oneDNN, and XNNPACK.