quantization-aware training

If post-training quantization loses too much accuracy — typically at int4, or for ViTs and attention-heavy nets — the fix is to let the network learn to cope with quantization. Quantization-aware training (QAT) inserts "fake quantize" operations during training that round values exactly the way deployment will, so the weights adapt to be robust to that rounding. In effect, the model practices being quantized before it ever ships.

A fake-quant node simulates quantize-then-dequantize in the forward pass, so values snap to the discrete levels they will take at inference, while the rest of training stays in float. The catch is that round() has zero gradient almost everywhere, which would block learning. The standard remedy is the straight-through estimator (STE): in the backward pass treat the rounding as the identity (pass the gradient through unchanged), usually zeroing the gradient for values that were clipped. Scales can be fixed during QAT or learned jointly (LSQ — learned step-size quantization).

QAT recovers most of the accuracy PTQ loses — int8 becomes essentially lossless, and int4 vision is often within ~1% — at the cost of a training run. In practice this is a short fine-tune of a few epochs starting from the FP checkpoint, not training from scratch. It is the standard escalation in production pipelines (PyTorch FX/Brevitas, TensorFlow Model Optimization Toolkit) once PTQ is shown to be insufficient.