quantization for inference

Models are trained in sixteen-bit numbers, but serving them that way is expensive: the weights barely fit in memory, and every decode step has to haul all of them across the memory bus. Quantization stores those numbers in fewer bits — eight, four, sometimes less — shrinking the model and speeding up the memory-bound parts of inference.

Each weight is mapped to a low-bit integer plus a scale, and sometimes a zero point, shared across a small group of weights, so the true value is approximately the integer times its scale. Post-training quantization does this after training using a small calibration set; done carefully at int8 or int4, the accuracy loss is small. The payoff is twofold: less VRAM used, and — because decode is bottlenecked by reading weights from memory — fewer bytes per weight means more tokens per second. The risk is that overly aggressive or coarse schemes can visibly degrade quality, so granularity and outlier handling matter.