quantization

Quantization shrinks a trained model by storing its numbers with far less precision — using coarse, whole-number steps instead of fine decimal values. Imagine describing every paint color by its nearest entry in a 256-shade catalog instead of an exact recipe. You lose some subtlety, but the description gets dramatically smaller and quicker to handle. A model's millions of parameters, normally stored as detailed 32-bit or 16-bit numbers, get rounded onto a much coarser grid — commonly 8-bit, sometimes as low as 4-bit.

The payoff is real and large: an 8-bit model is roughly a quarter the size of a 32-bit one, fits in less memory, and runs faster because the chip moves and multiplies smaller numbers. This is what lets a model that once needed a server-room GPU run on a phone or a laptop. The price is a small loss of accuracy from the rounding — usually tiny if done carefully, occasionally serious if pushed too far or applied to a fragile model.

Why it matters: quantization is the single most common technique for making big models cheap enough to deploy widely, especially on edge devices and consumer hardware. The honest caveats: it is lossy by definition, the damage is unpredictable and must be measured rather than assumed, and the cheapest 4-bit versions can quietly degrade quality in ways that don't show up until a user hits the wrong question. It is a real tradeoff between size, speed, and fidelity — not a free lunch.