weight-only quantization

Not everything in a model needs to be shrunk. In memory-bound decode, the expensive thing is reading the weights, while the activations flowing through are small by comparison. Weight-only quantization compresses just the weights down to four or eight bits and leaves the actual arithmetic running in higher precision.

The weights are stored as low-bit integers; at compute time each weight, or block of weights, is dequantized back to fp16 or bf16 on the fly and the matrix multiply runs in that higher precision. Because the bottleneck was memory traffic, you still get the speed and footprint win, while keeping activations in sixteen bits sidesteps the accuracy pitfalls of quantizing the dynamic, outlier-prone activations. Methods like GPTQ and AWQ choose the integer grid carefully — AWQ, for instance, protects the weights tied to the most salient activations — to hold quality even at int4.