4-bit (INT4) quantization

Storing each weight in 4 bits instead of 16 cuts a model's weight memory to about a quarter. A 70-billion-parameter model that needs roughly 140 GB in FP16 then fits in about 35 GB — small enough for a single high-end accelerator, and for smaller models, a laptop or phone.

Practical INT4 uses group-wise affine quantization: weights are split into groups, often 64 or 128 along the input dimension, each group sharing its own scale and zero-point, so a few outlier groups do not wreck the whole tensor. Variants include uniform INT4 and the non-uniform NF4 code, whose levels are placed to match a normal distribution of weights. Because token generation is bound by memory bandwidth, reading 4-bit weights also speeds up decoding, not just storage.

Four bits is roughly today's sweet spot where weight-only LLM quality stays close to FP16. Pushing weights to 2 or 3 bits, or quantizing activations as well, usually needs the activation-aware or training-time methods elsewhere in this field; below that, quality falls off sharply.