AWQ (activation-aware weight quantization)

Not all weights matter equally. AWQ observes that a small fraction of weight channels — the ones that get multiplied by large-magnitude activations — carry most of the model's quality, and that protecting roughly the top one percent of these salient channels recovers almost all the accuracy lost to low-bit quantization.

Keeping those channels in higher precision would break hardware kernels, so instead AWQ scales each input channel by a per-channel factor s before quantizing the weights and divides the activation by the same s, an exact reparameterization that leaves the product unchanged. Choosing s from a small grid, driven by the average activation magnitude of each channel, shrinks the relative quantization error on the salient channels. The search uses only forward passes on a tiny calibration set, with no backpropagation and no second-order Hessian.

AWQ delivers strong 4-bit weight-only accuracy, generalizes well across domains because it keys on activation statistics rather than the exact calibration text, and pairs with fast fused INT4 kernels, which has made it a workhorse for running large language models on a single GPU or on-device.