Wanda (pruning by weights and activations)

Wanda is a strikingly simple recipe for pruning large language models without any retraining. It scores each weight not by its magnitude alone but by its magnitude multiplied by the size of the input it is multiplied with — the intuition being that a small weight on a large, frequently active input can matter more than a large weight on a tiny input.

Concretely, for output neuron i the importance of weight W_ij is the product of |W_ij| and the L2 norm of input feature j measured over a small calibration set. Wanda compares weights within each output row, a per-output comparison rather than a global threshold, and zeros the lowest-scoring ones to hit a target sparsity, optionally arranged in a 2:4 pattern. There are no gradients, no Hessian inverse, and no weight updates at all.

Despite its simplicity, Wanda matches or beats far more expensive one-shot pruners such as SparseGPT at moderate sparsity, which makes it a strong and fast baseline. At very high sparsity the gap to retraining-based methods reopens, so it is best seen as the cheap first move.