Pruning, sparsity, and low-rank structure

Unstructured vs. structured: the speedup that isn't

Pruning sets weights to zero. The tempting version is unstructured pruning — zero out the smallest-magnitude weights anywhere in the matrix. You can often remove 50% of weights this way with little accuracy loss, and the parameter count looks fantastic. But a matrix with scattered zeros is, to a dense GPU kernel, exactly as expensive to multiply as a full matrix. You saved storage and zero wall-clock time.

Structured pruning fixes this by removing whole units the hardware understands: entire channels, attention heads, or layers. A network with fewer channels is genuinely a smaller dense network — it runs faster on any device with no special kernel. The price is that structured pruning is coarser, so it usually costs more accuracy per parameter removed and benefits from fine-tuning afterward, sometimes guided by distillation from the original model.

The 2:4 compromise: semi-structured sparsity

Between the flexibility of unstructured pruning and the hardware-friendliness of structured pruning sits a clever middle ground: N:M sparsity. The most important case is 2:4 — within every contiguous block of four weights, at most two may be nonzero. This pattern is fine-grained enough to preserve accuracy, yet regular enough that modern GPUs ship dedicated sparse tensor cores that skip the zeros and deliver a real ~2× matmul speedup.

# enforce 2:4 within each block of 4 weights along a row
for block in reshape(row, [-1, 4]):
    keep = top2_by_magnitude(block)      # indices of 2 largest |w|
    block[not in keep] = 0               # the other two -> 0

The 2:4 rule as a constraint: at most two nonzero weights in every contiguous group of four, giving exactly 50% sparsity.

2:4 is the rare case where the paper metric and the wall-clock agree, which is why it has become the default 'free' sparsity for production transformers. You typically apply it, then fine-tune briefly to recover the small accuracy dip.

Pruning without retraining: Wanda

Retraining a pruned billion-parameter model is expensive, so a key question is how well you can prune in one shot. Classic magnitude pruning removes the smallest weights — but in an LLM, a small weight that multiplies a large activation can matter more than a large weight that multiplies a tiny one. Wanda (Weights AND Activations) fixes the score: it ranks each weight by its magnitude times the norm of its corresponding input activation, then prunes the lowest-scoring weights per output. No gradients, no retraining, one calibration pass.

Wanda's importance score: weight magnitude times the L2 norm of that input feature's activations — a small weight on a high-activation channel can still be load-bearing.

Notice the rhyme with AWQ from the last guide: both realize that in a trained transformer, activation scale tells you which weights are load-bearing. The same insight — measure importance with activations, not weights alone — powers the best one-shot methods across both quantization and pruning.

Low-rank structure: compress the matrix itself

A third option keeps every output but re-expresses the weights. Low-rank factorization replaces an m×n weight matrix with the product of an m×r and an r×n matrix, where the rank r is much smaller than m or n. If the original matrix is well-approximated by something low-rank — many learned matrices are — you store and compute far fewer numbers. The classic tool is a truncated singular value decomposition (SVD): keep only the top-r singular directions.

Low-rank factorization replaces an m×n weight matrix with two thin factors, cutting parameters from mn to r(m+n) whenever the rank r is small.

Low-rank thinking runs deeper than compression. The same observation — that weight updates are often low-rank — is exactly what makes parameter-efficient fine-tuning work, the subject of its own track. Here the point is structural: when a layer is genuinely low-rank, factorizing it is a clean, kernel-friendly win that composes happily with quantization on top.