network pruning

Trained networks are heavily over-parameterized: many weights are near zero or redundant, contributing little to the output. Pruning removes the least important ones, like trimming dead branches off a tree — the model gets smaller and cheaper while keeping most of its accuracy. The whole field rests on a striking empirical fact: you can often delete a large fraction of a network's weights and recover the lost accuracy with a little fine-tuning.

An importance criterion scores each weight (or group) so you can remove the smallest. The simplest is magnitude (|w| small → prune), but there are gradient/Taylor scores (estimate the effect on the loss of removing a weight) and second-order ones (Optimal Brain Damage/Surgeon use the Hessian). Granularity is the key axis: unstructured pruning zeroes individual weights, yielding a sparse matrix; structured pruning removes whole channels, filters, or attention heads. The usual recipe is iterative: train → score → remove p% → fine-tune → repeat, gradually rather than all at once.

The catch is that unstructured sparsity rarely makes a dense kernel faster — you save memory but the hardware still multiplies by zeros unless the runtime specifically exploits the pattern. NVIDIA's 2:4 structured sparsity on Ampere and later (exactly two of every four weights are zero) is a practical middle ground that real tensor cores accelerate ~2×. The Lottery Ticket Hypothesis (Frankle & Carbin) adds a theory angle: dense nets contain sparse subnetworks that, trained from the original initialization, match the full model.