Making Models Smaller: Pruning, Distillation, and Compression

Why Networks Are Bigger Than They Need to Be

When you finish training a network, it almost never uses all of its capacity. Researchers find again and again that you can throw away a large fraction of the weights — sometimes 80% or 90% — and, after a little repair, the network performs almost exactly as before. That is the headline fact behind everything in this guide: trained networks are over-parameterized. They have far more knobs (weights) than the task strictly needs. The extra knobs help during training — they make the loss landscape easier to navigate — but once training is finished, many of them are redundant, near-zero, or simply doing the same job as their neighbours.

A helpful way to picture this is the lottery-ticket intuition. Imagine a big network as a giant bundle of lottery tickets. Hidden inside that bundle is a small subnetwork — a 'winning ticket' — that, had you trained it alone from the start, would have done the job almost as well as the whole bundle. Training the big network is partly a way of finding that winning ticket. To be honest rather than overclaim: in practice we usually only discover the winning subnetwork after training the big one, and extracting it cleanly is still an active research area. But the practical lesson is solid — a smaller, capable network is hiding inside your large one, and compression is the craft of pulling it out.

The umbrella term for every technique that extracts that smaller network is model compression. It is a family, not a single trick, and this guide tours the four members you will reach for most: pruning (delete unimportant weights or channels), knowledge distillation (train a small model to imitate a big one), low-rank factorization (replace one fat weight matrix with two thin ones), and — foreshadowed for the next guide — quantization (store and compute numbers with fewer bits). Each attacks 'bigness' from a different angle, and as we will see at the end, they stack.

Pruning: Cutting Away Dead Weight

The most intuitive form of network pruning — and the gateway to the whole topic — is magnitude-based pruning. The recipe is almost suspiciously simple: look at every weight in a layer, rank them by absolute value, and set the smallest ones to zero. The reasoning: a weight multiplies an input, so if the weight is tiny it barely changes the output, and deleting it should barely hurt. This is the simplest member of the model compression family, and it is where almost everyone starts.

Sparsity s, and the magnitude criterion: zero out any weight whose absolute value falls below a threshold τ.

Let us unpack both formulas. Sparsity s is just the fraction of weights you have zeroed out: N_{\text{zeroed}} is the number of weights set to zero and N_{\text{total}} is the total number of weights in the layer (or model). If a layer has 1,000,000 weights and you zero 600,000 of them, s = 600{,}000 / 1{,}000{,}000 = 0.6 — that is '60% sparsity.' The second formula is the rule for which weights to zero: prune weight w whenever its absolute value |w| is below a chosen threshold \tau (you set \tau so that exactly the target fraction falls below it). Why |w|? Because the contribution of a weight to the output is (weight × input), and a weight near zero contributes almost nothing regardless of its input — so it is the cheapest to lose. Crucially, this is only an approximation of true importance: a small weight can still matter a lot if it feeds a highly sensitive path, and a large weight can be redundant. Magnitude is a fast, surprisingly good proxy for importance, not a guarantee.

import torch

def magnitude_prune(weight, sparsity):
    # weight:   one layer's weight tensor
    # sparsity: fraction of weights to zero out, e.g. 0.6
    k = int(sparsity * weight.numel())          # how many weights to remove
    if k == 0:
        return weight
    # threshold = the k-th smallest |w| in the layer
    threshold = weight.abs().flatten().kthvalue(k).values
    mask = weight.abs() > threshold             # keep weights above threshold
    return weight * mask                         # zeroed weights stay in place (same shape!)

Structured Pruning: Removing Whole Channels

Structured pruning fixes exactly that disappointment. Instead of zeroing scattered individual weights, it removes whole structures — entire convolutional filters (output channels), attention heads, or even whole blocks. When you delete an output channel you delete a contiguous slab of the weight tensor that you can physically cut away, leaving a genuinely smaller, still-dense network. 'Dense' means ordinary hardware runs it at full speed with no special sparse-matrix tricks. This is the flavour of network pruning that actually pays off in latency.

How do we decide which channels to cut? We score each channel's importance and drop the lowest scorers. Two cheap, popular scores: (1) the L2 norm of the channel's weights — a filter whose weights are all small produces a small output, so it is probably expendable; and (2) the BatchNorm scale factor γ — most CNN channels pass through a BatchNorm layer that multiplies them by a learned γ, and a channel whose γ is near zero is being actively turned down by the network itself, a strong hint that it is unimportant. These are heuristics, exactly like the magnitude rule before, but they are cheap to compute and work well in practice.

The compute cost of a convolution layer (from guide 1). Output channels C_out appear as a direct multiplier.

This is the FLOPs formula from guide 1, and it tells us exactly why structured pruning buys speed. H_{\text{out}} and W_{\text{out}} are the height and width of the output feature map; C_{\text{in}} and C_{\text{out}} are the number of input and output channels; K is the kernel size (so K^2 is the area of one filter). The cost is a product, so dropping output channels shrinks the C_{\text{out}} term in direct proportion. Take a real layer: a 3\times3 conv with C_{\text{in}}=128, C_{\text{out}}=256, output 56\times56. That is 56\cdot56\cdot128\cdot256\cdot9 \approx 0.92 billion multiply-adds. Prune 25% of its output channels (256 \to 192) and the cost falls to 56\cdot56\cdot128\cdot192\cdot9 \approx 0.69 billion — a clean 25% cut, because 192/256 = 0.75. Better still, the cut cascades: the next layer now sees C_{\text{in}}=192 instead of 256, so it shrinks too. Removing one channel slims two layers.

Knowledge Distillation: Learning from a Teacher

Knowledge distillation takes a completely different angle on model compression: instead of shrinking a trained model, you train a small model — the student — from scratch to imitate a big, accurate model — the teacher. The twist is what the student imitates. A normal classifier is trained against hard labels: 'this image is a cat (1), not a dog (0), not a car (0).' But the teacher offers something far richer — its full probability distribution over all the classes — and that is what we ask the student to match.

Why is the teacher's full distribution so valuable? Consider a cat photo. A good teacher might output 'cat 0.84, dog 0.11, car 0.0001.' Notice the hidden information: the teacher is saying a cat looks a little like a dog and nothing like a car. That relative ranking over the wrong classes — often called dark knowledge — is a free lesson about how the categories relate, and the hard label '1, 0, 0' throws it all away. There is a snag, though: a confident teacher's probabilities are so spiky (0.84 vs 0.11 vs 0.0001) that the interesting structure among the small numbers is nearly invisible. The cure is temperature: divide the logits by a number T before the softmax, which softens the distribution and lifts the small probabilities up to where the student can actually learn from them.

Softmax with temperature T. Larger T flattens the distribution; T = 1 is the ordinary softmax.

Let us read this slowly, because the rest of the section rests on it. \sigma(z)_i is the output probability for class i. The z_j are the logits — the raw, un-normalized scores the network produces before softmax. e^{z_i/T} exponentiates each logit (after dividing by the temperature T) so everything is positive, and the denominator \sum_j e^{z_j/T} sums over all classes so the outputs add to 1. Now watch what T does with a real example. Let the teacher's logits for [cat, dog, car] be z = [4, 2, 1]. At T = 1 (ordinary softmax) we get [0.844,\,0.114,\,0.042] — cat dominates, dog and car are squashed flat. At T = 4 we first divide to get [1,\,0.5,\,0.25], and softmax now gives roughly [0.481,\,0.292,\,0.227]. Same ranking, but look how the gaps shrank: the student now plainly sees 'dog is a plausible second, car a weak-but-real third.' That is the dark knowledge made visible. As T \to \infty the distribution flattens toward uniform; as T \to 1 it sharpens back to the original. Values of T between 2 and 6 are typical.

The distillation loss: a hard-label term plus a soft 'imitate the teacher' term, balanced by α and rescaled by T².

The loss the student minimizes blends two goals. The first term, \mathrm{CE}(y, \sigma(z_s)), is the ordinary cross-entropy against the true hard label y — 'keep getting the actual answer right' — where z_s are the student's logits and \sigma here is the plain (T=1) softmax. The second term is the Kullback–Leibler divergence \mathrm{KL}(\,\cdot\,\|\,\cdot\,), a measure of how different two distributions are (zero when identical, growing as they diverge), between the teacher's softened distribution \sigma(z_t/T) and the student's softened distribution \sigma(z_s/T), where z_t are the teacher's logits. Minimizing it pushes the student's soft outputs to match the teacher's — that is where the dark knowledge transfers. The weight \alpha \in [0,1] trades the two off: high \alpha leans on the true labels, low \alpha leans on imitation (values around 0.1–0.5 are common). And why the T^2 factor? Dividing the logits by T shrinks the gradients of the soft term by roughly 1/T^2, so without correction that term would quietly fade as you raise the temperature. Multiplying by T^2 cancels the shrinkage, keeping the two terms comparable in size whatever T you pick.

Low-Rank Factorization: Matrices on a Budget

Low-rank factorization is the most linear-algebra-flavoured tool in the model compression kit, and once it clicks it feels almost like cheating. A fully connected layer is just a multiply by a weight matrix W. If W is, say, 1000 \times 1000, that is a million numbers to store and a million multiply-adds to run. The idea: replace that one fat matrix with the product of two thin ones — a 1000 \times r matrix times an r \times 1000 matrix — chosen so their product is close to the original W.

Why should two thin matrices be able to stand in for a fat one? The answer is SVD (singular value decomposition), which writes any matrix as a sum of simple rank-1 pieces, each tagged with a 'singular value' that measures how much that piece matters. In real trained layers the singular values fall off fast: a handful of directions carry most of the matrix's 'energy,' and the long tail contributes almost nothing. So if we keep only the top-r directions — a truncated SVD — we discard the negligible tail while keeping the structure that does the work. That kept count r is the rank of our approximation.

Approximate an m×n matrix W by the product of a tall U and a wide V of rank r. Parameter count drops from m·n to r·(m+n).

Symbol by symbol: W is the original weight matrix with m rows and n columns, so it holds m \cdot n numbers. We approximate it as U V, where U is m \times r (tall and thin) and V is r \times n (short and wide); r is the rank we chose to keep. U holds m \cdot r numbers and V holds r \cdot n, so together they cost r(m+n). The win is real only when r(m+n) < m\,n, i.e. when r < \dfrac{m\,n}{m+n}. Plug in m=n=1000: the break-even rank is \dfrac{1{,}000{,}000}{2000}=500. Choose r=50 and you store 50 \cdot (1000+1000)=100{,}000 numbers instead of 1{,}000{,}000 — a clean 10× saving, and the matrix multiply gets ~10× cheaper too.

The same trick works on convolutions: one expensive conv can be split into two cheaper ones (for example a k\times k conv factored into a k\times 1 followed by a 1\times k, or a thick conv split into a thin 'bottleneck' projection and an expansion). And here is the honest limit: factorization only helps when a low rank is enough. If the layer genuinely needs a large r — say r=450 in our 1000\times1000 example, giving 450\cdot2000=900{,}000 params versus the original million — you barely save anything, and the approximation error may not even be worth it. Many modern architectures (with 1\times1 convs and bottlenecks) are already low-rank by design, leaving less to squeeze. Always check the singular-value falloff before assuming factorization will pay.

import numpy as np

W = np.random.randn(1000, 1000)               # a big weight matrix: 1,000,000 params
U, S, Vt = np.linalg.svd(W, full_matrices=False)

r = 50                                         # keep only the top-50 directions
sqrtS = np.sqrt(S[:r])
U_r = U[:, :r] * sqrtS                          # 1000 x 50
V_r = (sqrtS[:, None]) * Vt[:r, :]             # 50 x 1000

W_approx = U_r @ V_r                            # close to W if singular values decay fast
# stored params: 50*(1000 + 1000) = 100,000  vs  1,000,000  ->  ~10x smaller

Combining Techniques and Recovering Accuracy

None of these tools are rivals — they are a toolbox, and the strongest results come from stacking them. A typical aggressive pipeline: start by using knowledge distillation to train a smaller architecture that already punches above its size, then apply structured pruning to trim the channels that survived but aren't earning their keep, and finally (next guide) quantize to shrink every remaining number. Each step is a different lever on the same goal, and model compression in practice means choosing and sequencing them for your particular budget.

1. Establish a baseline: measure the full model's accuracy, size, and latency so every later change is judged against real numbers, not vibes.
2. Distill into a smaller architecture using the teacher–student loss, then fine-tune the student to convergence.
3. Structured-prune a slice of channels (e.g. ~10%), guided by an importance score, then fine-tune to recover the accuracy you just lost.
4. Repeat prune → fine-tune iteratively until you bump into your accuracy floor or your latency/size target.
5. Quantize (next guide), then fine-tune or calibrate to absorb the rounding error from fewer bits.
6. Re-validate on the full test set AND on your known failure modes before you ship.

How far should you push? Treat it as an explicit budget. Pick the constraint that actually binds — a latency target, a memory ceiling, an app download size — and the accuracy floor you refuse to drop below. Then compress until you hit one or the other, watching the accuracy-versus-size curve as you go. Early compression is nearly free: you are deleting genuine dead weight, so the curve stays almost flat. At some point it bends sharply downward — that knee is your stopping point. Pushing past it trades a lot of accuracy for a little size, which is rarely a good deal.

That leaves the one technique we have only foreshadowed: quantization — representing weights and activations with 8-bit integers (or fewer) instead of 32-bit floats. It is often the largest 'free lunch' of all: a roughly 4× shrink in size and, unlike unstructured pruning, a real speed-up on hardware built for integer math, frequently with almost no accuracy loss. It is important enough to earn its own guide — the next one. After that we will look at the runtimes and accelerators (ONNX, TensorRT, and the hardware stack) that turn all this compression into actual deployed speed.