lottery ticket hypothesis

Imagine that hidden inside a freshly initialized dense network is a small, lucky subnetwork — a winning lottery ticket — whose random starting weights are already so well-matched to the task that, trained alone, it reaches the full network's accuracy in the same number of steps. The hypothesis claims such sparse subnetworks reliably exist; the dense network's success is partly the success of buying many tickets at once.

You find a ticket by iterative magnitude pruning: train the dense network, remove the smallest-magnitude weights, reset the survivors to their original initial values, and repeat. The resulting sparse mask plus those exact early weights is the ticket. The reset is essential — pruning the same connections but reinitializing them randomly trains far worse. For large networks the weights must be rewound not to step zero but to an early checkpoint, a refinement called stability of the lottery ticket.

The result reframes pruning: instead of compressing a trained model afterward, it suggests trainable sparse structure is present at initialization. The honest caveat is that the winning mask is discovered only by training the full network first, so this is not yet a recipe for training sparse networks from scratch.