dropout

A neural network with too much capacity can memorize its training set by building fragile cliques of neurons that only work together — one detector that only fires when three specific others do — a kind of co-adaptation that fails the moment the input differs slightly. Dropout (Srivastava et al., 2014) breaks up these cliques with deliberate sabotage: on every training step it randomly switches off a fraction of the neurons, so no neuron can rely on any particular partner being present. Forced to work amid unpredictable absences, each neuron must learn a feature that is useful on its own, which yields a more robust, redundant, less overfit network.

Mechanically, during training each activation is kept with probability 1−p and zeroed with probability p (a typical p is 0.5 for fully-connected layers, smaller for convolutional ones). To keep the expected magnitude of the signal unchanged so the next layer sees a consistent scale, 'inverted dropout' divides the surviving activations by 1−p. At test time dropout is turned off entirely and the full network is used; the 1/(1−p) scaling during training is precisely what makes the trained-with-noise network match the full network's expected output, so no test-time adjustment is needed.

There is a beautiful interpretation: because each step trains a different random sub-network (a different subset of neurons), dropout approximately trains an exponentially large ensemble of sub-networks that share weights, and using the full network at test time approximates averaging that ensemble's predictions. Ensembling is one of the most reliable ways to improve generalization, and dropout buys a cheap version of it inside a single model.

In modern computer vision the picture has shifted. Batch normalization already provides regularization and largely supplanted dropout inside convolutional backbones, so you rarely see heavy dropout in a modern CNN's conv layers. But dropout is alive and central in transformers — applied to attention weights and MLP activations in ViT, DETR, and CLIP — and a spatial cousin, stochastic depth (DropPath), which randomly drops entire residual blocks during training, is a standard regularizer for very deep ResNets and vision transformers.