weight initialization

Weight initialization is the choice of the random starting values for a network's parameters before training begins. It sounds trivial (they get updated anyway), but the starting point profoundly affects whether training works at all. Set the weights too large and signals and gradients blow up as they pass through layers; too small and they shrink to nothing; all identical and every neuron in a layer computes the same thing forever. Good initialization keeps the scale of activations and gradients roughly constant from the first layer to the last, so learning can start cleanly.

The guiding idea is to preserve variance: as a signal passes through a layer, you want its variance to stay about the same rather than grow or decay. For a dense layer with fan-in n (number of inputs), a linear neuron's output variance is n times Var(w) times Var(x), so keeping output variance equal to input variance requires Var(w) about 1/n. Xavier/Glorot initialization (2010) sets Var(w) = 2/(fan_in + fan_out) to balance the forward and backward passes and is designed for symmetric activations like tanh; He/Kaiming initialization (2015) uses Var(w) = 2/fan_in to compensate for ReLU zeroing half its inputs and is the standard for ReLU-family networks.

Weights must be random, not constant: if every weight in a layer started identical, every neuron would receive the same gradient and remain identical forever, so randomness breaks symmetry and lets neurons specialize. Biases are usually initialized to zero (the weights already break symmetry). Initialization interacts with the rest of the recipe: normalization layers (batch or layer norm) and residual connections greatly reduce sensitivity to it, but it still matters; for example, initializing the last batch-norm gamma of each residual block to zero makes the block start as identity and stabilizes very deep nets.

Every CNN and Transformer relies on principled initialization. He initialization is the default for ReLU convolutional nets (ResNet); Transformers and ViTs use scaled variants (such as truncated-normal with small standard deviation, sometimes scaled by 1/sqrt(2 times num_layers) to keep residual streams stable at depth), and special tricks like zero-init residual branches or LayerScale stabilize training. In transfer learning the question changes entirely: you initialize not randomly but from pretrained weights (an ImageNet or CLIP backbone), which is why fine-tuning converges so much faster than training from scratch.