weight initialization
/ wayt ih-nish-ul-ih-ZAY-shun /
Weight initialization is the choice of what numbers a network's weights start out as, before any learning happens. Since training works by nudging weights step by step from wherever they begin, the starting point genuinely matters — like where you set down on a hilly landscape before walking downhill toward the lowest valley. Start in a bad spot and you might get stuck, descend painfully slowly, or never get going at all. Initialization is picking that starting spot wisely.
Two naive choices both fail. Setting every weight to the same value (say, all zeros) is fatal: identical weights mean every neuron in a layer computes exactly the same thing and receives the same update forever, so the network can never break the symmetry to specialize — it behaves like a single neuron no matter how big it is. So weights must start random and different. But the scale of that randomness is delicate: too large and the signals blow up (exploding gradients); too small and they fade away (vanishing gradients). Either way, learning stalls.
The fix is to scale the random values just right, tuned to how many connections each neuron has, so signals keep a healthy size as they pass through the layers. The two famous recipes are Xavier (or Glorot) initialization, designed for sigmoid and tanh, and He initialization, designed for ReLU. They aren't deep — just sensible scaling rules — but they made a real difference, helping deep networks train reliably where careless initialization would have left them stuck. It's a small upfront decision with an outsized effect on whether training even succeeds.
Set every weight in a network to zero and run training: every neuron in a layer stays identical to its neighbors forever, and the network learns nothing useful. Set the weights to small random numbers scaled by the He recipe (suited to ReLU), and the same network trains smoothly. The only thing that changed was the starting numbers.
All-zero weights freeze a network; well-scaled random weights let it actually learn.
Never initialize all weights to the same constant — identical neurons can never differentiate (the "symmetry" problem). Random and well-scaled is the rule; the scale should match your activation (Xavier for sigmoid/tanh, He for ReLU).