Making It Actually Train: Initialization, Generalization, and the Road to CNNs

Why initialization decides life or death

We ended the last guide with a cliffhanger: backpropagation tells us how to change every weight, but it never says what those weights should be before the very first step. And they cannot be left undefined — a network is just a big pile of numbers, and on step zero every one of them has to hold some value. That single choice, made before any learning happens, decides whether the network springs to life or sits frozen forever. This is the job of weight initialization, and it is the difference between a model that trains and one that quietly fails.

Start with the most tempting bad idea: set every weight to zero (or, just as bad, set them all to the same nonzero number). It feels neutral and fair. It is fatal. Consider any fully connected layer where many neurons all look at the same inputs. If two neurons have identical weights, they compute the identical output for every input. In the backward pass they then receive the identical gradient, so the update nudges them in the identical direction — and they stay identical forever. The whole layer behaves as if it had a single neuron copied many times. This is the symmetry problem: the neurons can never specialise into different feature detectors because nothing ever breaks the tie between them.

So we go random. But random is not enough — the scale of those random numbers matters enormously, and this is the second naive failure. Picture a signal flowing forward through a deep stack of layers. If the weights are a bit too large, each layer amplifies the signal, and after many layers the activations blow up toward huge numbers (and so do the gradients on the way back — the exploding case). If the weights are a bit too small, each layer shrinks the signal, and after many layers it withers to almost nothing — which is exactly the vanishing gradient problem we met in guide 4, where the learning signal fades to zero before it can reach the early layers. Either way, learning stalls.

From this we can state the whole goal of good initialization in one sentence, and it will guide everything in the next section: keep the typical size of the activations (going forward) and the gradients (going backward) roughly constant as they pass through many layers. Not growing, not shrinking — stable. If we can engineer that stability at step zero, training has a fighting chance from the very first gradient step.

Variance-preserving initialization: Xavier and He

Let us turn "keep the scale stable" into a recipe, using almost no probability — just one plain fact about adding up random numbers. A neuron computes a weighted sum of its inputs: it takes n_in inputs, multiplies each by a weight, and adds them all together. Here is the key intuition: when you add up many independent random contributions, the spread of the total grows with how many you add. Add up more terms, and the sum sloshes around over a wider range. A neuron with 256 inputs therefore produces a much wider-ranging output than one with 4 inputs, even if the individual weights are the same size.

That immediately suggests the fix: if more inputs make the sum bigger, then scale the weights down in proportion so the output keeps the same spread as the input. The right amount turns out to be roughly 1/√(n_in) — divide the random weights by the square root of the fan-in. This is the heart of weight initialization schemes. The two most famous come from making this precise for two different activation functions. Xavier (also called Glorot) initialization is tuned for symmetric, S-shaped activations like sigmoid and tanh. He initialization is tuned for the rectified linear unit (ReLU) — the activation we leaned on in guides 2 and 4 — and it carries one extra twist.

He initialization: draw each weight from a normal distribution with this variance.

Read this slowly. Var(W) is the variance of the weights — a measure of how spread out the random weight values are (its square root, std(W), is the standard deviation, the typical distance of a weight from zero). n_in is the fan-in: the number of inputs feeding into the neuron. The formula says: choose a weight spread of exactly 2/n_in. The n_in in the denominator is our shrink-as-inputs-grow rule — a neuron with more inputs gets smaller weights, so the summed output keeps a stable spread. The 2 in the numerator is the ReLU correction: ReLU sets every negative input to zero, so on average about half the neuron's contributions are silenced. Halving the live inputs would halve the output spread, so we multiply the variance by 2 to compensate and land back at a stable scale. Xavier uses the same form but without the factor of 2 (Var(W) = 1/n_in), because sigmoid/tanh do not zero out half their inputs. The whole point of either choice is the goal from section 1: keep the activation variance roughly constant layer after layer, so signals neither explode nor vanish across depth.

Make it concrete with a number. Suppose a layer's neurons each have 256 inputs, and we feed them into ReLU, so we use He init. Then std(W) = √(2 / 256) = √0.0078125 ≈ 0.088. So we draw each weight from a normal distribution centred at zero with a standard deviation of about 0.088 — small, deliberate numbers, not the default 0-to-1 range a naive coder might reach for. For comparison, Xavier on the same layer would give √(1 / 256) = 0.0625. The factor-of-2 difference between 0.088 and 0.0625 is exactly the ReLU correction made visible — modest, but compounded over dozens of layers it is the difference between a healthy signal and a dead one.

import numpy as np

def he_init(n_in, n_out):
    # ReLU layers: variance 2 / n_in  ->  std = sqrt(2 / n_in)
    std = np.sqrt(2.0 / n_in)
    W = np.random.randn(n_out, n_in) * std   # break symmetry + right scale
    b = np.zeros(n_out)                       # biases can start at zero
    return W, b

def xavier_init(n_in, n_out):
    # sigmoid / tanh layers: variance 1 / n_in (no factor of 2)
    std = np.sqrt(1.0 / n_in)
    W = np.random.randn(n_out, n_in) * std
    b = np.zeros(n_out)
    return W, b

W, b = he_init(256, 128)
print(W.std())   # ~0.088, matching sqrt(2/256)

Overfitting vs generalization

Good initialization buys us a network that can train. But a network that drives its training loss to zero has not necessarily learned anything useful. The real prize is generalization: doing well on new images it has never seen, not just the ones it studied. The classic failure is overfitting, and the analogy is exact. Imagine a student who, instead of understanding the material, memorises the exact answer key for last year's practice exam. On those practice questions they score a flawless 100%. On the real exam, with new questions, they fall apart. An overfit network has done the same thing: it has memorised the training set instead of learning the underlying patterns.

The figure shows how we detect this, and it ties straight back to the loss function from guide 3 — the single number the network minimises. We measure the loss on two separate sets of images. The training loss is computed on the data the network learns from; the validation loss is computed on held-out images it never trains on. Early on, both fall together — the network is learning genuine patterns that help everywhere. But at some point the curves diverge: training loss keeps dropping toward zero while validation loss bottoms out and starts climbing back up. That widening gap is the visual fingerprint of overfitting. The network is now improving on the answer key while getting worse at the real exam.

Behind this sits the bias–variance tradeoff, the central tension in all of machine learning. A model that is too simple cannot capture the real pattern — it makes the same systematic mistakes no matter what data you give it. We call this underfitting, or high bias: like fitting a straight line to a curve. A model that is too flexible does the opposite — it contorts itself to fit every quirk and noise speck in the training data, so its predictions swing wildly depending on which exact examples it saw. That is overfitting, or high variance. The art is landing in the middle: flexible enough to capture the true signal, rigid enough to ignore the noise. This matters acutely for a multilayer perceptron on images, because — as the next section makes painfully concrete — these networks have an enormous number of weights, giving them more than enough flexibility to memorise. MLPs on images overfit easily and almost by default.

The full training loop, in practice

We now have all the pieces to upgrade guide 4's idealized "forward, loss, backward, update" loop into a real-world workflow — the kind of checklist a practitioner actually follows, and the kind of script you should be able to read line by line by the end of this guide. The new ingredients beyond guide 4 are the ones we just learned: splitting the data into separate sets, initializing weights sensibly, and watching for overfitting while we train.

1. Split your data into three disjoint sets: a training set (the network learns from this), a validation set (you watch this during training to detect overfitting and tune choices), and a test set (touched only once, at the very end, for an honest final score).
2. Initialize the weights with the scheme that matches your activations — He for ReLU layers, Xavier for sigmoid/tanh — so signals start at a stable scale (section 2). Set biases to zero.
3. Loop over epochs (one epoch = one full pass through the training set), and inside each epoch loop over mini-batches (small chunks of, say, 32 or 64 images at a time).
4. For each mini-batch run the four-step cycle: forward pass to get predictions, compute the loss, backward pass to get gradients, then update every weight by stepping against its gradient.
5. At the end of each epoch, measure BOTH the training loss and the validation loss. Watching both is how you catch overfitting the moment the validation curve turns upward.
6. Tune the learning rate (too high diverges, too low crawls), and stop when the validation loss stops improving (early stopping). Only then report the test-set score.

# A real training loop, stripped to its essentials
W = initialize_weights(layer_sizes, scheme="he")   # section 2: stable scale

best_val = float("inf")
for epoch in range(max_epochs):
    for x_batch, y_batch in train_loader:          # mini-batches
        preds = forward(W, x_batch)                # 1. forward propagation
        loss  = loss_fn(preds, y_batch)            # 2. loss (guide 3)
        grads = backward(loss)                     # 3. backprop / autodiff (guide 4)
        W     = update(W, grads, learning_rate)    # 4. step against the gradient

    train_loss = evaluate(W, train_set)
    val_loss   = evaluate(W, val_set)              # watch BOTH curves
    print(epoch, train_loss, val_loss)

    if val_loss < best_val:                        # early stopping:
        best_val, best_W = val_loss, W             #   remember the best model
    elif val_loss > best_val:                      #   validation turned upward
        break                                      #   -> stop before overfitting

test_score = evaluate(best_W, test_set)            # touched only once, at the end

Three lines in that loop carry the whole weight of this track. `forward(...)` is the forward propagation from guides 1–3 — pixels in, predictions out. `backward(...)` is the backpropagation engine from guide 4, computing every gradient by autodiff. And the very first line, `initialize_weights(..., "he")`, is the weight initialization from this guide that gives the other two a fighting start. One last note on `update`: the plain "step against the gradient" we have used is the simplest optimizer, called SGD. Real training usually uses smarter cousins — SGD with momentum, which builds up speed in consistent directions, and Adam, which adapts the step size per weight. We name them only; their internals belong to a later optimization track. The shape of the loop stays exactly the same.

Why MLPs struggle with real images

We can now build a network, initialize it well, train it, and guard against overfitting. So why does this whole track end by telling you the multilayer perceptron is the wrong tool for real images? Because of two concrete failures, and the first is a sheer wall of numbers. Recall from guide 1 that to feed an image into a fully connected layer we have to flatten it — unroll the 2D grid of pixels into one long vector. A modest colour image of 224×224 pixels with 3 colour channels flattens to 224 × 224 × 3 = 150,528 input numbers. That is the input to a single layer.

Parameter count of one fully connected layer: a weight for every input–output pair, plus one bias per output.

Unpack this and the blow-up becomes undeniable. n_in is the number of inputs (150,528 for our image), n_out is the number of neurons in the layer, and the layer needs one weight for every connection between an input and an output — that is the n_in × n_out term — plus one bias per neuron, the + n_out term. Now plug in a single hidden layer of just n_out = 1,000 neurons: params = 150,528 × 1,000 + 1,000 = 150,529,000 ≈ 150 million weights. One hundred and fifty million numbers, for one layer, before we have even built a real network. That is astronomically wasteful, and — recalling the bias–variance lesson — a model with 150 million free parameters is a guaranteed overfitting machine: it has more than enough capacity to memorise the training set outright. For contrast, hold this number in mind: the weight-sharing convolution we meet next can match a fully connected layer's job with only a few thousand parameters, a reduction of four or five orders of magnitude.

The second failure is deeper than mere size — it is structural. The moment we flatten the image, we destroy its spatial layout. Remember from guide 1 that a pixel's meaning lives in its neighbourhood: an edge is a local jump between adjacent pixels, a texture is a local pattern. Flattening into a tensor of shape (150528,) throws all of that away — pixel (0, 0) and pixel (0, 1), which sit side by side in the image, become just "input #1" and "input #2" in a list, no more related than two random entries. The MLP has no idea which inputs were neighbours. It must rediscover, from scratch and from data alone, the spatial structure we needlessly shredded.

And it gets worse, because a multilayer perceptron has no built-in notion of translation invariance. Suppose it has painstakingly learned, from many examples, a set of weights that recognise a cat in the top-left corner. Now show it the same cat shifted a few pixels to the right. To the MLP this is an almost entirely different input vector lighting up a different bank of weights — it does not automatically know that a cat is a cat wherever it appears. It would have to learn the cat detector again, independently, at every possible location in the image. That is hopelessly inefficient: it multiplies the learning problem by the number of positions, and squanders examples that should all reinforce one shared concept.

The bridge to convolutional networks

We will not teach convolution in full here — that is the opening of track 6, and it deserves its own runway. But we can plant the three big ideas that fix the two MLP failures we just diagnosed, so you arrive at the next track already knowing what problem each one solves. Think of this as reading the back cover of the next book.

Idea one — local connectivity. Instead of wiring every neuron to all 150,528 inputs, a convolutional neuron looks at only a small patch of the image — say a 3×3 window. This directly respects the lesson from guide 1 that meaning lives in neighbourhoods, and it slashes the connections per neuron from 150,528 down to a handful. The wall of parameters starts crumbling immediately.

Idea two — weight sharing. This is the masterstroke. We take one small set of patch-weights — a filter — and slide the same filter across every position in the image, reusing the identical weights everywhere. Two enormous wins fall out at once. First, parameters collapse: instead of 150 million, a filter might have just 9 or 27 weights, used millions of times — the four-to-five-orders-of-magnitude reduction we previewed. Second, and just as important, it grants translation awareness for free: because the same cat-detecting filter is applied at every location, a cat is recognised wherever it appears. The MLP's hopeless "re-learn it at every position" problem simply vanishes — one filter, learned once, works everywhere.

Idea three — keep the image as a spatial tensor. Because the filter slides over a 2D layout, a convolutional network never flattens the image at the start. It keeps it as a tensor with its height, width, and channels intact, so neighbourhood relationships survive all the way through the network. The spatial structure the MLP threw away is now preserved and exploited. Notice how cleanly these three ideas line up against section 5's two complaints: local connectivity plus weight sharing kill the parameter explosion, and weight sharing plus the spatial tensor restore the structural understanding.

Here is the reassuring punchline, and the right note to end the whole track on: convolution changes only how neurons are wired — not what they are. Every single thing you have built across these five guides carries over unchanged. A convolutional unit is still a neuron computing a weighted sum and an activation (guide 1); filters stack into layers and depth (guide 2); the output is still scored by logits, softmax, and a loss function (guide 3); it still learns by forward and backward passes through a computational graph (guide 4); and it still needs sensible initialization and overfitting control (this guide). Even the multilayer perceptron does not disappear — a CNN almost always ends in a few fully connected layers to make the final decision. You are not starting over. You now own the complete neural-network foundation of computer vision, and you are ready to teach those neurons to see in two dimensions. That is exactly where track 6 begins.