Stacking Neurons: Layers, Depth, and the Multilayer Perceptron

From one neuron to a whole layer

In the last guide we built the smallest learning machine: a single artificial neuron. It takes an input vector x, forms a weighted sum z = w·x + b using its private weights w and bias b, and then squashes that sum through an activation function. One neuron can only ask one near-yes/no question of the image — 'does this one pattern fit?' To recognise something as rich as a handwritten digit, one question is nowhere near enough.

So we do the obvious thing: copy the neuron many times. Each copy reads the same input vector x — the same pixels — but carries its own weights and bias, so each one hunts for a different pattern. Picture it not as one voting panel but as a whole row of panels side by side: one panel scores 'roundness', another 'a dark centre', another 'a vertical stroke'. Wire a group of neurons so they all stare at the same input in parallel and you have built a fully connected layer — 'fully connected' because every input feeds every neuron.

Read this as 'do guide 1's weighted sum for every neuron at the same time, then squash.' Here x is the input vector of length n_in (the number of inputs — in our tiny example, 2 pixel-features). W is a matrix with n_out rows and n_in columns; row j is exactly neuron j's weight vector w from guide 1. The product W x is a length-n_out list of weighted sums — one dot product per row. b is the bias vector of length n_out, adding each neuron's personal offset. g is the activation, applied to each entry independently. The result h has length n_out, one number per neuron. Concretely with 2 inputs and 3 neurons: x is length 2, W is 3×2, W x gives 3 sums, +b keeps 3, and h is length 3. Nothing new is happening — it is z = w·x + b stacked three times, with the matrix simply keeping the books on the three dot products. If x = [1, 2] and the three weight rows are [0.5, −0.5], [1, 0], [−1, 2] with biases [0, 1, −3], the raw sums W x come out to −0.5, 1.0, 3.0; adding the biases gives z = [−0.5, 2.0, 0.0]; a ReLU squash (keep positives, zero out negatives) yields h = [0, 2.0, 0] — neuron 2 fired, the others stayed silent.

import numpy as np

x = np.array([1.0, 2.0])           # input vector, length n_in = 2

# Each ROW of W is one neuron's weights; n_out = 3 neurons
W = np.array([[ 0.5, -0.5],        # neuron 1 looks for this pattern
              [ 1.0,  0.0],        # neuron 2
              [-1.0,  2.0]])       # neuron 3   ->  W has shape (3, 2)
b = np.array([0.0, 1.0, -3.0])     # one bias per neuron, length 3

z = W @ x + b                      # 3 weighted sums at once -> [-0.5, 2.0, 0.0]
h = np.maximum(0.0, z)             # ReLU, elementwise       -> [ 0.0, 2.0, 0.0]
print(z, h)

Stacking layers: the multilayer perceptron

Stack these layers and you get a multilayer perceptron (MLP): several fully connected layers chained in a line, where the output vector of one layer becomes the input vector of the next. Layer 1 turns the pixels into a list of pattern-scores; layer 2 treats those scores as its input and produces a new list; and so on. A layer neither knows nor cares whether its input arrived from another layer or from raw pixels — to it, an input vector is an input vector.

Three names you will use forever. The input layer is just the raw data going in — for us, the flattened row of pixels (it does no computation; it is the picture itself). The hidden layers are the in-between layers whose outputs we never directly read off as the answer; they hold the network's learned intermediate features. The output layer is the final layer, whose numbers we interpret as the answer — the class scores. 'Hidden' simply means 'neither the input nor the final output' — buried in the middle.

Here is the idea that powers all of deep learning. Each layer can only build from what the layer below hands it. So the first hidden layer, reading raw pixels, can learn only simple things — a bright edge here, a dark blob there. The next layer reads those edge-and-blob scores and can combine them into something richer — a corner, a curve, a loop. A layer above that can assemble loops and strokes into 'the top of a 9' or 'a cat's ear'. Depth builds a hierarchy of features: complex patterns expressed as combinations of simpler patterns from the layer beneath. No single neuron is told what to look for — the useful intermediate features emerge from training (guide 4).

Two dials describe an MLP's shape. Depth is the number of layers — how many times the data gets re-combined on its way through. Width is the number of neurons in a layer — how many different patterns that layer can look for at once. Both are design choices the engineer makes before training, not values the network learns; we call such knobs hyperparameters. A wider layer can spot more patterns at one level; a deeper network can build more abstract patterns. Picking them well is part craft, part experiment.

Forward propagation: data flowing layer by layer

Forward propagation (also called a forward pass) is simply the act of pushing an input through every layer, in order, to produce the final output. You feed in the image, layer 1 transforms it, layer 2 transforms that, and so on until the output layer hands you the scores. Crucially, with the weights and biases fixed, this is completely deterministic — the same image always yields the same output. No learning happens during a forward pass; learning is a separate process — adjusting the weights — which we tackle in guide 4. For now the weights are frozen and we just watch data flow.

1. Start with the input: the flattened pixel vector becomes h⁰ = x.
2. Layer 1: compute z = W¹h⁰ + b¹, then apply the activation to get h¹.
3. Layer 2: feed h¹ in and compute h² the same way.
4. Repeat layer by layer — each layer's output becomes the next layer's input.
5. Last layer: its output is the prediction — the class scores.

Do not let the superscript scare you — the parenthesised (l) just means 'belonging to layer l'. So W⁽ˡ⁾ is layer l's weight matrix, b⁽ˡ⁾ its bias, g⁽ˡ⁾ its activation, and h⁽ˡ⁾ the vector it outputs. The formula reads: to get layer l's output, take the previous layer's output h⁽ˡ⁻¹⁾, run it through this layer's own weighted-sum-plus-bias, and squash. It is exactly the single-layer rule h = g(Wx + b) from section 1, except the input is now named h⁽ˡ⁻¹⁾ (the layer below) instead of x. The base case h⁽⁰⁾ = x just says 'layer 0 is the input itself.' Apply the rule for l = 1, 2, … up to the last layer, and the final h is your prediction.

Let us watch one value travel through a tiny 2→2→1 network. Feed in x = [2, −1]. Layer 1 has weight rows [1, 1] and [1, −1] with biases [0, −4]: the raw sums are 2+(−1)=1 and 2−(−1)=3, so after the bias z¹ = [1, −1]. A ReLU keeps the 1 but clips the −1 up to 0, giving h¹ = [1, 0] — the second hidden neuron is switched off for this input. Layer 2 (the output) has weights [2, 5] and bias 1: its score is 2·1 + 5·0 + 1 = 3. So the journey is [2, −1] → [1, −1] → [1, 0] → 3. Notice the activation actually changed the answer: without the ReLU clip, the output would have been 2·1 + 5·(−1) + 1 = −2, not 3.

import numpy as np

def relu(z):
    return np.maximum(0.0, z)

x  = np.array([2.0, -1.0])             # h^(0): the input

# ---- Layer 1 (hidden) ----
W1 = np.array([[1.0,  1.0],
               [1.0, -1.0]])           # shape (2, 2)
b1 = np.array([0.0, -4.0])
h1 = relu(W1 @ x + b1)                  # z=[1,-1] -> relu -> [1., 0.]

# ---- Layer 2 (output) ----
W2 = np.array([[2.0, 5.0]])            # shape (1, 2)
b2 = np.array([1.0])
out = W2 @ h1 + b2                      # 2*1 + 5*0 + 1 = 3.0  (a raw score)

print(h1, out)                         # [1. 0.] [3.]

Choosing the squash: ReLU vs sigmoid

In guide 1 we met two ways to squash a neuron's sum and promised a proper comparison. Here it is. The job of an activation function is to bend the straight weighted sum into something non-linear — without it, as section 5 will prove, the whole deep stack collapses to nothing. Two classic choices: the sigmoid is like a dimmer switch, smoothly sliding any input to a value between 0 and 1; the ReLU is like a one-way valve, blocking anything negative and letting positive values straight through unchanged.

Read both. ReLU(z) = max(0, z) outputs z when z is positive and 0 when z is negative — a hard hinge at the origin. The sigmoid σ(z) = 1/(1 + e^(−z)) squeezes any real z into (0, 1): a big positive z → near 1, a big negative z → near 0, and z = 0 → exactly 0.5 (since e⁰ = 1, so 1/2). The number that will matter most when the network learns is each function's slope — its derivative, i.e. how fast the output changes as the input nudges. The ReLU's slope is a flat 1 for every z > 0 and 0 for every z < 0: on the positive side the signal passes through undimmed. The sigmoid's slope is steepest near z = 0 (about 0.25) and shrinks toward 0 for large |z| — out in the tails the curve goes flat. Hold onto 'slope', because in guide 4 the network learns by passing signals backward, multiplied by exactly these slopes.

This is why modern hidden layers almost always use ReLU. It is dirt cheap (a single max), and on the positive side its slope of 1 lets the learning signal flow through layer after layer without shrinking — whereas the sigmoid saturates: feed it a largish input and its near-zero slope chokes the signal, a problem that compounds across depth into the vanishing gradients we will meet in guide 4. ReLU is not flawless, though: a neuron whose input stays negative for every example outputs 0 forever and has slope 0, so no learning signal ever reaches it — it is stuck. This dying ReLU pitfall is real but usually affects only a fraction of neurons (and variants like 'leaky ReLU' patch it). Sigmoid still has its place — at the output layer of a binary classifier, where we genuinely want a single number between 0 and 1 to read as a probability (guide 3). The headline: ReLU inside, save the saturating squashes for the output.

Why non-linearity matters: the universal approximation theorem

We keep insisting the activation function is essential. Time to prove it. Suppose we got lazy and dropped the activation, so each layer is a pure linear map — just multiply by a matrix and add a bias. What does stacking two such layers actually compute?

Walk the algebra. The first layer turns x into W₁x + b₁. The second layer multiplies that by W₂ and adds b₂, giving W₂(W₁x + b₁) + b₂. Distribute the W₂ and regroup: the x-terms collect into (W₂W₁)x and the constants into (W₂b₁ + b₂). But W₂W₁ is just some matrix — call it W' — and W₂b₁ + b₂ is just some vector, b'. So the two-layer stack equals W'x + b': a single linear layer. The same trick folds any number of activation-free layers into one. Intuitively, composing two flat maps gives another flat map — stretch a flat sheet, then stretch it again, and it is still flat. Without a non-linear g wedged between layers, depth buys you literally nothing.

Put a non-linear activation back between the layers and everything changes. The universal approximation theorem makes the payoff precise: an MLP with at least one hidden layer and a non-linear activation can approximate essentially any continuous function to any accuracy you want — provided the hidden layer is allowed enough neurons. 'Any continuous function' includes the fiendishly curved one that maps a grid of raw pixel values to 'cat' versus 'dog'. In principle, then, a plain stack of neurons is expressive enough to be a vision system: the theorem says the right weights exist.