Learning = walking downhill on the loss
In the last guide, the loss function gave us a single number that scores how wrong the network's guess was: small loss means good predictions, large loss means bad ones. But scoring the mistake is only half the story. Learning means doing something about it — nudging the network's weights so that, next time, the loss comes out smaller. This guide is about exactly that machinery: how a network turns one error number into a precise instruction for every single weight.
Here is the picture to hold in your head. Imagine the loss as a hilly landscape. The horizontal coordinates are not north and east — they are the network's weights. Every possible setting of the weights is a spot on the terrain, and the height at that spot is the loss for those weights. Training is a hiker standing somewhere on this terrain in thick fog, trying to walk down to the lowest valley, where the loss is smallest. The hiker cannot see the whole map; they can only feel the slope right under their feet.
The tool that tells the hiker about the slope is the gradient. The gradient is a compass that points in the direction of steepest uphill — the way that makes the loss climb fastest. That is the exact opposite of what we want, so the move is simple: step in the opposite direction of the gradient and you head downhill, toward smaller loss. Read the slope, turn around, take a step. Repeat.
A curved loss surface with a marker descending in small steps toward the lowest point; arrows show each step moving opposite to the uphill slope.
This loop has a name: gradient descent. Stripped to its essence it is just *repeat: measure the slope, take a small step downhill.* Almost everything else in deep-learning training is bookkeeping around this one idea. The size of each step is controlled by a knob we get to choose, and that single number turns out to matter enormously.
The gradient-descent update for a single weight.
Read this as an instruction, left to right: replace the old weight w with a new value. The pieces are: w is one weight we are updating (the network has millions of them, and this rule applies to each). ∂L/∂w is the gradient — it answers 'if I nudge this one weight up a tiny bit, how much does the loss L change?' A value of +2 means the loss rises twice as fast as you raise w (this weight is pushing us uphill); a value of −3 means the loss falls as you raise w. η (the Greek letter eta) is the learning rate, the step size. The minus sign is the whole trick: it makes us move against the uphill gradient, i.e. downhill. Concretely: say w = 0.50, the gradient ∂L/∂w = +2.0, and η = 0.1. Then w ← 0.50 − 0.1×2.0 = 0.50 − 0.20 = 0.30. We lowered the weight because raising it would have raised the loss — exactly right.
So gradient descent is the easy part — it is one subtraction per weight. The hard part, the thing the rest of this guide solves, is hidden inside that ∂L/∂w. How do you actually compute the gradient — the slope of the loss with respect to every one of millions of weights — quickly, for a network with many layers? Doing it the obvious way is hopeless. The elegant answer is the computational graph and backpropagation.
The computational graph view of a network
Back in guide 2 we ran data forward through the network — multiply by weights, add biases, apply activations, layer after layer — to produce a prediction. That process is called forward propagation. To make gradients tractable, we now reframe it not as a wall of matrix algebra but as a wiring diagram: a computational graph. Same computation, far more revealing picture.
In a computational graph, each node is one small operation — a single multiply, an add, an activation, the loss — and the edges are the values flowing between them. Data enters on the left and flows right. Take the tiniest meaningful slice of a network — a single neuron feeding into a loss — and lay it out node by node:
# A tiny computational graph: one neuron, then a loss. # Inputs / parameters (the leaves of the graph): x = 2.0 # an input feature (a pixel value, say) w = 0.5 # the weight on that input (we want dL/dw) b = 0.1 # the bias # Each line below is ONE node. It knows how to compute its # output from its inputs (the forward direction), and -- # crucially -- it also knows its own LOCAL derivative rule. z = w * x + b # node: weighted sum -> z = 1.1 a = relu(z) # node: activation -> a = 1.1 (z > 0) L = loss(a, y) # node: how wrong we are -> one number
Here is why this picture is worth the trouble. Every node is a simple operation, and for a simple operation we know its local derivative off by heart. For the weighted-sum node, nudging w changes z at a rate of x. For the ReLU node, the output changes at a rate of 1 when its input is positive and 0 when negative. The loss node has its own known slope. None of these is hard on its own. The deep claim — which the next section proves — is that we can find the slope of the final loss with respect to any weight just by multiplying the local derivatives along the path that connects them. The graph turns one terrifying global derivative into a chain of trivial local ones.
Backpropagation: the chain rule, layer by layer
Now the heart of the guide — let's go slow. We have a final loss, and we want to know how much each weight, buried deep in the graph, is to blame for it. The mechanism is the chain rule from calculus, but you do not need to fear it. Picture blame flowing backwards like a relay race run in reverse. The loss hands a baton of blame to the node just before it; that node keeps its share, multiplies the baton by its own local sensitivity, and passes the rest further back. By the time the baton reaches a weight, it has been scaled by every link along the way — and what arrives is exactly that weight's gradient.
Let's trace one weight — the w from our tiny graph — end to end. The weight w affects the loss through a single path: w changes z, z changes the activation a, and a changes the loss L. The chain rule says: to get how the loss depends on w, multiply together how each link depends on the next.
Blame for weight w = (blame at the output) × (the activation's local slope) × (the input feeding the weight).
Take the three factors one at a time, reading right to left along the path. ∂z/∂w asks 'how fast does the weighted sum z change as I raise w?' Since z = w·x + b, raising w by 1 raises z by x — so ∂z/∂w = x, the input feeding the weight. ∂a/∂z is the activation's own local slope: for ReLU that is 1 when z > 0 and 0 when z < 0 (an open gate or a closed one). ∂L/∂a is how much the loss changes as the neuron's output a changes — this is the baton of blame handed down from the loss. Now plug in our numbers: x = 2, so ∂z/∂w = 2; z = 1.1 > 0, so the ReLU gate is open and ∂a/∂z = 1; and suppose the downstream loss hands back ∂L/∂a = 0.4. Multiply: ∂L/∂w = 0.4 × 1 × 2 = 0.8. That is a real, concrete gradient. Feed it into the update rule with η = 0.1 and the weight moves: w ← 0.5 − 0.1×0.8 = 0.42.
Notice we reused the values we already computed going forward — x, the sign of z, the prediction a. That is the engine of the backward pass: run the network forward once and remember each node's output; then start at the loss, compute its gradient, and push that gradient backwards through the graph, at each node multiplying by the local derivative you can read off from the stored forward values. The bias falls out of the very same machine: ∂L/∂b = (∂L/∂a)(∂a/∂z)(∂z/∂b) = 0.4 × 1 × 1 = 0.4, since raising b raises z one-for-one (∂z/∂b = 1).
Here is the payoff that makes deep learning possible at all. You might imagine measuring each weight's gradient by experiment: nudge weight 1, re-run the whole network, see how the loss changed; then put it back, nudge weight 2, re-run, and so on. For a million weights that is a million forward passes per step — utterly hopeless. Backpropagation computes all the gradients in roughly a single backward sweep over the graph, costing about as much as one forward pass. Same answer, a million times cheaper. That is the entire reason we can train networks with billions of parameters.
The backward step for a whole layer, written once for all of its neurons at the same time.
This is the same single-weight chain rule, just written for an entire layer of neurons at once. δ^{(l)} (read 'delta at layer l') is the vector of blame arriving at layer l — one number per neuron, telling how much that neuron's pre-activation z is responsible for the loss. Read the right-hand side as a two-step recipe. First, (W^{(l+1)})^⊤ δ^{(l+1)}: take the blame the next layer already computed, δ^{(l+1)}, and pull it backwards through that layer's weight matrix (the transpose ⊤ just routes each downstream blame back to the neurons that fed it). Second, ⊙ g'(z^{(l)}): the symbol ⊙ means multiply element-by-element, and g'(z^{(l)}) is this layer's activation slope at each neuron — so we gate the incoming blame by how responsive each neuron actually was. In plain words: *pull the next layer's blame back through the weights, then gate it by this layer's activation slope.* Apply that formula layer by layer, from the output back toward the input, and you have filled in every δ — and hence every weight's gradient.
Automatic differentiation: how frameworks do it for you
When you use PyTorch or TensorFlow you never write the chain rule yourself — you build a model, call something like loss.backward(), and every gradient simply appears. So what is the engine doing? It is performing automatic differentiation (autodiff). It helps to first say clearly what autodiff is not.
It is not symbolic differentiation — it does not crank your network through algebra to produce one giant closed-form derivative formula. For a real network that formula would explode into millions of terms, and it is unnecessary. It is also not finite differences — it does not estimate slopes by nudging each input by a tiny ε and re-running, ∂L/∂w ≈ (L(w+ε) − L(w))/ε. That numeric trick is both slow (one re-run per weight — the very thing backprop avoids) and inaccurate (rounding error eats the answer). Autodiff is a third thing: exact and cheap.
What it actually does: during the forward pass it records the computational graph — every operation you perform is logged, in order, together with the inputs it saw and the local-derivative rule for that operation. Then, on the way back, it applies those stored local derivatives node by node in reverse, multiplying and accumulating exactly as the chain rule prescribes. That reverse sweep is precisely backpropagation generalised to arbitrary graphs — it even has a formal name, reverse-mode autodiff. Backprop is the special case for neural networks; reverse-mode autodiff is the same idea for any computation you can write down.
import torch # Leaves we want gradients for: requires_grad=True tells autodiff # to record every operation that touches them. x = torch.tensor(2.0) w = torch.tensor(0.5, requires_grad=True) b = torch.tensor(0.1, requires_grad=True) # FORWARD PASS -- autodiff silently builds the graph as we go. z = w * x + b # node logged: multiply + add a = torch.relu(z) # node logged: relu L = (a - 1.0) ** 2 # node logged: a simple squared-error loss # BACKWARD PASS -- replay the recorded graph in reverse. L.backward() print(w.grad) # = dL/dw, computed by the chain rule automatically print(b.grad) # = dL/db, from the very same backward sweep
The mental model to keep: think of every operation you write as quietly leaving behind a sticky note that says 'here is my local derivative.' Calling .backward() walks the stack of sticky notes from the loss back to the leaves, multiplying as it goes, and deposits the result in each parameter's .grad slot. Then your optimiser reads those .grad values and applies the w ← w − η ∂L/∂w step from Section 1. Forward to record, backward to differentiate, step to learn.
One full training step, start to finish
We now have every part. Let's bolt them into a single loop you can recite from memory — the loop that, repeated, turns a random network into one that sees. One step of training does four things in order, then moves on to the next batch of images.
- Forward pass — push a batch of images through the network with forward propagation to get its current predictions.
- Compute the loss — use the loss function to score those predictions against the true labels, collapsing all the errors into one number.
- Backward pass — run backpropagation / autodiff to get the gradient of that loss with respect to every weight.
- Update — take one gradient-descent step on every weight, w ← w − η ∂L/∂w. Then return to step 1 with the next batch.
A cycle diagram: data feeds a forward pass producing predictions, the loss scores them, a backward pass yields gradients, a weight update follows, and an arrow loops back to feed the next batch of data.
Two words of vocabulary, kept light. We rarely feed images one at a time; we process a batch (or mini-batch) — a handful of images, say 32 or 64 — together in one forward-and-backward sweep, which is faster on a GPU and gives a steadier, averaged gradient. When the loop has worked its way through every image in the training set once, that is one epoch. Training usually runs for many epochs, the loop spinning thousands or millions of times in total.
One training step in four lines. Repeated thousands of times, this block IS training.
Line by line, this is the four phases. ŷ = forward(x) pushes a batch x through the network to get predictions ŷ (phase 1). L = loss(ŷ, y) scores them against the true labels y, producing the single loss number (phase 2). g = backward(L) is the backward pass that produces g, the collection of gradients ∂L/∂w for every weight (phase 3). And w ← w − η ∂L/∂w nudges every weight a small step downhill (phase 4). That four-line block, run over batch after batch for many epochs, is the entire act of training a neural network. Everything in the next guide is about making this loop converge reliably instead of stalling or blowing up.
A preview of what can go wrong: vanishing gradients
Let's end with an honest warning that sets up the final guide. Look again at the backward pass: a deep weight's gradient is a product of local slopes, one factor for every layer the blame must travel through — ∂L/∂w = (slope) × (slope) × (slope) × … stretching all the way back. Products are fragile. If many of those factors are smaller than 1, the product collapses toward zero the deeper you go.
This is the vanishing gradient problem, and the sigmoid activation is the classic culprit. Recall from guide 2 that a sigmoid flattens out for large positive or negative inputs; in those flat regions its local slope is nearly zero (at its very best the slope peaks at only 0.25). String ten such layers together and the blame gets multiplied by something like 0.2 ten times over: 0.2^10 ≈ 0.0000001. By the time the error signal reaches the early layers it has dwindled to almost nothing — so the first layers barely update and effectively stop learning, even though they are the very layers meant to learn the most basic visual features.
The mirror-image failure is exploding gradients: if the factors are consistently larger than 1, the product blows up exponentially instead, and the weights lurch by huge amounts — the loss diverges, often all the way to NaN. Both pathologies share the same root cause you just learned: backpropagation multiplies a long chain of numbers, and long products of numbers either shrink to dust or explode, unless something keeps each factor close to 1.
So how do we keep those per-layer factors near 1? Part of the answer you have already glimpsed: swapping the saturating sigmoid for the rectified linear unit, whose slope is a clean 1 for every positive input, stops the signal from decaying through the open gates. But activation choice alone is not enough — you also have to start the weights at the right scale, so that signals neither grow nor shrink as they pass through each layer at the very first step. Getting that initial scale right, and the broader craft of making training actually converge and generalise, is exactly what the next and final guide of this track is about.