backpropagation
Backpropagation is how a network learns: after a forward pass produces a prediction and a loss (a number measuring how wrong it was), backprop figures out how much each individual weight contributed to that error, so each weight can be nudged to reduce it. The clever part is doing this efficiently for millions of parameters at once. The idea is to assign blame backward, layer by layer: the output layer's error is known directly, and you propagate that error to earlier layers using the chain rule of calculus.
Backpropagation is reverse-mode automatic differentiation applied to the loss. The chain rule says that to get the gradient of the loss L with respect to an early parameter, you multiply the local derivatives along the path from that parameter to L. Concretely, starting from the output you compute the error signal delta^(L) = dL/dz^(L), then propagate it backward with delta^(l) = (W^(l+1))^T·delta^(l+1), elementwise-multiplied by phi'(z^(l)) (the transpose sends error to the previous layer, and phi' applies the activation's local slope), and you read off the parameter gradients dL/dW^(l) = delta^(l)·(a^(l-1))^T and dL/db^(l) = delta^(l).
For a function with many inputs (parameters) and one output (the scalar loss), reverse-mode differentiation computes all gradients in a single backward pass costing about the same as one forward pass, vastly cheaper than perturbing each parameter individually (which would cost one forward pass per parameter, hopeless at billions of weights). This asymmetry is the entire reason deep learning is computationally feasible. The gradients are then handed to an optimizer (SGD, Adam) that takes a small step against them.
Backprop only computes gradients; it does not by itself guarantee learning. Its multiplicative chain is exactly what makes very deep or recurrent networks suffer vanishing or exploding gradients, motivating ReLU activations, careful weight initialization, normalization layers, and residual (skip) connections. ResNet's skip connections create a gradient shortcut so the error signal reaches early layers undiminished, which is why networks with hundreds of layers became trainable. Every vision model, from CNNs to ViTs to diffusion U-Nets, is trained by backpropagation.
For a single linear unit y_hat = wx + b with squared-error loss L = (y_hat - y)^2, the chain rule gives dL/dw = 2(y_hat - y)·x and dL/db = 2(y_hat - y). The shared factor 2(y_hat - y) is the error signal delta; backprop computes it once and reuses it for both gradients.
Why it matters: backprop is exact, not an approximation; given the computational graph it computes the true gradient up to floating-point error. What is approximate is the optimization (stochastic, on mini-batches) and the model, not the differentiation.