automatic differentiation
/ aw-toh-MAT-ik dif-er-en-shee-AY-shun /
Automatic differentiation is a technique that lets a computer work out, exactly and automatically, how sensitive a calculation's result is to each of its inputs — that is, the derivatives, or slopes. Training a neural network is all about these slopes: to nudge a weight in the right direction, you must know whether increasing it would raise or lower the error, and by how much. With millions of weights, doing this by hand is unthinkable. Automatic differentiation does it for you, no calculus homework required.
It's worth distinguishing from two cousins. You could try to write out the derivative formula by hand (symbolic differentiation), but for a big network the formula explodes into an unmanageable mess. Or you could nudge each input a tiny bit and watch the output (numerical differentiation), but that's slow and imprecise. Automatic differentiation is the clever third way: it breaks the calculation into elementary steps — the computational graph — and applies the chain rule of calculus step by step, getting answers that are exact (no approximation) and fast.
In practice, modern deep-learning frameworks do this behind the scenes. You write ordinary code that computes the network's output; the framework quietly records every operation, then runs the graph backward to hand you the gradient — the full bundle of slopes for every parameter at once. This backward pass is exactly what backpropagation is, viewed as a special case of automatic differentiation. It's one of the quiet pieces of engineering that makes training huge models feel almost effortless to the programmer.
Suppose your output is computed as y = (w × x + b), then squared into an error. Automatic differentiation records those two operations, then walks back: the squaring contributes one factor, the multiply-and-add contributes another, and the chain rule multiplies them to tell you exactly how the error responds to w — with no formula written by you, and no approximation.
Autodiff applies the chain rule along the recorded graph — exact slopes, computed automatically.
Automatic differentiation is exact, not an approximation — that's what separates it from the "nudge and measure" numerical method. Backpropagation is simply automatic differentiation applied, in reverse, to a neural network.