Three ways to get a derivative — and the gap the first two leave
By now this rung has shown you two ways to differentiate a function you can only evaluate, and warned you about both. Numerical differentiation by a finite difference is cheap and works on any black box, but it pays a tax: subtracting nearly equal values divides by a tiny h and runs into the step-size trade-off, so the best a plain central difference ever reaches is roughly 11 of your 16 double-precision digits. The other classical route is symbolic differentiation: feed the formula for f into a computer algebra system, apply the differentiation rules exactly, and read off a closed-form f'(x). That gives an exact expression — but it suffers expression swell: differentiating a nested product by the product rule roughly doubles the term count at each layer, so the symbolic derivative of a deeply composed function can balloon into millions of terms and crawl.
So there is a gap. We want what symbolic differentiation has — an exact derivative, with no h and no truncation error — but at the cost of numerical differentiation, a handful of arithmetic operations rather than a swollen formula. Automatic differentiation (AD, sometimes called algorithmic differentiation) fills exactly this gap. Its key insight: a program that computes f is, when you look closely, nothing but a long chain of elementary operations — add, multiply, sin, exp, sqrt — each of whose derivative we already know. If we carry derivative information alongside the value through every one of those steps, applying the chain rule mechanically at each, we arrive at the exact derivative of the whole composition. No formula is ever written out, no limit is ever taken.
Forward mode: carry a derivative alongside every value
The cleanest way to meet AD is the forward mode, and the cleanest way to implement it is the dual number. Imagine an arithmetic where every value carries a second slot: a number a is replaced by the pair (a, a'), read as 'value a, with derivative a'. Define the operations so that the second slot always tracks the derivative. Addition: (a, a') + (b, b') = (a + b, a' + b'). Multiplication, by the product rule: (a, a') * (b, b') = (a*b, a'*b + a*b'). For an elementary function like sine: sin of (a, a') = (sin a, cos(a) * a'), which is just the chain rule. Each rule is the familiar calculus derivative rule, applied to the second slot.
To differentiate f at a point x, seed the input as (x, 1) — value x, derivative dx/dx = 1 — and run the ordinary code for f, but with dual-number arithmetic. Constants enter as (c, 0). The first slot computes f(x) exactly as before; the second slot, by induction over every elementary step, accumulates df/dx. When the program finishes, the second slot of the output is f'(x), correct to machine precision. No step size to tune, no cancellation, no V-shaped error curve — the trade-off that haunted guide 1 simply does not arise.
Differentiate f(x) = x * sin(x) at x = 2, forward mode (dual numbers)
seed: x = (2, 1) # value 2, derivative 1
s = sin(x)
= (sin 2, cos(2) * 1) # chain rule in slot 2
f = x * s
= (2*sin 2, 1*sin 2 + 2*cos 2)
\______/ \_______________/
f(2) f'(2) = sin 2 + 2 cos 2 (exact)Reverse mode: one sweep, all the derivatives
Forward mode is perfect when there is one input and you want its effect on everything. But the headline applications — fitting a model with thousands of parameters, training a neural network, computing a gradient for Newton's method on a system — flip the shape: many inputs, one scalar output (a loss, an energy). Running forward mode would mean one sweep per input, which is hopeless when inputs number in the millions. The fix is reverse mode, and it is the same algorithm that machine learning calls backpropagation.
Reverse mode works in two passes. The forward pass runs the program normally, recording the graph of elementary operations and storing every intermediate value (this record is the 'tape' or computational graph). The backward pass then walks that graph in reverse, propagating sensitivities — the derivative of the final output with respect to each intermediate, called the adjoint — from the output back to the inputs, multiplying by each local derivative as it goes. The magic of the accounting: the gradient with respect to all inputs, however many there are, falls out of a single backward sweep. The whole gradient costs only a small constant multiple of the cost of evaluating f once — independent of the number of inputs. That single fact is what makes training models with billions of parameters feasible.
Honest limits: what AD does and does not promise
AD is exact, but be precise about what is exact. AD differentiates the exact function your code computes, not the mathematical function you meant to write. Two consequences follow. First, AD is still floating-point: it does not divide by a tiny h and so escapes the round-off floor of guide 1, but each operation still rounds, so a derivative through an ill-conditioned computation can still lose digits — AD gives the exact chain-rule answer to the rounded intermediate values. Second, AD differentiates whatever your control flow actually executed. At a point where f has a kink (think abs(x) at 0, or a branch in an if-statement), AD returns the derivative of the branch it happened to take, which may be a one-sided value where the true derivative does not exist. The honest summary: AD removes truncation error, not conditioning.
There is also a cost to weigh against finite differences. AD is not free: forward mode roughly doubles the work and storage of evaluating f; reverse mode adds the tape and a backward pass. A single finite-difference derivative is one extra function evaluation and two lines of code — sometimes that is genuinely good enough, and reaching for AD machinery would be over-engineering. The judgment is the same one this whole subject keeps teaching: match the tool to the cost and accuracy you actually need. When you need many accurate derivatives — a full gradient, a Jacobian for Newton on a system, sensitivities for optimization — AD is usually the right answer; for a one-off slope estimate, a tuned central difference may be simpler.
Why this closes the rung
Look back at the arc of this rung. It opened with finite differences and the sobering lesson that shrinking h hits a round-off wall; Richardson extrapolation then beat that wall partly, by cleverly cancelling the leading error term. The quadrature guides told the integration half of the same story — Newton-Cotes, then Gaussian quadrature wringing maximal accuracy from each function value, then adaptive refinement, all fighting truncation error within the finite-precision world. Automatic differentiation closes the rung by stepping outside the fight entirely on the differentiation side: it does not balance truncation against round-off, it deletes the truncation term. The exact-and-cheap derivative it returns is the cleanest answer the rung had been circling toward.
And it is a bridge forward. The very next rungs lean on derivatives constantly: Newton's method and its multivariate cousin need a Jacobian; optimization needs gradients and Hessians; sensitivity analysis and inverse problems need to differentiate whole simulations. In every one of those, AD — especially reverse mode — is the quiet engine that supplies exact derivatives at the cost of a few function evaluations, with no step size to agonize over. You now hold all three differentiation tools and, just as importantly, you know honestly when each one earns its place.