automatic differentiation
There are three ways to get a derivative on a computer, and they are easy to confuse. Symbolic differentiation manipulates formulas (and blows up into giant expressions). Finite differences poke the function with a small step h (and suffer the step-size trade-off, never quite exact). Automatic differentiation is the third, modern way: it computes the derivative EXACTLY — to full machine precision — by applying the chain rule mechanically to the actual sequence of operations your program performs, with no step size and no formula manipulation.
The key insight is that any function a program computes, however complicated, is built from a graph of elementary operations (add, multiply, sin, exp, ...) whose individual derivatives are known. Automatic differentiation propagates derivative information through that computation graph by the chain rule. FORWARD mode carries, alongside each value, its derivative with respect to one input (operationally, run the program on dual numbers a + b*epsilon where epsilon^2 = 0, and the epsilon part emerges as the derivative); it is efficient when there are few inputs and many outputs. REVERSE mode runs the program forward recording the graph, then sweeps backward accumulating the derivative of one output with respect to every input in a single pass; it is efficient when there are many inputs and one output. That reverse mode is precisely backpropagation — the engine that trains neural networks, where you need the gradient of one scalar loss with respect to millions of parameters.
Automatic differentiation is the modern, exact alternative to finite differences and the reason gradient-based optimization and deep learning scale: a single reverse pass gives the entire gradient at a cost comparable to one function evaluation, no matter how many parameters. The honest fine print: it differentiates exactly what the code COMPUTES, including any bugs or approximations in that code; it needs care at non-differentiable points (the kink of abs(x) or a relu at 0, branches, loops with data-dependent length); reverse mode must store intermediate values, so memory can be the bottleneck on deep computations; and it gives derivatives, not the function's integral — it is the differentiation counterpart to the quadrature half of this field.
Differentiate f(x) = x * sin(x) at x = 2 by forward-mode dual numbers. Carry x as (2, 1) meaning value 2, derivative 1. Then sin gives (sin 2, cos 2 * 1) = (0.909, -0.416), and the product rule on (2,1)*(0.909,-0.416) yields value 1.819 and derivative 2*(-0.416) + 1*0.909 = -0.926 — the exact f'(2), with no step size at all.
The chain rule run through the computation graph: exact, no step h.
Automatic differentiation is NOT finite differences and NOT symbolic algebra — it has no step-size error and no expression swell. But it differentiates the code as written: at a non-differentiable point (abs, relu at 0, an if-branch) it returns one subgradient, not a warning, and reverse mode's memory for stored intermediates can dominate on deep graphs.