numerical differentiation

Calculus gives you the derivative f'(x) as the limit of a slope — rise over run — as the run shrinks to zero. But on a computer you cannot take a true limit; you can only pick a small, finite run and measure the slope it gives. Numerical differentiation is the craft of estimating a derivative from a handful of function values, by mimicking that slope without ever reaching the limit.

The simplest recipe is the forward difference: pick a small step h and compute (f(x + h) - f(x)) / h, the slope of the line joining two nearby points on the graph. As h shrinks this approaches f'(x). You can also look backward, (f(x) - f(x - h)) / h, or — better — straddle x with the central difference (f(x + h) - f(x - h)) / (2h). The same idea extends to second derivatives: f''(x) is approximated by (f(x + h) - 2 f(x) + f(x - h)) / h^2, the discrete curvature. These are the building blocks (called finite-difference formulas) used everywhere a derivative is needed but only sampled values of f are at hand.

Numerical differentiation matters because in real problems you often have data or a black-box function, not a clean formula to differentiate by hand: think of estimating velocity from logged positions, or a gradient inside an optimizer. Its honest danger is subtle and worth burning into memory: making h smaller does NOT keep improving the answer. Past a sweet spot, subtracting two nearly equal numbers loses significant digits to round-off, and dividing by a tiny h amplifies that noise — so there is an optimal step size, below which accuracy gets WORSE. Where exact derivatives are available, automatic differentiation sidesteps this trap entirely.