From the definition of the derivative to a recipe you can compute
We open the integration-and-differentiation rung where calculus opened its: with the limit f'(x) = lim of (f(x + h) - f(x)) / h as h goes to zero. On paper you take that limit and get an exact formula. But suppose f is something you can only evaluate — a value spat out by a simulation, a black-box function with no symbolic form, or experimental data. Then there is no limit to take; all you can do is pick a small but nonzero h and compute the quotient. Dropping the limit and keeping a finite h turns the definition into a finite-difference formula, the workhorse of numerical differentiation. The forward difference (f(x + h) - f(x)) / h is the simplest one: it asks for just two function values and approximates the slope of the tangent by the slope of a nearby secant line.
How wrong is that secant slope? Taylor's theorem, which underwrote so much of the calculus ladder, answers exactly. Expand f(x + h) = f(x) + h*f'(x) + (h^2 / 2)*f''(x) + ... and solve for f'(x): the forward difference equals f'(x) + (h/2)*f''(x) + ... So the error it makes — the truncation error, the price of truncating that infinite Taylor series — is about (h/2)*f''(x), which is O(h). Halve h and you halve the error. This is a first-order method: the leading error term is proportional to h to the first power.
A symmetric trick buys an extra order — for free
We can do better at the same cost. Instead of stepping only forward, step both ways and use the central difference (f(x + h) - f(x - h)) / (2h). Write out both Taylor expansions: f(x + h) and f(x - h) share the same even-power terms but their odd-power terms (the h*f' and (h^3/6)*f''' pieces) flip sign. Subtracting them cancels the f(x) terms and also cancels the h^2*f'' term that ruined the forward difference. The leading error left over is about -(h^2 / 6)*f'''(x), which is O(h^2). That is the payoff: a second-order formula. Now halving h cuts the error not by half but by four.
This same Taylor-subtraction game generates a whole zoo of stencils: combine values at x, x ± h, x ± 2h with cleverly chosen weights and you can cancel ever-higher error terms to reach fourth-order accuracy or beyond, or estimate second derivatives via (f(x + h) - 2*f(x) + f(x - h)) / h^2. Those weighted patterns of sample points are exactly the finite-difference stencils that the differential-equations rungs lean on to turn a derivative into arithmetic on a grid.
The plot twist: smaller h is not always better
The truncation-error story says: shrink h and the error vanishes, so make h as tiny as you possibly can. That advice is a trap, and seeing why is the heart of this guide. The trouble is that we are not doing real arithmetic — we are doing floating-point arithmetic, where each f(x) carries a tiny relative rounding error of order the unit roundoff u (about 1e-16 in double precision). When h is small, f(x + h) and f(x) are nearly equal numbers, and subtracting two nearly equal numbers is the textbook recipe for catastrophic cancellation: the leading digits agree and annihilate, leaving a difference dominated by the rounding noise in the original values.
Quantify it. Each evaluation of f is good to an absolute error of about u*|f(x)|. The difference in the numerator carries that same absolute noise, but we then divide by h, so the round-off error in the derivative estimate grows like u*|f(x)| / h. Notice the direction: as h shrinks, this term grows. Meanwhile the truncation error shrinks like h^2 (central) or h (forward). So the total error is a sum of two terms pulling in opposite directions — one falling with h, one rising as h falls. That tug-of-war is the famous step-size trade-off.
total error(h) ~ C_trunc * h^2 + u * |f| / h
\__________/ \__________/
truncation (falls) round-off (rises)
large h: truncation dominates -> decrease h
small h: round-off dominates -> increase h
best h: the two are balanced (a V-shaped error curve)Finding the sweet spot
Because total error is (roughly) C*h^2 + u*M/h for the central difference, with M of order |f|, we can find the minimum the way calculus taught us: differentiate with respect to h, set it to zero, and solve. The result is h_opt proportional to the cube root of u — about u^(1/3), roughly 1e-5 in double precision — and at that h the smallest achievable error is itself of order u^(2/3), roughly 1e-11. Read that twice: with double precision good to about 16 digits, the best a central difference can ever do is around 11 correct digits, and a first-order forward difference does worse still, optimal at h ~ u^(1/2) ~ 1e-8 with an error floor of about u^(1/2) ~ 1e-8. You can never recover full machine precision from a plain finite difference, no matter how you tune h.
- Write the total error model: truncation (C * h^p, with p = 1 forward, p = 2 central) plus round-off (u * |f| / h).
- Differentiate that sum with respect to h and set the derivative to zero to balance the two competing terms.
- Solve for the optimal step: h_opt ~ u^(1/3) for the central difference, ~ u^(1/2) for the forward difference.
- Substitute back to read the error floor: about u^(2/3) (~1e-11) for central, about u^(1/2) (~1e-8) for forward — the best accuracy that formula can ever reach.
Where this leaves us — and where the rung goes next
Step back and the lesson generalizes far beyond derivatives. Accuracy in scientific computing is almost never bottlenecked by one error alone; it is a balance between a discretization error you control by refining (here, h) and a floating-point error that refining cannot beat and eventually amplifies. The same V-shaped curve reappears whenever a method subtracts nearby quantities and divides by a small number. Numerical differentiation is the cleanest place to first meet this pattern, because both halves are so easy to write down. And it carries a humbling honesty: every value here is an approximation, the round-off floor is a wall set by the hardware, and no amount of cleverness in choosing h pretends otherwise.
So can we beat the floor? Two roads open from here, and the rung walks both. The next guide, on Richardson extrapolation, stays with finite differences but plays a smarter game: compute the central difference at h and at h/2, then combine the two so the leading h^2 error term cancels exactly, jumping you to O(h^4) accuracy without ever needing a dangerously tiny h. The other road, the rung's finale, abandons differences entirely: automatic differentiation propagates derivative information through the very code that computes f, delivering a derivative correct to full machine precision with no h, no truncation error, and no cancellation at all — the trade-off of this guide simply does not exist there. Both are answers to the wall we just hit.