the step-size trade-off

Intuition says that to measure a slope more accurately you should shrink the run — bring the two points closer and closer. On a computer this intuition is a trap. Shrink the step too far and the answer does not keep improving; it starts getting WORSE. The step-size trade-off is the tug-of-war between two opposing errors that decides the best step h to use.

Two errors pull in opposite directions. TRUNCATION error comes from using a finite step instead of a true limit; for a central difference it is about C1 * h^2, so it SHRINKS as h shrinks. ROUND-OFF error comes from the floating-point subtraction f(x + h) - f(x - h): when h is tiny those two numbers are nearly equal, so their difference loses most of its significant digits (catastrophic cancellation), and dividing the noisy result by a tiny h magnifies it — this part is about C2 * epsilon / h, where epsilon is machine precision, so it GROWS as h shrinks. The total error is roughly C1 h^2 + C2 epsilon / h, a U-shaped curve. Minimizing it gives an optimal h around epsilon^(1/3) — for double precision (epsilon about 10^-16) that is near 10^-5, where you can hope for roughly 10 to 11 correct digits, not the full 16.

This is one of the most important honesty lessons in numerical computing: the naive limit-taking of calculus cannot be reproduced on a finite machine. The lesson generalizes — a forward difference has truncation about C1 h and round-off about C2 epsilon / h, giving optimal h around sqrt(epsilon), near 10^-8. The practical escapes are to choose h near the theoretical optimum, to use Richardson extrapolation to kill the truncation term so you can use a larger, safer h, or — best of all when you can — to use automatic differentiation, which computes derivatives exactly and has NO step size at all.