round-off versus truncation error

Every numerical answer is wrong, and it is wrong for two completely different reasons that you must not confuse. One comes from the METHOD: replacing a smooth curve with straight steps, replacing a limit with a finite formula, throwing away Taylor terms. The other comes from the MACHINE: a computer cannot store most numbers exactly, so every single arithmetic operation rounds the result to a fixed number of digits. The first is truncation error; the second is round-off error; and they pull in opposite directions as you refine.

Precisely, truncation error is the discretization error — the gap between the true solution and what the exact method formula would give in perfect arithmetic. It SHRINKS as you make the step h smaller, like h^p for a method of order p. Round-off error is the accumulated effect of finite floating-point precision (about 16 significant digits in standard double precision); each step adds a tiny rounding, and crucially this contribution GROWS as h shrinks, because a smaller step means more steps and more roundings to accumulate, and because subtracting nearly-equal numbers loses precision. The total error is roughly truncation (falling like h^p) plus round-off (rising as h falls).

This trade-off sets a hard, honest floor on how accurate a numerical solution can be. As you shrink h, the total error falls — dominated by truncation — until you reach a sweet-spot step size; below that, round-off takes over and the total error rises again. Pushing h smaller past that point makes your answer WORSE, not better, while costing far more computation. This is why no numerical solution is exact and why 'just use a tinier step' is not a cure-all. A good practitioner knows both errors are present, knows roughly where the sweet spot lies, and never trusts more digits than the method and the machine can honestly support.