catastrophic cancellation

Suppose you measure two lengths as 1.00000 m and 0.99998 m, each good to five decimal places, and you want their difference. The answer is 0.00002 m — but now your five reliable digits have collapsed to a single one, because all the leading digits cancelled and only the uncertain tail survived. Subtracting two nearly-equal numbers does not create error, but it brutally EXPOSES the error that was already lurking in their last digits. That sudden loss of significant digits is catastrophic cancellation.

In floating point the mechanism is sharp. The two inputs were each already rounded, carrying a relative error of about u. When you subtract numbers that agree in their leading bits, the result is small, but its ABSOLUTE error is unchanged — so the RELATIVE error explodes by the ratio of the inputs' size to the tiny difference. The subtraction step itself is exact (by a result called Sterbenz's lemma, the difference of two close floats is computed exactly); the damage was baked in earlier and the subtraction merely magnifies it. The fix is almost never 'use more precision' but 'rearrange the formula so the dangerous subtraction never happens'.

The textbook cure is the stable quadratic formula. The standard root x = (-b + sqrt(b^2 - 4ac)) / (2a) suffers cancellation when b > 0 and 4ac is small, because -b and +sqrt(b^2-4ac) nearly cancel. Instead compute the 'safe' root q = -(b + sign(b)*sqrt(b^2-4ac))/2 first, then get the other root from x1*x2 = c/a, i.e. the second root is c/(a*q). No near-equal subtraction, no lost digits. The same trick — multiply by a conjugate, use a Taylor expansion, or reorder terms — rescues expressions like (1 - cos x)/x^2 near 0 and 1 - sqrt(1+x) for small x.