backward error

Here is a clever change of question that powers reliable computing. Instead of asking 'how wrong is my answer?' (forward error), backward error asks: 'for what slightly different problem is my answer exactly right?' You computed some y-hat. It is not the exact answer to YOUR problem, but maybe it is the exact answer to a nearby problem with slightly perturbed input. The backward error is the size of the smallest such perturbation. A tiny backward error is a badge of honour: it says your algorithm essentially solved the right problem, just with data nudged within the noise.

Precisely: you computed y-hat as your approximation to y = f(x). The backward error is the smallest ||delta-x|| (often measured relatively, ||delta-x|| / ||x||) such that f(x + delta-x) = y-hat exactly. In words, you push the answer back through the problem to find which input would have produced it cleanly, then measure how far that input is from the real one. The beauty is that you can often compute or bound the backward error without ever knowing the true answer y — which is exactly why it became the workhorse of numerical analysis under Wilkinson.

Why this reframing wins: input data are already inexact (measurement noise, rounding into floating-point). If your algorithm's backward error is no larger than the uncertainty already in the data, then your computed answer is as good as the exact answer — you cannot tell your perturbation apart from the noise you started with. That is the whole goal of a backward-stable algorithm. The forward error then follows from the master rule: forward error is at most the condition number times the backward error.