JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Forward Error vs Backward Error

Two honest ways to ask 'how wrong is my answer?' — one measures the gap in the output, the other measures how much you would have to perturb the input to make your answer exactly right. Learning to think backward is the single mental shift that makes the rest of numerical analysis click.

Two questions about one wrong answer

You have a problem with a true answer y, and your computer hands you an approximation y_hat. The obvious question is: how far is y_hat from y? That gap is the forward error — the error you can see in the output. It is what you actually care about, and it is exactly what you usually cannot measure, because if you knew the true y you would not have run the algorithm in the first place. The previous guides built up condition numbers; now we meet the other half of the story.

So a numerical analyst asks a sneakier question instead: for what nearby problem is my computed answer y_hat the exact answer? Imagine nudging the input data by a little, just enough that y_hat becomes perfectly correct for the nudged problem. The size of that smallest nudge is the backward error. It flips the whole frame: instead of asking 'how wrong is my output?', it asks 'how wrong would my input have had to be for this output to be right?'

Why backward is the one you can actually compute

The magic of the backward question is that you can often answer it without ever knowing the true y. Take the linear system A x = b. Your solver returns x_hat. You cannot compare it to the exact x, but you can compute the residual r = b - A x_hat with arithmetic you already have. A small residual means x_hat exactly solves the slightly perturbed system A x = b - r, i.e. the same matrix with a slightly different right-hand side. The residual you can measure is, up to scaling, the backward error you wanted.

This is why backward error is the practical currency of the field. The forward error needs the unknown answer; the backward error often needs only the data and your computed result, both in hand. A solver with a tiny backward error is making a strong, checkable promise: 'I did not solve your problem, but I solved one indistinguishable from it at the level of your own data's uncertainty.' For data that came from a noisy sensor or a rounded constant, that is frequently all the truth there is to have.

A tiny worked example

Suppose the true problem is to compute sqrt(2), so y = 1.41421356... Your routine returns y_hat = 1.41421. The forward error is the plain output gap, about 3.6e-6. Now ask the backward question: of what number is 1.41421 the exact square root? It is the exact square root of 1.41421^2 = 1.99998... So your answer is the perfectly correct square root of 1.99998 instead of 2.0. The backward error — how far the input had to move — is about 1e-5. You computed a flawless answer to an almost-right question.

true input x = 2.0          exact output  y     = 1.41421356...
computed   y_hat = 1.41421

forward error  = |y_hat - y|          = 3.6e-6   (gap in the OUTPUT)
backward error = |x - y_hat^2|        = 1.0e-5   (gap in the INPUT)
               y_hat = sqrt(x + delta_x),  delta_x = -1.0e-5
The same approximation, measured forward (in the output) and backward (in the input).

Notice the two numbers differ — here 1e-5 versus 3.6e-6 — and the ratio between them is no accident. It is governed by how sensitive the square root is to its input near x = 2, which is exactly a condition number. That ratio is the bridge between the two worlds, and it is the heart of the master rule we assemble next.

The master rule that ties them together

Here is the single inequality this rung is built around, the rule-of-thumb error bound: forward error <= condition number x backward error (all in relative terms). The condition number belongs to the problem; the backward error is produced by the algorithm. Multiply the two and you bound the thing you really wanted — the forward error — using two quantities you can each get a handle on. The square-root example obeys it: a tiny backward error of 1e-5, amplified by the problem's sensitivity, yields the small forward error you saw.

Read the rule as a budget. Your algorithm controls only its half: keep the backward error down near the unit roundoff, a few times 1e-16 in double precision, and it has done everything an algorithm can do. The condition number is fixed by the problem you were handed — you do not get to negotiate it. If that number is 1e8, the rule says you may lose about 8 of your ~16 honest decimal digits no matter how flawless the code is. The bound is the reason 'my code is correct' and 'my answer is accurate' are two genuinely separate claims.

  1. Estimate the backward error of your computed answer — for A x = b, scale the residual: ||r|| / (||A|| ||x_hat||). This is the algorithm's report card.
  2. Estimate the problem's condition number — for A x = b that is the matrix condition number cond(A), how much A magnifies relative perturbations.
  3. Multiply them for a forward-error estimate: relative error in x is roughly cond(A) x (backward error). That is your honest expectation of accuracy.
  4. Interpret it: a small product means trust the digits; a large product warns you the answer may be poor even though the algorithm behaved perfectly.

Two failures that look identical and are not

Now the payoff of separating the two errors: it diagnoses why an answer is bad. Suppose your forward error is large. The rule offers exactly two suspects. Either the backward error is large — your algorithm is unstable, sloppy, amplifying rounding it should have absorbed — which is your fault and fixable by a better method. Or the backward error is tiny but the condition number is enormous — the problem itself is ill-conditioned, and no algorithm on Earth could have done better in this precision. Same symptom, opposite cures.

This is why chasing a 'more accurate' library on an ill-conditioned problem is wasted effort — you are blaming the algorithm for the problem's sin. The honest move is to compute the backward error and the condition number separately. If the backward error is already near roundoff, switching solvers buys you nothing; you must instead reformulate the problem, gather better-conditioned data, or use higher precision to enlarge the digit budget the condition number is eating. Diagnosis before treatment.

An algorithm that always achieves a tiny backward error — one whose every computed answer is the exact answer to a problem only a roundoff-sized nudge away from yours — earns the field's highest praise: it is backward stable. That, and why it is the most you can sensibly demand, is exactly where the next guide goes.