From 'is it sensitive?' to 'how sensitive, exactly?'
In the previous guide we asked a yes-or-no question: does a tiny wiggle in the input cause a tiny wiggle in the output, or a huge one? That is the idea of problem conditioning, and it lives in the problem itself, not in any algorithm you might use. But 'a little' and 'a lot' are not measurements. The whole job of this guide is to replace that adjective with a single honest number — the condition number — so we can say precisely how many times the input error gets magnified on its way to the output.
Picture the problem as a black box that takes an input x and returns an output y = f(x). You never hand it the exact x; you hand it x plus a small relative error (round-off in storing it, noise in measuring it). The condition number is simply the answer to one question: by what factor does that relative input error reappear, magnified, in the output? If a 0.001% wobble in x produces a 0.001% wobble in y, the condition number is about 1 and life is easy. If that same 0.001% wobble produces a 100% wobble in y, the condition number is about 100000, and you are in trouble.
Relative in, relative out: the working definition
Because errors are best judged on a scale-free basis — recall the relative error from the foundations rung — the version we actually use is the relative condition number. In words: it is the relative change in the output divided by the relative change in the input, taken in the worst case as the input wobble shrinks. A condition number of kappa means a relative input error of size eps can blow up into a relative output error as large as kappa times eps. That single multiplication is the entire mechanism of error amplification.
|relative change in output|
kappa = max -------------------------- (as the input wobble -> 0)
|relative change in input |
relative output error <= kappa x relative input error
for a smooth f(x): kappa(x) = |x f'(x)| / |f(x)|The little formula kappa(x) = |x f'(x) / f(x)| repays a moment's stare. The slope f'(x) says how fast the output moves; the factors of x and 1/f(x) rescale that slope into relative terms. Try f(x) = sqrt(x): then f'(x) = 1/(2 sqrt(x)), and kappa works out to exactly 1/2 everywhere — square roots are beautifully well conditioned, halving any relative error. Now try f(x) = x - 1 near x = 1: there f(x) is tiny while x f'(x) is about 1, so kappa explodes. That is subtraction of nearly equal numbers seen from the conditioning side: the problem 'compute x - 1 for x near 1' is ill-conditioned no matter how you code it.
The accounting: condition number eats your digits
Now the part that turns the condition number from an abstraction into a number you feel. Double precision gives you about 16 correct significant decimal digits to start with — a fixed budget, set the moment your inputs are stored, since storing x already costs a relative error near machine epsilon, about 1e-16. Multiply that unavoidable starting error by the condition number and you get the best relative error any method could possibly deliver. Take the base-10 logarithm and the picture becomes a clean piece of arithmetic: a condition number near 10^k costs you about k digits.
So a well-conditioned problem with kappa around 1 keeps nearly all 16 digits. A problem with kappa around 1e8 — large but utterly ordinary in real applications — multiplies your 1e-16 starting error up to about 1e-8, leaving roughly 8 trustworthy digits. Push to kappa around 1e16 and the math is grim: 16 minus 16 is 0, so even the best possible computation may return an answer with no correct digits at all. This is not a coding bug you can hunt down; it is the question itself refusing to be answered accurately in double precision.
The matrix condition number for A x = b
The same idea scales up to the workhorse problem of the next rung: solving a linear system A x = b. Here the input is the data (A and b) and the output is the solution x. The matrix condition number of A, written kappa(A), answers exactly the old question: if I perturb b (or A) by a small relative amount, how much can the relative error in x grow? Using the 2-norm it equals the ratio of the largest to the smallest singular value, kappa(A) = sigma_max / sigma_min — a number that is always at least 1 and that you can read off A's geometry.
Geometrically, kappa(A) measures how badly A stretches space unevenly: it is the ratio of the longest to the shortest axis of the ellipse you get by applying A to a circle. When that ratio is huge, the matrix is nearly singular — its rows or columns are almost linearly dependent — and the solution x sits on a knife edge where a hair's-width change in b slides it far away. A classic offender is the Vandermonde matrix from fitting a high-degree polynomial: its condition number grows exponentially with the degree, which is precisely why naive high-degree fits go haywire.
A near-universal beginner reflex is to 'check' a system by computing the inverse of A. Resist it. Forming an inverse is more expensive, less accurate, and tells you nothing the condition number does not — you almost never invert a matrix to solve A x = b in the first place. The condition number is the honest health report you want; the inverse is an expensive detour that can make matters worse.
Why a perfect algorithm still cannot save you
Here is the sentence that ties this rung together, stated now and unpacked over the next three guides. The accuracy you actually get obeys a master rule: forward error <= condition number x backward error. The forward error is how wrong your answer is; the backward error is, loosely, how much you would have to fiddle the input to make your computed answer the exact answer of that fiddled problem. A great algorithm keeps the backward error down near round-off — that is the gold standard called backward stability, the topic of guide 4.
But stare at the master rule and the cruelty is obvious. Even if your algorithm is flawless and drives the backward error all the way down to machine epsilon, the forward error is still multiplied by the condition number. If kappa is 1e12, that tiny 1e-16 backward error becomes a 1e-4 forward error — four trustworthy digits, no matter how perfect your code. A stable algorithm on an ill-conditioned problem gives you the exact answer to a nearby question, which can be far from the exact answer to the question you actually asked. That is not the algorithm's failure; it is the problem's.
This is the whole identity of the rung in one breath: accuracy = conditioning x stability. The condition number, this guide's star, is the conditioning half — fixed by the problem and beyond any algorithm's reach. Guide 3 sharpens the difference between forward and backward error; guide 4 explains why backward stability is the most you can honestly ask of a method; guide 5 multiplies the two halves back together. The rule-of-thumb error bound you now hold — believe about 16 minus log10(kappa) digits — is the practical shadow of that grand identity.