Conditioning: Is the Problem Itself Sensitive?

Two questions that look the same but are not

In the floating-point rung you learned that every numerical answer is an approximation, and that the very act of storing a number already rounds it — most decimals, even 0.1, have no exact binary form. So your inputs arrive slightly wrong before any computation runs. The natural worry is: will my algorithm turn those small input errors into a large wrong answer? But there is a sneakier, more fundamental question hiding underneath it, and this whole rung is about telling the two apart.

Question one is about your method: *did the algorithm add error of its own?* That is stability, and we will spend later guides on it. Question two is about the problem itself, before any algorithm is chosen: *if the true answer is computed perfectly but the input wobbles a little, how much does the true answer move?* That is conditioning. The key honest claim of this rung is that these are genuinely different. A perfect algorithm cannot rescue a problem whose true answer is already hypersensitive to its inputs.

Crucially, conditioning is a property of the problem — the mathematical map from input to output — and not of any code. Think of the problem as a black box: feed in data x, out comes the exact answer f(x). Conditioning asks only how f behaves when x is nudged. The black box does not know or care which language, library, or method you will eventually use. So you can, and should, judge conditioning before writing a single line of an algorithm.

A picture: the gentle slope and the knife-edge ridge

Picture the input as a spot on a map and the output as your altitude. A well-conditioned problem is a gentle meadow: step a few metres in any direction and your altitude barely changes — small input error, small output error. An ill-conditioned problem is a knife-edge ridge: the same small sideways step drops you off a cliff. Same size of footstep; wildly different consequence. The difference lives entirely in the shape of the terrain, which is the problem, not in how carefully you walk, which is the algorithm.

Here is a tiny worked example you can check by hand. Evaluating sin(x) near x = 1 is well-conditioned: nudge x by 0.1% and the answer barely moves, because sin is gently sloped there. Now contrast subtraction of two nearly equal numbers, say a - b with a = 1.00001 and b = 1.00000. The true difference is 0.00001. If a is uncertain in its sixth digit — a relative error of about one part in 100000, utterly typical — that uncertainty lands almost entirely on the answer, which only had five significant digits to begin with. A wobble that was invisible in the inputs becomes the whole answer. That blow-up is a hallmark of an ill-conditioned operation, and it is the seed of catastrophic cancellation.

Putting a number on the sensitivity

We do not want to wave our hands and say 'pretty sensitive'. We want a number. The condition number of a problem is exactly the gain knob: feed a relative wobble of size epsilon into the input, and the output wobbles by roughly (condition number) times epsilon. A condition number of 1 means the problem passes errors through untouched — the gentle meadow. A condition number of 10^6 means it magnifies them a millionfold — the knife-edge. The next guide is devoted to this number and how it translates directly into 'digits lost', so here we only meet it.

For a smooth scalar function f, the relative condition number has a clean closed form you can compute with a single derivative. The two example sketches below make the abstract idea concrete: square root is tame everywhere, while the exponential gets touchier the larger its argument.

relative condition number of  y = f(x):   kappa = | x * f'(x) / f(x) |

f(x) = sqrt(x):   f'(x) = 1/(2 sqrt(x))
                  kappa = | x * (1/(2 sqrt(x))) / sqrt(x) | = 1/2        (well-conditioned, always)

f(x) = e^x:       f'(x) = e^x
                  kappa = | x * e^x / e^x | = |x|                       (fine near 0, touchy for large x)

Notice that the condition number can depend on where you ask the question. The exponential is benign near 0 and increasingly sensitive far out. So 'is this problem ill-conditioned?' is often really 'is it ill-conditioned at this input?'. Conditioning is local: it describes how the terrain slopes right where you are standing.

When the problem is a whole linear system

Most real computational problems are not one scalar function but something bigger, like solving the linear system A x = b — find the x that the matrix A maps to b. We still want one number that says how much error in b (or in A) gets amplified into error in x. That number is the matrix condition number, written kappa(A). It is the linear-system version of the gain knob, and you will meet it again whenever you solve, fit, or invert.

The geometric picture is the cleanest entry point. Imagine A taking the unit sphere of inputs and stretching it into an ellipsoid. If the ellipsoid is nicely round, every direction is treated about equally and kappa(A) is close to 1 — well-conditioned. If A squashes the sphere almost flat in some direction, that direction is nearly destroyed and can barely be recovered when you solve back for x; kappa(A) is huge and the system is ill-conditioned. Formally kappa(A) is the ratio of the largest stretch to the smallest stretch (the largest singular value over the smallest). It is always at least 1: no matrix is better than perfectly conditioned.

A famous trap shows how innocent an ill-conditioned matrix can look. The Hilbert matrix, with entries 1/(i+j-1), is small and tidy, yet its 10-by-10 version has kappa(A) about 1.6 x 10^13. Solving a Hilbert system in double precision is essentially hopeless past size ~12 — not because any solver is broken, but because the smallest singular value has all but vanished. By the way: this is why you almost never form A^{-1} explicitly to solve A x = b. The inverse is both more expensive and less faithful than a direct factor-and-solve; the conditioning issue is the problem's, but inverting needlessly invites extra error on top of it.

Why this changes how you debug

Here is the payoff that makes conditioning practical rather than academic. When an answer looks wrong, the very first diagnostic is not 'where is the bug in my loop'. It is: *is the problem itself ill-conditioned?* If yes, then a wrong-looking answer may be the honest verdict of the problem, and switching algorithms, adding checks, or polishing code will not save you. The cure for ill-conditioning is never a better algorithm — it is reformulating the problem (avoiding a near-cancellation, choosing a better-posed quantity) or carrying more precision.

1. Identify the problem as a map from input to output, f(x), separate from whatever code you will run.
2. Estimate its condition number — by the derivative formula for a scalar f, or kappa(A) for a linear system (libraries report it, e.g. LAPACK's RCOND).
3. Read off the ceiling: a condition number near 10^k costs roughly k of your ~16 double-precision digits, no matter how perfect the algorithm.
4. Only after that ceiling is known do you ask the separate question — is my algorithm stable enough to reach it? That is the subject of the rest of this rung.

This separation is the engine of the whole rung. It is captured by a master rule you will meet in full later: the forward error (how wrong your answer is) is at most the condition number (how hard the problem is) times the backward error (how good your algorithm is). Conditioning is the first factor — the one you can read off the problem alone, before any code exists. The clean slogan, which guide 5 unpacks, is accuracy = conditioning x stability: a stable algorithm still fails on an ill-conditioned problem, and a well-conditioned problem can still be wrecked by an unstable one. You need both.

A caution and a connection

Two honest caveats keep conditioning from being oversold. First, the condition number is a first-order, linearized measure — it describes the slope of the terrain at your input, in the limit of tiny perturbations. For large perturbations the real behaviour can differ, so treat it as an excellent rule of thumb, not an exact guarantee. Second, the relative condition number is undefined where the true output is zero (you cannot divide by an answer of size zero); near such points only an absolute error notion makes sense.

Conditioning is also the numerical cousin of well-posedness. A well-posed problem has a solution that exists, is unique, and depends continuously on the data. Conditioning sharpens that last clause into a number: it says how continuously. A well-posed but severely ill-conditioned problem is technically continuous yet practically a knife-edge — the dependence is so steep that finite-precision arithmetic cannot keep up. Many infamous cases hide in plain sight: differencing nearly equal numbers, the Hilbert matrix above, fitting high-degree polynomials through many points (the Vandermonde conditioning problem), and recovering polynomial roots from coefficients.