the iteration residual

When you have a candidate solution x to A x = b, you want to ask 'how good is it?' The honest answer would be to compare x with the true solution x_star and measure the error e = x - x_star. But you cannot do that — if you knew x_star you would not be iterating. The residual is the next best thing, and it is something you CAN compute: plug your x back into the equation and see by how much it misses. The residual is r = b - A x, the leftover, the amount by which the equation fails to balance.

Error and residual are tied together by the matrix A. Since A x_star = b exactly, subtracting gives r = b - A x = A x_star - A x = -A e, that is, r = -A e, or e = -A^{-1} r. So a small residual means a small error ONLY when A^{-1} is not too large — precisely, the error can be as much as the condition number of A times the (relative) residual. For a well-conditioned A the residual is a faithful proxy for the error; for an ill-conditioned A a tiny residual can hide a large error, because dividing by a near-singular A magnifies things. The residual is what every iterative method actually drives toward zero, since it is the only computable measure of progress.

Practically, the residual does double duty. It is the engine of the iteration — methods like conjugate gradient and GMRES build their next step out of the current residual and its images under A — and it is the basis of the stopping test: quit when the residual norm ||b - A x|| has shrunk enough, usually relative to ||b||. Just remember the residual's blind spot: it certifies that x nearly satisfies the equations (small backward error), not necessarily that x is close to the true answer (small forward error). On an ill-conditioned problem those two are very different things.