a stationary iteration

The simplest way to solve A x = b by stepping toward the answer is to rearrange the equation into the form x = (something easy) times x plus (something easy) times b, and then just keep substituting your current guess into the right-hand side. Because the recipe never changes from step to step — the same matrix and the same shift are used every time — it is called stationary, as opposed to methods (like conjugate gradient) whose coefficients adapt as they go. It is exactly a fixed-point iteration: a true solution is a point that the map leaves unchanged.

Concretely, write the iteration as x_{k+1} = G x_k + c, where G is a fixed matrix called the iteration matrix and c is a fixed vector. The error e_k = x_k - x_star (the gap between your guess and the true solution x_star) obeys the beautifully simple rule e_{k+1} = G e_k, so after k steps e_k = G^k e_0. The whole question of whether you converge — and how fast — collapses to the behaviour of powers of G: if applying G repeatedly shrinks every vector, the error dies out; if it can grow even one direction, you diverge. The precise condition is that the spectral radius of G (the largest size among its eigenvalues) must be below 1.

Stationary iterations — the Jacobi, Gauss-Seidel, and SOR methods are the classic three — are easy to code, need almost no memory, and were the workhorses of early scientific computing. Today they are rarely used alone to drive a solve to full accuracy because their convergence is often slow (the error shrinks only by a constant factor per step, and that factor can be agonizingly close to 1). Instead they live on as smoothers inside multigrid and as preconditioners for Krylov methods, where their cheap, local error-reduction is exactly what is wanted.