fixed-point iteration

Press the cosine button on a calculator over and over, starting from any number (in radians): 1, then cos(1) = 0.540, then cos(0.540) = 0.858, and so on. The numbers hop around but settle on 0.739085..., and pressing cos again leaves it unchanged. You have found a FIXED POINT — a value the function maps to itself. Fixed-point iteration turns this 'keep applying the same rule' game into a general method for solving equations.

The idea is to rewrite the equation you want, say f(x) = 0, in the form x = g(x), so a solution is a point left fixed by g. Then iterate: pick a guess x_0 and compute x_{n+1} = g(x_n), x_1 = g(x_0), x_2 = g(x_1), and so on. If the sequence converges, its limit x* satisfies x* = g(x*), the fixed point you wanted. There are many ways to rearrange f(x) = 0 into x = g(x); some converge and some do not, and the difference is governed by how steeply g rises near x* (see the contraction-mapping condition). Newton's method is itself a clever fixed-point iteration with g(x) = x - f(x)/f'(x).

Fixed-point iteration is one of the great unifying ideas in numerical analysis: it underlies stationary iterations for linear systems (Jacobi, Gauss-Seidel), time-stepping for differential equations, and many machine-learning updates. Its honest limitation is that the SAME equation can be cast as x = g(x) in good ways and bad ways: a careless rearrangement diverges, oscillates, or crawls, while a good one converges fast. The art is choosing g so that the iteration both converges and converges quickly.