Foundations: Algorithms, Approximation & Error

the rate of convergence

The rate of convergence is how quickly an iterative process approaches its limit — the speed at which the error shrinks. The phrase is used in two closely related ways, and it pays to keep them straight. In the broad sense it just means 'how fast', covering both the order (does the error square or merely scale each step) and, for a given order, the constant factor that multiplies the error. In the narrow, technical sense often meant by 'rate', it is the asymptotic constant in linear convergence: the factor C with e_{n+1} approximately C times e_n, where smaller C means faster.

For linearly convergent methods the rate is everything, because the order is fixed at 1 and only the contraction factor distinguishes a quick method from a crawling one. If the error is multiplied by C each step, then after k steps it is about C^k times the start, and you gain roughly -log10(C) decimal digits per iteration: C = 0.5 gains about 0.30 digits per step, C = 0.1 gains a full digit per step, while C = 0.99 crawls, needing about 230 steps to gain one digit. Many fundamental methods are linear and live or die by this constant — the Jacobi and Gauss-Seidel iterations converge at a rate set by the spectral radius rho of their iteration matrix, and steepest descent slows as the condition number grows.

Rate of convergence matters because it converts directly into iteration count and therefore cost. It is also the lever that preconditioning pulls: a good preconditioner does not change a method's order but shrinks its effective rate constant (or, for conjugate gradient, the condition number that controls it), turning hundreds of iterations into a handful. As with order, the rate is an asymptotic, local description — it tells you the eventual speed once the iteration settles into its regime, not necessarily the behaviour in the first few erratic steps.

A linearly convergent iteration with rate C = 0.1 cuts the error tenfold per step, gaining one decimal digit each iteration — so reaching 1e-8 from an order-1 error takes about 8 steps; with C = 0.9 the same target needs roughly 175 steps.

For linear convergence, the contraction factor C is the whole story — smaller C, fewer iterations.

Do not conflate rate with order: a quadratic method (order 2) is far faster than any linear one regardless of the linear rate. Within linear convergence, though, the rate constant C — set by the spectral radius for stationary iterations — decides everything.

Also called
convergence rateasymptotic ratecontraction factor收斂率收斂速度