why iterate instead of factor

Suppose you must solve A x = b where A is a million-by-million matrix, but almost every entry is zero — say each row has only five nonzeros, as happens when you discretize a physical problem on a grid and each point talks only to its neighbours. A direct method like Gaussian elimination would, in principle, give the exact answer in finitely many steps. The trouble is that as elimination proceeds it creates new nonzeros where there used to be zeros — this is called fill-in — and the once-sparse matrix becomes dense. Storing and processing a dense million-by-million matrix needs roughly 10^12 numbers and 10^18 operations: hopeless. So instead of computing the answer, we sneak up on it.

An iterative method starts with a guess x_0 (often just zero) and produces a sequence x_1, x_2, x_3, ... that, if all goes well, marches toward the true solution. Each step costs only a few matrix-vector products A times a vector, and multiplying the sparse A by a vector is cheap — about as many operations as there are nonzeros, here roughly 5 million, not 10^12. We never modify A, never create fill-in, and never store anything dense. We stop as soon as the current x is close enough, which is often after far fewer steps than n. The price is that the answer is approximate and we must judge when to stop; the reward is that problems with millions of unknowns become solvable at all.

The rule of thumb: use a direct solver when the matrix is small, or moderately sized and dense, or when you must solve many right-hand sides with the same A (you factor once and reuse). Reach for an iterative solver when A is large and sparse — and crucially, the iteration is only as good as its preconditioner. A naive iteration on a hard problem can crawl or stall; the real engineering is choosing a good preconditioner so that convergence is fast. This is why iterating, not factoring, is the default for the giant sparse systems that come out of discretized partial differential equations.