From halving to squaring: the tangent-line idea
In the first guide of this rung the bisection method gave you a rock-solid guarantee: bracket a root, and the interval halves every step, gaining a little under one decimal digit per three steps. Safe, but slow. In the second guide, fixed-point iteration sped things up when the map was a contraction, shrinking the error by a constant factor each step — that is linear convergence, the error e_{n+1} roughly equal to C times e_n with C below 1. Newton's method now reaches for something qualitatively faster, and the whole secret is a single bold approximation: near the current guess, pretend the function is a straight line.
Picture the curve y = f(x) and your current guess x_n sitting somewhere on it. Draw the tangent line at that point — the straight line that touches the curve and shares its slope f'(x_n). A straight line crosses zero somewhere obvious, and you can solve for that crossing in one easy step. The bet Newton makes is that the tangent's zero is a much better guess for the curve's zero than where you started. So you slide to the tangent's x-intercept, call it x_{n+1}, and repeat. Each step replaces a hard nonlinear equation with a trivial linear one — and then asks the same question again at the new spot.
- Write the tangent at x_n: the line y = f(x_n) + f'(x_n) * (x - x_n), which touches the curve and copies its slope there.
- Set y = 0 and solve that linear equation for x — call the answer x_{n+1}. Algebra gives the Newton iteration x_{n+1} = x_n - f(x_n)/f'(x_n).
- Evaluate f and f' at the new point and repeat, halting once the residual |f(x_n)| (or the step) is below tolerance — the only ingredients ever needed are f and its derivative f'.
What 'quadratic convergence' really means
When Newton works, it works spectacularly because of quadratic convergence. The precise statement is about how the error e_n = x_n - r (the gap to the true root r) shrinks: near a simple root the next error is roughly proportional to the square of the current one, e_{n+1} approximately equal to M times e_n^2, for some constant M. Squaring a small number makes it tiny: if your error is 0.01, one step takes it to about 0.0001, the next to about 0.00000001. In plain terms, the number of correct digits roughly doubles every step. Three or four good iterations can carry you from a rough guess to full machine precision.
Where does the squaring come from? Expand f around the root with a Taylor series. The Newton step cancels the constant and the linear term of the error exactly — that is the whole point of using the tangent's slope — and the leading leftover is the quadratic term, governed by f''. Working it through gives e_{n+1} approximately equal to (f''(r) / (2 f'(r))) times e_n^2. Read that constant: the curvature f''(r) is the enemy (more bend, slower), and the slope f'(r) in the denominator is the friend (steeper crossing, faster). This is the order of convergence equal to 2 — strictly faster than the linear order 1 of bisection or a generic contraction.
A tiny worked example: the square root of 2
Let us actually feel the doubling. To find sqrt(2), look for the root of f(x) = x^2 - 2, whose derivative is f'(x) = 2 x. Plugging into x_{n+1} = x_n - f(x_n)/f'(x_n) and simplifying gives the famous Babylonian formula x_{n+1} = (x_n + 2/x_n) / 2 — average your guess with what it would have to be paired with to multiply to 2. Start deliberately rough at x_0 = 1, a full 0.41 away from the answer.
n x_n error |x_n - sqrt(2)| 0 1.0000000000000000 4.1e-1 1 1.5000000000000000 8.6e-2 2 1.4166666666666667 2.5e-3 3 1.4142156862745099 2.1e-6 4 1.4142135623746899 1.6e-12 5 1.4142135623730951 0 (to double precision) sqrt(2) = 1.41421356237309515 ...
Two honest lessons hide in that last line. First, the error does not march to literally zero — it stalls near 10^(-16) because every numerical answer is approximate and a floating-point number simply cannot represent sqrt(2) exactly. Newton converges to the best double available, not to the real-number truth. Second, this very smoothness is why you must never test convergence with x_{n+1} == x_n using exact equality: round-off can make consecutive iterates jitter in the last bit forever. You stop on a tolerance, which the next section pins down.
Stopping honestly, and the multiple-root trap
How do you know to quit? You cannot watch the true error e_n because you do not know the root r. Instead you watch surrogates. The cheapest is the residual |f(x_n)|: when it is below a small tolerance, x_n is at least a near-root in the sense that f is nearly zero there. The other common signal is the step size |x_{n+1} - x_n|: once a step barely moves you, you have likely arrived. A robust stopping criterion uses a relative tolerance, like halting when |x_{n+1} - x_n| <= tol times |x_{n+1}|, because absolute thresholds are meaningless without scale — a tolerance of 10^(-8) is loose for nanometers and absurd for light-years.
There is one structural way Newton loses its prized speed even with a perfect start: a multiple root. If f touches zero tangentially — like f(x) = (x - r)^2, where f'(r) = 0 too — then the derivation that cancelled the linear term breaks, and Newton degrades from quadratic all the way down to merely linear convergence, crawling toward r at a constant factor near 1/2 per step. Worse, dividing by an f' that is heading to zero amplifies round-off, so the last digits get noisy. The cure, once you suspect multiplicity m, is the modified step x_{n+1} = x_n - m times f(x_n)/f'(x_n), which restores the quadratic rate. The deeper point: Newton's stellar speed is a property of simple roots, not a universal law.
Why it matters and where it leads
Newton's method is the workhorse beneath an astonishing amount of computing. Every time software computes a reciprocal or a square root in hardware, evaluates an implied volatility, fits a nonlinear model, or takes an implicit time step on a stiff equation, a Newton-style iteration is usually doing the heavy lifting — precisely because doubling the digits per step makes it cheap to reach full precision. The same tangent idea reappears in optimization as Newton's method for minimization, where you apply it to f' to find where the gradient is zero, gaining the same quadratic speed near a well-behaved minimum.
But Newton carries a real cost: it demands the derivative f'(x) at every step. Sometimes you have a clean formula; often you do not, or it is expensive, or the function is only known as a black box. That single requirement motivates the next guide twice over. The secant method replaces the exact tangent with a line through the two most recent points — no derivative needed — and pays only a little speed (a convergence order of about 1.618, the golden ratio, between linear and quadratic). And the failure modes we kept gesturing at — overshoot, cycling, divergence from a poor start — are what motivate safeguarded hybrids that wrap Newton's speed inside bisection's guarantee.
Finally, hold onto the shape of the idea, because the rung's last guide scales it up. Solving a single equation became 'linearize, step, repeat'. Solving a whole system of nonlinear equations becomes exactly the same dance, with the slope f' promoted to a matrix of partial derivatives — the Jacobian — and the division f/f' promoted to solving a linear system J s = -f for the step. Every linear-algebra tool from the earlier rungs (LU factorization, conditioning, residual checks) comes flooding back in service of one nonlinear step. Newton's tangent is not just a trick for one equation; it is the template for nonlinear solving across all of numerical computing.