Numerical Optimization

a trust-region method

Imagine you are mapping unknown terrain through fog, and you build a simple smooth model of the ground from what you can feel right where you stand. That model is only trustworthy nearby — a few steps out it becomes fiction. So you draw a circle of radius you TRUST, find the lowest point of your model within that circle, and step there. If the real ground matched your model well, you trust further next time and grow the circle; if it lied, you shrink the circle and try again. That is a trust-region method.

Concretely, around the current point x_k the method builds a quadratic model m(p) = f(x_k) + grad f(x_k)^T p + (1/2) p^T B_k p, where B_k is the Hessian or an approximation. Instead of choosing a direction first and a step length second (as a line search does), it solves both at once: minimize the model m(p) subject to ||p|| <= Delta_k, the trust-region radius. The trial step p is the model's best move inside that ball. Then it measures the agreement ratio rho = (actual decrease in f) / (decrease predicted by the model). If rho is close to 1 (model was good), accept the step and ENLARGE Delta; if rho is small or negative (model overpromised), REJECT the step and SHRINK Delta. The radius adapts automatically to how reliable the local model is.

Trust-region and line-search are the two pillars of step control, and trust-region has a key advantage: it works gracefully when the Hessian is NOT positive definite — near a saddle or in a region of negative curvature, where a pure Newton line-search step might point uphill or blow up, the trust region simply caps the step and still makes safe progress. This robustness makes it the backbone of methods like Levenberg-Marquardt (whose damping parameter is exactly an implicit trust-region radius) and many production nonlinear solvers. The honest cost is that solving the constrained subproblem (the trust-region subproblem) exactly each iteration is more involved than a backtracking line search, so practical codes solve it approximately (for example by the dogleg or Steihaug-CG method).

Suppose the quadratic model predicts f will drop by 10 if you take the full Newton step, but the step lands outside the trust radius Delta = 2, so you take the best step of length 2 instead. The model predicted a drop of 6 there; the real function drops 5.4, so rho = 5.4 / 6 = 0.9 — a good match, so accept and grow Delta. If instead f had RISEN, rho < 0: reject, shrink Delta, re-solve.

Trust the model only inside a ball; resize the ball by how well it predicted.

Trust-region and line-search solve the same problem from opposite ends: line search fixes a direction then picks a length, while trust-region fixes a maximum length then picks the best direction inside it. The trust-region wins when curvature is indefinite (near saddles), where a Newton line-search direction can be meaningless.

Also called
trust regionrestricted-step method信賴區域法信任域法