a penalty method
How do you obey a rule without a hard wall enforcing it? Add a fine. The further you stray from what is allowed, the bigger the fine, until staying in line is simply the cheaper choice. A penalty method turns a constrained optimization problem into an UNconstrained one by exactly this idea: replace each constraint with a cost term that grows whenever the constraint is violated, then minimize the combined objective with ordinary unconstrained methods.
To minimize f(x) subject to a constraint g(x) = 0, the quadratic penalty method minimizes instead f(x) + (rho/2) * g(x)^2 over all x, with no constraints. The squared term g(x)^2 is zero when the constraint holds and positive otherwise, and the penalty weight rho controls how harshly violations are punished. For small rho the constraint is loosely enforced; you then INCREASE rho (rho = 10, 100, 1000, ...) and re-solve, and as rho grows the minimizer is squeezed toward satisfying the constraint. Inequalities g(x) <= 0 use a one-sided penalty like (max(0, g(x)))^2. Each subproblem is a standard unconstrained minimization, so all the machinery of gradient descent, Newton, and quasi-Newton applies directly — that simplicity is the method's whole appeal.
Penalty methods are valued for being simple to implement and for handling general nonlinear constraints, and they underlie practical tools (including the soft constraints in many engineering and machine-learning setups). The honest and important catch: as rho grows large to enforce the constraint tightly, the unconstrained subproblem becomes ILL-CONDITIONED — the penalty term dominates and creates a steep, narrow valley, so the inner solver slows to a crawl and accuracy suffers. The quadratic penalty also only satisfies the constraint exactly in the limit rho -> infinity, never at finite rho. The fix is the AUGMENTED LAGRANGIAN method, which adds explicit Lagrange-multiplier estimates to the penalty, f(x) - lambda * g(x) + (rho/2) g(x)^2; this lets the constraint be satisfied at a MODERATE, fixed rho (avoiding the ill-conditioning blow-up), and it is the form used in serious solvers like ADMM and LANCELOT.
Minimize f(x) = x^2 subject to x = 3. The quadratic penalty minimizes x^2 + (rho/2)(x - 3)^2, whose minimizer is x = 3 rho / (rho + 2): at rho = 1 it gives x = 1, at rho = 100 it gives x = 2.94, at rho = 10000, x = 2.9994 — only approaching the true answer 3 as rho grows, while the valley gets ever steeper and harder to minimize.
Fine the violation; raise the fine until the constraint nearly holds.
The plain quadratic penalty satisfies the constraint exactly only as rho -> infinity, and large rho makes the subproblem severely ill-conditioned, so accuracy and the inner solver both degrade. Use the AUGMENTED LAGRANGIAN instead: explicit multiplier estimates let a moderate, fixed rho do the job without the ill-conditioning.