convergence

Convergence is the point where training stops making meaningful progress — the loss has flattened out and further steps barely change anything. Picture our hiker descending into a valley: convergence is reaching a spot where every direction looks roughly level, so there's nowhere lower worth walking to. The model has settled, and continuing to train mostly just spins its wheels.

In practice you can't prove the model has hit the true bottom, so you use stopping criteria: the loss changes by less than some tiny amount between steps, or a fixed number of epochs has passed, or validation performance has stopped improving. "Converged" is therefore a practical judgment, not a mathematical certainty — it means "good enough, and no longer improving," not necessarily "globally optimal."

Two honest caveats matter. First, converging is not the same as converging to something good — a model can converge to a poor solution (a high, flat region) just as easily as a good one. Second, for the non-convex loss landscapes of deep networks there's no guarantee of reaching the lowest possible point at all; you reach some low-enough region and stop. Whether you converge quickly, slowly, or not at all depends heavily on the learning rate, the optimizer, and how well-conditioned the problem is — which is why divergence (the loss blowing up instead of settling) is the failure mode you watch for.