convexity

A convex function is one shaped like a single smooth bowl: pick any two points on its curve, stretch a straight string between them, and the string never dips below the curve. The everyday consequence is wonderful — there is exactly one lowest point, and no false bottoms to trap you. Wherever you stand, walking downhill always leads to the one true minimum. A non-convex function, by contrast, is a rugged mountain range full of local dips that each look like the bottom but are not.

This 'one bowl, no traps' property is the dream of anyone doing optimization, which is the art of finding the lowest (or highest) point of a function. With a convex problem, simple downhill methods are guaranteed to find the global best answer, and they get there efficiently. You never have to wonder whether a better solution is hiding somewhere you did not look — convexity promises there isn't one.

Classic machine learning methods like linear regression, logistic regression, and support vector machines are built to be convex precisely so their training is reliable and reproducible: run it twice, get the same answer. But here is the candid truth about deep learning — neural networks are emphatically not convex. Their error landscapes are wildly rugged, riddled with countless valleys and saddle points, and there is no guarantee gradient descent finds the very best one. The pleasant surprise of the last decade is that, in these enormous landscapes, many of the 'good enough' valleys turn out to work remarkably well in practice — which is why deep learning succeeds despite forfeiting the comfort of convexity.