mode connectivity

Train a network twice from different seeds and you land in two different minima that look like isolated valleys, each surrounded by high loss. Mode connectivity is the surprising finding that they are not isolated at all: you can usually draw a curved path through weight space joining the two solutions along which the loss stays low the entire way. The basins are doorways into one connected low-loss region.

Concretely, one fits a simple curve — a quadratic Bezier or piecewise-linear path with a few free knots — between two trained weight vectors and optimizes the knots so that loss along the path remains near the endpoints' loss. Garipov and colleagues, and Draxler and colleagues, showed such low-loss curves exist routinely, implying the set of good solutions is better described as a connected manifold than as separate isolated points. A naive straight line between the two minima, by contrast, usually crosses a loss barrier.

Beyond reshaping the picture of the loss landscape, mode connectivity has practical payoff: ensembling models sampled along the connecting path, and weight-averaging schemes that exploit the flat connected region, both improve generalization.