linear mode connectivity

Linear mode connectivity is the stronger, stranger cousin of mode connectivity. Ordinary mode connectivity only promises a curved low-loss path between two minima. Linear mode connectivity claims that once you correct for the network's permutation symmetries, two independently trained solutions are joined by a straight line of low loss — you can average their weights directly and pay almost no loss penalty.

The key is that a network has many symmetries: you can shuffle the neurons in a hidden layer, permuting weights accordingly, and compute exactly the same function. Two independent solutions usually look far apart only because they used different, arbitrary neuron orderings. Find the permutation that best aligns one solution to the other — the idea behind Git Re-Basin and the conjecture of Entezari and colleagues — and the linear path between the aligned weights has little or no loss barrier. Without this alignment, naive interpolation typically crosses a high barrier.

This reframes the loss landscape as essentially convex up to symmetry, and it underwrites the practice of merging independently trained models by weight averaging, with applications from federated learning to cheaply combining fine-tuned checkpoints.