JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Loss-Landscape Geometry: Connectivity, Collapse, and Lottery Tickets

The set of good solutions is far more structured than the non-convex picture suggests. Minima are connected by low-loss paths, even linearly after fixing symmetry; last-layer features collapse to an elegant simplex; and dense nets hide trainable sparse subnetworks.

Mode connectivity: minima are not islands

Train the same architecture twice from different seeds and you land in two different minima of a wildly non-convex loss landscape. Naive intuition says a straight line between them climbs over a high loss barrier — and it usually does. The surprise of mode connectivity is that there exists a curved path between them along which the loss stays essentially flat. The minima are not isolated local minima; they are connected nodes on a single low-loss manifold.

Roll down the non-convex loss surface into a minimum — the seed and starting point decide which basin you land in.

An interactive ball rolling down a bumpy loss surface into one of several minima.

This reframes the landscape entirely. In high dimensions the solution set is closer to a connected web than a field of separate craters — consistent with overparameterization carving out a large, navigable valley of interpolants. It also has a practical payoff: ensembling models sampled along a low-loss curve gives cheap diversity, and averaging weights along the way (stochastic weight averaging) lands in a flatter, better-generalizing spot.

Linear mode connectivity and the role of symmetry

Curved paths are nice; straight ones are profound. Linear mode connectivity asks whether the straight line between two solutions stays low-loss. Across random seeds it usually does not — until you account for permutation symmetry. A network's neurons can be relabeled without changing the function, so two solutions may be the 'same' point viewed through different permutations. Align them (solve for the matching permutation) and the linear path between them often becomes barrier-free.

B(\theta_A,\theta_B)=\max_{\alpha\in[0,1]}\;\mathcal{L}\big(\alpha\,\theta_A+(1-\alpha)\,\theta_B\big)-\big[\alpha\,\mathcal{L}(\theta_A)+(1-\alpha)\,\mathcal{L}(\theta_B)\big]

The loss barrier along the straight line between two solutions; linear mode connectivity means this barrier is near zero.

The startling conjecture this motivates: modulo permutation, SGD solutions may live in a single basin — there is essentially one minimum, dressed up in many symmetric costumes. This is directly actionable for model merging: permute one model into another's frame and you can average independently trained networks into a working model, no retraining required. Symmetry is not a nuisance; it is the hidden coordinate system of the loss landscape.

Neural collapse: geometry at the very end of training

Keep training a classifier well past zero error, into the terminal phase, and the last-layer geometry crystallizes into something startlingly rigid. Neural collapse is a set of four coupled phenomena: within-class feature variability vanishes (every example of a class maps to its class mean); the class means arrange into a maximally separated simplex equiangular tight frame; the last-layer classifier weights align with those means (self-duality); and classification reduces to nearest-class-mean.

\cos\angle\!\big(\mu_c-\mu_G,\;\mu_{c'}-\mu_G\big)=\frac{K\,\delta_{cc'}-1}{K-1}

Neural collapse: centered class means become equiangular and maximally separated, forming a simplex equiangular tight frame.

Why care? Collapse is a clean, mathematically derivable endpoint — it falls out of minimizing cross-entropy with weight decay on separable features — and it hints that the representation the network settles on is determined more by the loss geometry than by the data details. It connects to generalization, transfer (collapsed features may transfer worse, being maximally task-specialized), and even to imbalanced learning, where the simplex distorts in predictable ways.

Lottery tickets: sparse subnetworks hiding in dense ones

One more piece of structure. The lottery ticket hypothesis claims a randomly initialized dense network contains a sparse subnetwork — a 'winning ticket' — that, trained in isolation from the original initialization, matches the full network's accuracy. You find it by training, pruning the smallest-magnitude weights, and rewinding the survivors to their original (or early) values rather than re-initializing them.

f\big(x;\,m\odot\theta_0\big),\qquad m\in\{0,1\}^{d},\quad \lVert m\rVert_0\ll d

A winning ticket is the sparse mask paired with the rewound initialization that, trained in isolation, matches the dense network.

W = init()                      # keep a copy of the original weights
for round in range(N):
    train(W)
    mask = keep_top_magnitude(W, sparsity)   # prune smallest weights
    W = init_copy * mask        # REWIND survivors to original init
Iterative magnitude pruning with weight rewinding — the recipe that surfaces winning tickets.

The 'rewind, don't re-init' detail is the crux: the specific pairing of mask and initialization is what wins, which says the lucky structure is set very early in (or even before) training. Read alongside connectivity and collapse, lottery tickets complete the chapter's thesis: the solution set of an overparameterized network is not a chaotic non-convex mess but a richly structured object — connected, symmetric, sparse-cored, and geometrically rigid at its endpoint.