neural collapse

Keep training a classifier past the point where it already makes zero errors, into the so-called terminal phase, and a striking geometric order emerges in the last layer. The penultimate features of every example in a class collapse onto a single point — the class mean — and the class means themselves arrange into the most symmetric configuration possible. The messy cloud of representations crystallizes into a rigid, elegant skeleton.

The phenomenon has four coupled properties. Within-class feature variability shrinks to zero. The class-mean vectors form a simplex equiangular tight frame, all equal length, all pairwise angles equal and maximal. The last-layer classifier weights align with those means, a self-duality. And the network's decision reduces to nearest-class-mean. This is typically analyzed with the unconstrained-features model, which treats the last-layer features as free variables and recovers the simplex frame as the optimum.

Neural collapse offers a clean, quantitative description of what well-trained classifiers converge to, and connects to questions of generalization, transferability, and the harm of class imbalance, where the symmetric frame is distorted.