alignment and uniformity

Alignment and uniformity are two measurable properties that explain, more cleanly than mutual information, what a good contrastive embedding actually looks like on the unit hypersphere. Alignment says that positive pairs — two views of the same thing — should map to nearby points. Uniformity says that, overall, the embeddings should spread out to cover the sphere evenly, preserving as much information as possible. A useful representation needs both: alignment alone permits collapse, uniformity alone ignores semantics.

Wang and Isola showed that the InfoNCE loss asymptotically decomposes into exactly these two forces and proposed measuring them directly. Alignment is the expected distance between positive pairs (lower is better). Uniformity is the log of the average Gaussian-potential energy between all pairs of points; minimizing it spreads points apart like mutually repelling charges, and it is minimized by the uniform distribution on the sphere. Plotting a representation on these two axes diagnoses contrastive methods more interpretably than the loss value alone.

The framing matters because it separates two requirements a single contrastive temperature secretly trades off: a low temperature emphasizes uniformity and hard negatives, a high temperature emphasizes alignment. It also clarifies why non-contrastive methods need explicit anti-collapse terms — they must manufacture uniformity that the InfoNCE denominator supplies for free.