modality gap

The modality gap is a surprising geometric fact about jointly trained embeddings like CLIP's: even though the contrastive loss is supposed to align images and text in one shared space, the image embeddings and the text embeddings do not actually intermingle. Instead each modality lives in its own narrow cone, and the two cones sit apart, separated by a clear gap, so a picture of a cat and the words a cat are close in the relative sense the loss cares about but globally far from each other.

The gap arises from two forces. At initialisation, deep networks with normalisation map their outputs into narrow cones on the hypersphere, and because the two encoders start in different random cones, they begin far apart; the contrastive temperature then preserves rather than closes this separation, because the loss only needs matched pairs to rank above mismatched ones, which it can achieve while keeping the modalities globally displaced. The result is a roughly constant offset vector between the image and text manifolds.

The gap is more than a curiosity: it shapes retrieval, where comparisons are usually within or relative rather than absolute; it complicates schemes that try to swap a text embedding for an image embedding or vice versa; and it can be deliberately closed or even exploited, since shifting embeddings along the gap direction has been used to improve zero-shot and generation behaviour. It is a reminder that a shared space need not be a homogeneous one.