Multimodal & Vision-Language

image-text contrastive loss

The image-text contrastive loss is the engine behind CLIP-style alignment: within a large batch, each image is paired with its true caption and pitted against every other caption in the batch as a negative. The loss rewards the model for making a matched image-text pair more similar than all the mismatched ones, so over training the two encoders learn to push corresponding pictures and sentences toward the same point on a unit sphere while shoving everything else away.

Formally it is a symmetric InfoNCE objective over the batch similarity matrix. Each image and text is encoded and L2-normalised, all pairwise cosine similarities are scaled by a learned temperature, and a cross-entropy is taken twice, once treating images as queries against the texts and once the reverse. Because every other example in the batch acts as a negative, the gradient signal grows with batch size, which is why these models are trained on tens of thousands of pairs per step across many accelerators.

The loss only enforces that matched pairs rank above mismatched ones, not that the geometry be otherwise sensible, which leaves room for the modality gap and for shortcut features. Hard-negative mining, sigmoid variants that drop the softmax normalisation, and caption quality all materially change what the embedding space ends up encoding.

\mathcal{L}=-\frac{1}{2N}\sum_{i=1}^{N}\left[\log\frac{e^{\langle u_i,v_i\rangle/\tau}}{\sum_{j}e^{\langle u_i,v_j\rangle/\tau}}+\log\frac{e^{\langle u_i,v_i\rangle/\tau}}{\sum_{j}e^{\langle u_j,v_i\rangle/\tau}}\right]

Symmetric InfoNCE over a batch of N pairs: image embeddings u, text embeddings v, learned temperature tau.

Also called

InfoNCE圖文對比損失CLIP loss