infonce loss

InfoNCE is the loss function that makes contrastive learning concrete. Intuitively it turns the vague goal pull positives together, push negatives apart into a familiar multiple-choice exam: given a query and a lineup of candidates consisting of one true positive and many negatives, the model must assign the highest similarity to the positive. The loss is just the cross-entropy of getting that classification right. Doing well means the positive scores clearly above all the distractors.

Concretely, let q be a query embedding, k+ its positive key, and k_1...k_K the negative keys. The loss for that query is L = -log [ exp(sim(q,k+)/τ) / ( exp(sim(q,k+)/τ) + Σ_i exp(sim(q,k_i)/τ) ) ], where sim is usually cosine similarity (a dot product of L2-normalized vectors) and τ is the temperature. This is exactly a softmax over K+1 logits followed by the negative log-likelihood of the positive class. The name InfoNCE comes from noise-contrastive estimation, and the loss is a lower bound on the mutual information between the two views: minimizing it maximizes how much one view tells you about the other.

The temperature τ is small but mighty. It rescales the similarities before the softmax: a low τ sharpens the distribution so the loss focuses hard on the most confusing nearby negatives (the hard negatives), which speeds learning but can be unstable and overly sensitive; a high τ softens it and treats all negatives more equally. Typical values are around 0.07 to 0.2. The other lever is the number of negatives K, more negatives tighten the mutual-information bound and generally improve features, which is precisely why SimCLR uses big batches and MoCo uses a queue.