instance discrimination

Instance discrimination is the pretext task at the root of the whole contrastive family. The idea is almost absurdly literal: treat every single image in the dataset as its own class. With a million images you have a million-way classification problem where each class has exactly one member, and the network must learn to tell every image apart from every other. To do that well it has no choice but to discover features that capture what makes images distinct — which turn out to be exactly the semantic features we want.

Done naively this is intractable, because a softmax over a million classes is impossibly expensive and each class has only one example. The original formulation solved this with a non-parametric classifier backed by a memory bank: an embedding for every image is stored and slowly updated, and a query is compared against its own stored embedding (the positive) versus a sampled set of others (negatives) using noise-contrastive estimation. Augmentation supplies the positive view so that the embedding of an image must be stable to crops and color changes while still being unique among instances.

This is the conceptual seed from which SimCLR, MoCo, and the rest grew. MoCo's queue and momentum encoder are a refinement of the memory bank that keeps the stored negatives consistent; SimCLR replaced the bank entirely with in-batch negatives. Recognizing the lineage clarifies that 'contrastive learning' is, at heart, instance discrimination made efficient.