contrastive predictive coding (CPC)

CPC is the work that introduced the InfoNCE loss and the modern template for predictive self-supervision on sequences — audio, text, and time-ordered patches of images. Its premise is that a good representation of the present should let you predict the future, but predicting raw future signal (the next audio waveform sample) is hopeless and dominated by noise. So CPC predicts the future in a learned latent space and frames the prediction as a contrastive choice rather than a regression.

An encoder maps each timestep to a latent, and an autoregressive model (originally a GRU) summarizes the past latents into a context vector. To predict k steps ahead, a learned linear map of the context is scored against the true future latent and a set of negatives sampled from elsewhere in the data, using the InfoNCE loss. Predicting several steps into the future, rather than just the next one, encourages the context to capture slowly-varying, high-level structure (the identity of a speaker, the topic of a sentence) rather than fast local detail.

CPC's lasting influence is conceptual: it linked self-supervised prediction to mutual-information maximization and showed the recipe transfers across modalities. wav2vec for speech and many sequence-representation methods descend directly from it, and the contrastive-prediction idea reappears in JEPA's representation-space prediction.