Masked and Predictive Modeling: MAE, VQ-VAE, JEPA

The generative route to representations

The previous three guides all compared views. There is an older, equally powerful idea: hide part of the input and train the model to fill it back in. In language this is masked language modeling — the engine behind BERT — where you mask tokens and predict them from context. The natural question for vision was whether the same denoising-autoencoder recipe could rival contrastive methods on images. The answer, after a few false starts, was a resounding yes.

Masked autoencoders (MAE)

Masked autoencoders (MAE) made masked image modeling both effective and efficient with two design choices. First, an aggressive masking ratio — mask 75% of image patches, far more than language's 15%, because images are spatially redundant and an easy task teaches nothing. Second, an asymmetric encoder-decoder: the heavy ViT encoder sees only the ~25% visible patches (huge compute savings), and a lightweight decoder reconstructs the missing pixels from encoder outputs plus mask tokens. After pretraining you discard the decoder and keep the encoder.

\mathcal{L}_{\mathrm{MAE}} = \frac{1}{|\mathcal{M}|}\sum_{i\in\mathcal{M}} \big\lVert \hat{x}_i - x_i \big\rVert_2^2

MAE's objective: a pixel-reconstruction loss computed only over the masked patch set M.

Discrete targets: VQ-VAE and tokenized prediction

Reconstructing raw pixels spends capacity on imperceptible high-frequency detail. A discrete alternative is to predict tokens. The VQ-VAE learns a finite codebook and maps each image patch to its nearest codebook entry, turning a continuous image into a grid of discrete tokens — exactly the substrate a language-model-style objective wants. Masked token prediction (predict the missing codebook indices rather than raw pixels) often gives cleaner targets and underlies many image and multimodal generators.

z_e = encoder(x)                       # continuous patch features
idx = nearest_codebook_index(z_e, C)   # quantize to discrete tokens
z_q = C[idx]                           # straight-through estimator
x_hat = decoder(z_q)                   # reconstruct from codes

VQ-VAE: a learned codebook turns continuous patches into discrete, predictable tokens.

Predict in representation space: JEPA

Reconstructing pixels forces the model to model everything, including unpredictable detail (the exact texture of grass). Joint-Embedding Predictive Architectures (JEPA) argue you should instead predict the representation of the masked region, not its pixels. An encoder embeds the visible context; a predictor guesses the embedding a (momentum) target encoder would assign to the hidden region; the loss lives entirely in feature space. This abstracts away pixel-level noise and echoes CPC's 'predict-the-latent' philosophy — but without negatives, using the same stop-gradient momentum target as BYOL.

\mathcal{L}_{\mathrm{JEPA}} = \big\lVert\, g_\phi\!\big(E_\theta(x)\big) - \operatorname{sg}\!\big(E_{\bar{\theta}}(y)\big) \big\rVert_2^2

JEPA predicts the target's embedding from the context, matching in representation space — the stop-gradient target encoder prevents collapse.

Pixels vs features: what should you predict?

This is the live tension of the field. Reconstructing pixels (MAE) is simple, stable, and gives a useful generative prior, but wastes capacity on noise and yields weaker linear separability. Predicting discrete tokens (VQ-VAE-style) cleans up the target but inherits the codebook's biases and a tricky quantization step. Predicting in latent space (JEPA) is the most data-efficient and abstract, but reintroduces the collapse risk that pixel reconstruction was immune to — you are back to needing a stop-gradient and momentum target to keep the latent target honest.

Frontiers and open problems

Where is this going? Three honest open problems. (1) Unification — contrastive, distillation, redundancy-reduction, and masked methods all fight collapse and all maximize a spread-out, view-invariant representation; a single theory that subsumes them is still emerging. (2) Beyond images — video, audio, and embodied data have temporal and causal structure that static-image augmentations ignore, and the best pretext task there is unsettled. (3) Evaluation honesty — linear probing is convenient but can mislead; the community keeps discovering that the 'best' SSL method depends heavily on the downstream task, the architecture, and the compute budget.