Masked Image Modeling: The BERT Moment for Vision

A different paradigm: predict the missing pixels

Across this track we have learned representations by comparing views. Contrastive learning (Guides 2-3) pulled two augmented crops of the same image together and pushed different images apart; the distillation and clustering methods (Guide 4) dropped the negatives but still compared a student view against a teacher view. In every case, the learning signal came from a relationship between two transformed pictures. This final guide introduces a genuinely different idea: instead of comparing views, we reconstruct hidden content. We hide most of an image and train the network to fill in what is missing. This is a generative pretext task, and the family it defines is called masked image modeling (MIM).

The clearest way to grasp MIM is by analogy to natural language processing. The model BERT revolutionized NLP with masked language modeling: take a sentence like "the cat sat on the [MASK]", hide a few words, and train the model to predict the missing word from the surrounding context. To guess "mat" the network must understand grammar, meaning, and the world. Masked image modeling borrows this trick wholesale, but the tokens are now image patches rather than words: we hide some patches of a picture and predict their content from the patches we left visible. Same recipe — hide, then predict from context — moved from text to pixels.

So let us define it cleanly. Masked image modeling is a self-supervised learning approach that masks a large fraction of an image's patches and trains a model to reconstruct the hidden content (either the raw pixels or some encoding of them) from the visible patches. The rest of this guide unpacks the two landmark recipes — MAE, which reconstructs pixels, and BeiT, which predicts discrete visual tokens — then steps back to compare what MIM learns versus what contrastive methods learn, and closes the whole track with the state of self-supervised vision.

Images as patches: the ViT foundation

Before we can "mask a patch", we need one prerequisite: the picture must already be cut into patches. That is exactly what a Vision Transformer (ViT) does. Given an input image — say 224x224 pixels — the ViT chops it into a regular grid of small, non-overlapping squares, typically 16x16 pixels each. A 224x224 image therefore becomes a grid of 14x14 = 196 patches. Each patch is a tiny tile of the original picture, and crucially the tiles do not overlap, so every pixel belongs to exactly one patch. This is the same grid you saw in the figure above.

Each patch is then flattened into a token. A 16x16 patch with 3 colour channels contains 16x16x3 = 768 numbers; the ViT runs this vector through a small linear layer to produce one fixed-length embedding — a single token — and adds a positional encoding so the model knows where in the grid the patch came from. The result is a sequence of 196 tokens, which the transformer processes with self-attention: every token can look at every other token and decide how much each one matters. This is precisely analogous to how a language transformer treats the words of a sentence, where each word-token attends to all the others. A patch is to an image what a word is to a sentence.

This patch tokenization is the quiet reason BERT-style masking transfers so cleanly from text to images. Once a picture is a sequence of discrete tokens, "hide some tokens and predict them" is a well-defined operation — you literally remove a patch token from the sequence, just as BERT removes a word. With a convolutional network there is no natural token to drop, but with a ViT the masking is trivial. That is why masked image modeling rose hand-in-hand with the ViT: the architecture made the BERT recipe possible for vision in the first place.

MAE: the masked autoencoder

The cleanest and most scalable instance of MIM is the Masked Autoencoder (MAE). The name says it: it is an autoencoder — an encoder that compresses, a decoder that reconstructs — trained on masked inputs. What made MAE famous were two deliberate, somewhat counter-intuitive design choices. Get these two ideas and you understand MAE.

Choice 1: a very high masking ratio, around 75%. In BERT, masking about 15% of words is enough because language is information-dense — each word carries a lot. Images are the opposite: they are spatially redundant. A missing patch of blue sky can be guessed from neighbouring sky patches almost trivially. If you mask only 15% of an image, the network can fill the gaps by naive copying and interpolation from its immediate neighbours, learning almost nothing about objects. By hiding 75% of patches, MAE makes the gaps so large that local copying fails — the model is forced to reason about whole objects and scene structure to reconstruct what is missing. A hard task is an informative task.

Choice 2: an asymmetric encoder-decoder. This is the trick that makes MAE cheap to scale. The heavy ViT encoder processes only the visible 25% of patches — the masked patches are simply not fed to it at all. Because self-attention cost grows fast with sequence length, running the encoder on a quarter of the tokens is dramatically cheaper. The masked positions are reintroduced only afterwards, at the decoder: the encoded visible tokens are placed back in their grid positions, and every masked slot is filled with a shared, learnable mask token (plus its positional encoding so the decoder knows which location to reconstruct). A lightweight decoder then attends over this full set and outputs pixel predictions for the masked patches.

Patchify the image into a grid of non-overlapping tokens (the ViT step).
Randomly select ~75% of patches to mask; keep the remaining ~25% as visible.
Run the deep encoder on the visible patches ONLY — the masked ones never enter the encoder.
Insert a shared learnable mask token (with positional encoding) at every masked location.
Run the lightweight decoder over the full sequence and predict the pixels of each masked patch.
Compute the reconstruction loss on the masked patches only; backpropagate.
After pretraining, DISCARD the decoder and keep the encoder as your transferable backbone.

That last step is the whole point. The decoder exists only to make reconstruction possible during training; once the encoder has been forced to build rich internal representations, the decoder has done its job. We throw it away and keep the encoder, then fine-tune it (or attach a linear head) for downstream tasks. The training objective that drives all of this is a simple pixel reconstruction loss:

\mathcal{L} = \frac{1}{|M|} \sum_{i \in M} \lVert x_i - \hat{x}_i \rVert^2

The MAE reconstruction loss: mean-squared error over the masked patches only.

Let us read this symbol by symbol. M is the set of masked patch indices — the patches the network was forced to guess — and |M| is how many there are (about 75% of all patches). The sum runs over `i ∈ M`, so the loss is computed only over masked patches; the visible patches the encoder already saw contribute nothing, because reconstructing what you were handed is no test of understanding. For each masked patch, x_i is the original pixels of that patch (in practice per-patch normalized, i.e. mean-and-variance standardized within the patch, which empirically sharpens the target), and x̂_i is the decoder's reconstruction of those pixels. The term `‖x_i − x̂_i‖²` is the squared L2 norm of the difference: square every pixel error and add them up — ordinary mean-squared error. Dividing by |M| averages it.

Concretely: suppose a masked sky patch has true (normalized) pixel values and the decoder outputs values that are off by 0.1 on each of its 768 numbers. Then this patch contributes about 768 x 0.1² = 7.68 to the inner sum; average that over all ~147 masked patches and you have the loss to minimize. The intuition behind the whole objective: to make x̂_i match x_i, the network must infer missing content from the surrounding visible context, and with 75% hidden the only way to do that is to learn object shapes, textures, and scene layout. Meanwhile, because the encoder only ever touches the visible 25%, scaling MAE to huge ViT encoders and enormous datasets stays affordable. High masking makes the task informative; the asymmetric design makes it cheap.

# MAE training step (pseudocode) — see the asymmetry: encoder sees only visibles
def mae_step(image, mask_ratio=0.75):
    patches = patchify(image)                 # [N, patch_dim], N = 196 for 224x224 @ 16px
    target  = per_patch_normalize(patches)    # reconstruction target x_i

    # 1) randomly partition patches into visible / masked
    keep_idx, mask_idx = random_split(N=len(patches), mask_ratio=mask_ratio)

    # 2) ENCODER runs on the visible ~25% ONLY (this is the compute saving)
    vis_tokens = linear_embed(patches[keep_idx]) + pos_embed[keep_idx]
    enc_vis    = encoder(vis_tokens)           # deep ViT, short sequence

    # 3) rebuild full sequence: encoded visibles + shared mask token at holes
    full = scatter(enc_vis, keep_idx, into_length=len(patches))
    full[mask_idx] = mask_token                # one shared learnable vector
    full = full + pos_embed                     # tell decoder WHERE each slot is

    # 4) lightweight DECODER reconstructs pixels everywhere
    pred = decoder(full)                        # [N, patch_dim]

    # 5) loss is MSE over MASKED patches only  ->  (1/|M|) * sum_{i in M} ||x_i - x_hat_i||^2
    loss = mse(pred[mask_idx], target[mask_idx])
    return loss

# After pretraining: throw away `decoder`, keep `encoder` as the backbone.

MAE in pseudocode: the encoder touches only visible patches; the loss covers only masked ones.

BeiT: predicting visual tokens, not pixels

MAE asks the network to regress raw pixel values. A second important flavour of MIM — BeiT (Bidirectional Encoder representation from Image Transformers) — stays even closer to the original BERT recipe by classifying instead of regressing. The idea: rather than predict the exact pixels of a masked patch, predict a discrete visual token that summarizes what that patch contains, exactly as BERT predicts the discrete word-id of a masked word over a fixed vocabulary.

Where do these visual tokens come from? Before training the main model, BeiT uses a separate, frozen tokenizer — a discrete variational autoencoder (a discrete VAE) trained beforehand — that maps each patch to one integer id drawn from a learned visual vocabulary (think of a codebook of, say, 8192 'visual words'). So an image becomes a grid of integer token ids, much like a sentence is a sequence of word ids. During pretraining, BeiT masks patches of the input image, and the ViT must predict, by classification, the tokenizer's id for each masked patch. The tokenizer is never updated; it just provides ground-truth targets.

\max \; \sum_{i \in M} \log p\big(z_i \mid \tilde{x}\big)

The BeiT objective: maximize the log-probability of the correct visual token at each masked patch (equivalently, minimize cross-entropy).

Reading the objective symbol by symbol: M is again the set of masked patch indices. z_i is the discrete token id of patch i, the integer produced by the frozen dVAE tokenizer — this is the label we want the model to output, the visual equivalent of 'the correct word'. x̃ (x-tilde) is the corrupted image: the input with the masked patches removed or replaced. p(z_i | x̃) is the model's predicted probability — a softmax over the entire visual-token vocabulary — that masked patch i is token z_i, given the corrupted image. We maximize the sum of the log-probabilities of the correct tokens; equivalently we minimize the cross-entropy between the model's softmax and the true token id, exactly the loss used to train any classifier. Intuitively: for each blanked-out patch, the network outputs a distribution over thousands of 'visual words' and is rewarded for putting mass on the right one.

The contrast with MAE is now sharp. MAE's loss `‖x_i − x̂_i‖²` is regression onto continuous pixel values; BeiT's loss is classification onto a discrete token id. The trade-off being made is real: a discrete visual token tends to capture more semantic, higher-level information and discards some low-level pixel detail, because the tokenizer compresses the patch into a single id. That can make the target less about precise textures and more about 'what is here'. The cost is an extra tokenizer stage — you must train (or download) a discrete VAE first, an added moving part that MAE, with its direct pixel target, avoids entirely.

Pixels vs tokens vs contrast: what each learns

We now have three broad families of self-supervised vision: contrastive learning (Guides 2-3), distillation/clustering (Guide 4), and masked image modeling (this guide). They do not produce interchangeable representations, and choosing well means understanding the empirical pattern that has emerged across many papers. Here it is, stated honestly.

Contrastive learning and distillation methods produce features that are highly linearly separable: you can freeze the backbone, fit a single linear layer on top, and already get strong classification accuracy. They therefore shine under the linear-probe and kNN evaluation protocols from Guide 1 — the protocols that test how much useful structure is readable from the frozen features with almost no extra learning. Masked image modeling is the opposite: MAE/BeiT features are less linearly separable, so their linear-probe numbers look comparatively weak. But after full fine-tuning — unfreezing the whole backbone and adapting it to the task — masked models often match or beat the contrastive ones, and they tend to excel on dense prediction tasks like object detection and semantic segmentation, where fine local structure matters.

Why this split? It traces straight back to what each objective optimizes. Contrastive and distillation methods are built on instance discrimination: the signal is global and image-level — 'is this the same image as that one?' That pressure organizes the whole image into one clean, linearly-readable vector, which is exactly what a linear probe loves, but it can wash out fine spatial detail. Masked modeling optimizes a local, spatially-detailed signal — 'what exactly belongs in this 16x16 hole at this position?' That forces the network to preserve per-location structure across the feature map, which is precisely what detection and segmentation need, but it does not pre-arrange the features into a single neatly separable direction, so a bare linear probe reads them less easily.

So the practical guidance is concrete, not hand-wavy. If you will freeze the backbone and need strong features off the shelf — few-shot, kNN retrieval, or a quick linear evaluation protocol readout — reach for a contrastive or distillation model (DINO is a standout here). If you will fine-tune and especially if your target is dense (detection, segmentation), masked image modeling is a superb initializer. Both are valid routes to good representation learning; the right one depends on whether your downstream budget allows fine-tuning and on how local your task is.

The state of self-supervised vision

Step back and look at the whole journey this track has taken. We began (Guide 1) with hand-crafted pretext tasks — predicting rotations, solving jigsaw puzzles, colourizing — clever but ad hoc, each task teaching only a sliver of what vision needs. Then came contrastive learning (Guides 2-3): SimCLR and MoCo turned 'is this the same image?' into a powerful, general signal, with MoCo's momentum queue solving the hunt for enough negatives. Guide 4 showed we could even drop the negatives: BYOL and SimSiam learn by self-distillation with stop-gradients, SwAV adds online clustering, and DINO produces strikingly semantic features through self-distillation with a momentum teacher. And this guide brought masked image modeling (MAE, BeiT), importing BERT's hide-and-predict recipe into vision. Four families, one goal: learn from images without labels.

Where is the frontier headed? The most fruitful direction has been to combine masking with distillation rather than treat them as rivals. The clearest example is the DINOv2-style training recipe, which blends DINO's self-distillation (for clean, semantic, linearly-separable global features) with masked-modeling objectives (for rich local structure) — getting, in effect, the best of both columns of the previous section. Alongside this, the field has pushed hard on scale: training these objectives on enormous, uncurated image collections scraped from the internet, often with automated data-curation pipelines to filter and balance them, because self-supervision finally lets us use images that no human ever labelled.

The payoff of combining better objectives with massive scale is the foundation backbone: a single pretrained encoder whose frozen features transfer to a huge range of tasks — classification, detection, segmentation, depth estimation, retrieval — often with only a light head trained on top and little or no task-specific labeled data. This is the same role large language models play in NLP, now realized for vision. A general-purpose visual encoder, learned once from unlabeled images, that you reuse everywhere.

And that brings the whole track full circle to the promise of Guide 1. Supervised learning chained vision to the labeling bottleneck — every new capability demanded a fresh, expensive, human-annotated dataset. Self-supervised learning broke that chain: by turning the internet's endless supply of unlabeled images into a training signal, it produced general-purpose visual representation learning at a scale labels could never reach. Masked image modeling is the latest and one of the most powerful tools in that toolkit — vision's BERT moment. The labels were never the point; the pictures were always enough.