masked image modeling

Masked image modeling is the visual translation of the idea that made BERT work in language: hide part of the input and train the model to fill in the blanks. You split an image into a grid of patches, randomly mask out a large fraction of them, and ask the network to predict the missing content from the patches it can still see. To do this well the model must build an internal understanding of how images are structured, what tends to be next to what, so that it can hallucinate plausible content for the hidden regions.

This is a generative or denoising form of self-supervision, in contrast to the joint-embedding family (SimCLR, BYOL, DINO) that compares augmented views. Its natural partner is the Vision Transformer, because a ViT already operates on a sequence of patch tokens, so masking some tokens and inserting learned mask placeholders is straightforward. The supervision is automatic, the ground truth for a masked patch is simply the patch that was there, so again no human labels are involved.

The family is defined by what you predict for each masked patch. SimMIM and MAE regress the raw pixel values. BEiT predicts discrete visual tokens produced by a pretrained image tokenizer. MaskFeat predicts hand-crafted HOG features, and data2vec predicts the masked region's deep features from a momentum teacher. A practical hallmark is that MIM methods tend to produce features that fine-tune extremely well (often the best for detection and segmentation) but score lower under a frozen linear probe than contrastive methods, because the features are rich but not as immediately linearly separable.