masked autoencoder

The masked autoencoder (MAE) is a particularly clean and scalable instance of masked image modeling. The recipe is brutal in its simplicity: randomly mask a very high fraction of the image patches, typically 75 percent, and train an encoder-decoder to reconstruct the missing pixels. Such a high masking ratio is deliberate, because images are spatially redundant, masking only a little would let the model copy nearby pixels, but masking three quarters forces it to reason about global structure to reconstruct.

The architectural insight that makes MAE scale is its asymmetry. The encoder, a large Vision Transformer, processes only the visible 25 percent of patches, it never even sees the mask tokens. This makes the expensive encoder roughly four times cheaper to run, enabling training of very large models like ViT-Huge. Only afterwards are learned mask tokens inserted, and a deliberately lightweight, shallow decoder reconstructs the full image. The reconstruction loss is mean-squared error on the pixels of the masked patches only (often computed on per-patch normalized pixels for a small quality boost), and after pretraining the decoder is discarded, keeping just the encoder.

MAE's significance is that it made masked pretraining both simple and efficient enough to scale, and it strongly fine-tunes to state-of-the-art results on classification, detection and segmentation. Its characteristic trade-off is the classic MIM signature: weak under a frozen linear probe but excellent under fine-tuning, the opposite balance from contrastive methods like SimCLR. This is because MAE optimizes reconstruction rather than instance discrimination, so its features encode rich local and structural information that needs a little adaptation, not just a linear layer, to become task-ready.