masked autoencoders (MAE)

Imagine tearing three quarters of the tiles off a mosaic, then asking someone to repaint the missing pieces from the few that remain. To do it convincingly they must understand what the scene is — a face, a dog, a street. Masked autoencoders turn this into a pretext task for images: hide most of the picture, force a network to fill it back in, and the representations it builds along the way turn out to be excellent features for downstream classification, detection, and segmentation.

Concretely an image is split into non-overlapping patches and a large fraction (commonly 75%) is randomly dropped. An asymmetric design does the heavy lifting: a Vision Transformer encoder processes only the small set of visible patches, so pretraining is cheap, while a lightweight decoder receives the encoder output plus learnable mask tokens at the dropped positions and reconstructs the raw pixels there. The loss is mean-squared error on the masked patches only. After pretraining the decoder is discarded and the encoder is fine-tuned or linearly probed.

The high masking ratio is the crucial design choice: because natural images are spatially redundant, light masking lets the model cheat by interpolating neighbours, whereas aggressive masking forces genuine semantic inference. MAE is the vision analogue of masked language modeling, but reconstructing continuous pixels rather than discrete tokens, which is why a per-patch normalized pixel target works better than predicting tokens for fine-tuning.