beit

BEiT (Bidirectional Encoder representation from Image Transformers) is the masked image modeling method that most faithfully mirrors BERT, because it predicts discrete tokens rather than continuous pixels. The key difficulty it addresses: language has a natural vocabulary of words to predict, but an image has only continuous pixel values, so what is the discrete target for a masked patch? BEiT's answer is to first turn the image into a sequence of discrete visual tokens using a separate, pretrained image tokenizer, and then predict those tokens.

Concretely, training has two representations of each image. A patch view feeds the Vision Transformer, and a visual-token view comes from a discrete variational autoencoder tokenizer (BEiT used the dVAE from DALL-E) that maps each patch region to an index in a fixed codebook of, say, 8192 visual words. During pretraining you mask a subset of the input patches, and the ViT must predict the codebook index, the visual token, of each masked patch. This is a classification problem over the codebook, trained with cross-entropy, directly analogous to BERT predicting masked word IDs.

The motivation for predicting tokens instead of pixels is that raw-pixel regression spends model capacity on reproducing high-frequency, low-level detail (exact textures, noise) that is not very semantic. By forcing prediction of discrete, already-abstracted visual tokens, BEiT pushes the model toward higher-level structure. The cost is the dependence on a separately trained tokenizer, an extra stage and an extra design choice; later methods like MAE showed that simple pixel reconstruction with a high mask ratio and asymmetric encoder can be competitive without any tokenizer, while BEiT v2 improved the tokenizer to use more semantic, distilled targets.