class token

The class token is an extra, learnable vector you prepend to the patch sequence whose only job is to collect a summary of the whole image. None of the patch tokens is a natural place to hang a single 'what is this image?' answer — each one is tied to one location. So, borrowing the idea from BERT in NLP, the ViT adds one more slot at the front that belongs to no patch; as it flows through the attention layers it is free to gather information from every patch, and its final state becomes the image's global descriptor.

Concretely it is a single D-dimensional parameter vector (often written x_class or [CLS]), the same for every image and learned by backprop. It is placed at position 0 of the sequence, given its own position embedding, and processed by exactly the same attention and MLP layers as the patches — it is not special in the computation, only in its role. After the final encoder block, you take the class token's output vector, pass it through layer norm and a small linear classification head, and that produces the class logits. The patch outputs are typically discarded for classification.

There is a competing design: instead of a class token, simply average all the patch outputs (global average pooling, GAP) and classify that. GAP often matches or slightly beats the class token and removes a small asymmetry, so several modern ViTs drop the class token entirely. The class token persists partly for historical fidelity to BERT and partly because it gives one clean vector to read for downstream tasks and for visualizing where the model 'looks' (its attention to the patches).

A subtle point: the class token starts as a fixed, content-free vector, so the only way it can become a good summary is by attending to the patches — it acts as a learned query that pools the sequence. This is why its attention weights over the patches are a popular, if imperfect, saliency map: they show which regions the summary decided to listen to.