transformer encoder block

A transformer encoder block is the single repeated building unit you stack to make a transformer — the ViT is just L copies of this block in a row. Each block does two things in sequence: first it lets tokens exchange information across the sequence (attention), then it processes each token on its own to refine it (an MLP). Mixing across tokens, then thinking per token; mix, then refine; repeat. Stacking many such blocks turns shallow comparisons into deep, compositional representations.

Inside one block there are two sublayers, each wrapped in the same protective pattern. Sublayer one is multi-head self-attention, which moves information between patches. Sublayer two is a position-wise MLP (a two-layer feedforward network applied identically to every token), which adds nonlinearity and mixes feature channels. Each sublayer is wrapped with a residual connection — the sublayer's output is added back to its input — and a layer normalization, which rescales each token's features to stable statistics. Residuals give gradients a clean highway through depth; normalization keeps activations well-conditioned.

The order of normalization matters and ViT uses the pre-norm arrangement: normalize first, then apply the sublayer, then add the residual, i.e. x = x + Attention(LN(x)) followed by x = x + MLP(LN(x)). This pre-norm placement (versus the original transformer's post-norm) is what lets very deep transformers train without careful learning-rate warmup tricks, because the residual path stays an unobstructed identity. A final layer norm is usually applied after the last block.

Conceptually the block is a clean division of labour: attention is the only operation that is global and content-dependent (it decides which tokens talk), while the MLP is local and identical everywhere (it decides how to transform each token's features). Everything a ViT computes is an interleaving of these two. The block has no convolution, no pooling, and no recurrence — its only structural prior is this mix-then-refine rhythm, which is why ViTs need data or added biases to learn what convolutions assume.