feedforward mlp block

The feedforward MLP block is the second of the two sublayers in every transformer block, and it is where each token is transformed on its own after attention has let it gather context. If attention is the conversation among tokens, the MLP is each token going home to think privately about what it heard. It is applied identically and independently to every position — the same small two-layer network runs on patch number 5 and patch number 50 with the same weights — which is why it is called position-wise.

Its structure is deliberately simple: a linear layer that expands the token from dimension D up to a wider hidden dimension (typically 4 times D), a pointwise nonlinearity (GELU in ViTs, a smooth relative of ReLU), and a second linear layer that projects back down to D. So a 768-dimensional token in ViT-Base is lifted to 3072 dimensions, passed through GELU, and squeezed back to 768. The expand-then-contract shape gives the nonlinearity room to work in a higher-dimensional space before recompressing.

What does it actually contribute? Two things attention cannot. First, nonlinearity: attention is, weights aside, a linear combination of values, so without the MLP a stack of attention layers would collapse toward a single linear map; the GELU is much of what makes the network a universal function approximator. Second, channel mixing and memory: attention mixes information across tokens but treats feature channels somewhat uniformly, whereas the MLP recombines the feature channels within each token, and its weights are widely argued to store much of a transformer's factual 'knowledge'.

It is also where most of the parameters and a large share of the compute live: with a 4x expansion the two MLP matrices hold roughly twice the parameters of the attention projections. Consequently the MLP is a prime target for efficiency work — mixture-of-experts replaces the single MLP with many that are sparsely activated, and the hidden-expansion ratio is a standard knob for trading capacity against speed.