feed-forward block
/ feed-FOR-werd blok /
The feed-forward block is the second half of every Transformer layer, the quiet partner to attention. After attention has let the tokens share information with each other, the feed-forward block takes each token on its own — independently, with no looking sideways — and pushes it through a small two-layer neural network. If attention is the part where tokens talk, the feed-forward block is the part where each token sits and thinks privately about what it just heard.
Its shape is simple but deliberate: expand, activate, contract. The token's vector is first projected up to a much larger width (often four times bigger), passed through a nonlinear activation that lets it represent more intricate functions, then projected back down to the original size. The exact same little network is applied to every position, which is why it is sometimes called "position-wise." Plain as it looks, this block holds a surprisingly large share of a model's total parameters — often around two-thirds.
Why care about such a humble component? Because evidence suggests the feed-forward blocks are where a lot of a model's learned knowledge actually lives. Attention moves information around; the feed-forward layers seem to store and transform it, acting somewhat like a giant set of learned lookup rules. This is also why mixture-of-experts designs target exactly this block — they replace one big feed-forward network with many smaller specialized ones and activate only a few per token, getting more capacity without proportionally more computation.
In a layer with width 512, the feed-forward block expands each token to 2,048 dimensions, applies an activation, then squeezes it back to 512. The same network runs on every token separately — token 5 never sees token 6 inside this block.
Expand, activate, contract — applied to each token alone.
It's easy to think attention does all the real work, but the feed-forward blocks hold the majority of a Transformer's parameters and appear to store much of its learned knowledge. Attention routes information; the feed-forward layers are where a lot of it is actually kept.