feed-forward network (FFN)

If attention is where tokens talk to each other, the feed-forward network is where each token sits and thinks on its own. After a token has gathered context from its neighbors, the FFN processes that token in isolation — the same little network applied independently at every position. It does no mixing across the sequence; it just transforms each token's vector into something more useful.

The block is a small two-layer perceptron: expand the token vector up to a much wider hidden size, often four times wider, apply a nonlinear activation, then project it back down. That wide middle layer is where a surprising amount of the model's knowledge lives — many factual associations behave like key-value memories stored in these weights. In fact the feed-forward layers usually hold the majority of an LLM's parameters.

Because the FFN runs the same way on every token, it is cheap to parallelize and easy to scale. It is also the natural place to add a mixture-of-experts: instead of one shared network, the model routes each token to a few specialized FFNs, growing capacity without growing the cost of every token.