The Rest of the Block: Feedforward, Residuals, Norm, and Position

Attention moves information; the feedforward thinks about it

Attention is purely a mixing operation — it shuffles existing information between words but does no per-word reasoning. After each attention step comes a feedforward network (often called the MLP), applied to each word's vector independently. It expands the vector to roughly four times its size, applies a non-linear activation, then projects back down.

That fat hidden layer is where much of a model's stored knowledge lives. Interpretability work suggests feedforward layers behave like a giant key-value memory: a pattern in the input (say, *"the capital of France is"*) triggers specific hidden units that write *"Paris"* into the vector. Attention decides what to look at; the feedforward decides what to make of it.

The residual stream: a shared highway

Here is the most important structural idea in the whole architecture, and the easiest to overlook. Attention and feedforward do not replace a word's vector — they add to it. Each component reads the current vector, computes an update, and adds that update back. The running vector that flows straight through the model, untouched except for these additions, is the residual stream.

Every component adds its correction back onto the word's vector instead of replacing it — the read–update–add residual stream.

This read–update–add view is also how modern interpretability reasons about models: each head and each feedforward writes a contribution into a shared stream, and what the model finally predicts is the sum of all those contributions.

Normalization keeps the numbers sane

Adding update after update onto the residual stream risks the numbers drifting to wildly different scales, which makes training unstable. Layer normalization fixes this by rescaling each word's vector to a consistent magnitude before it enters attention or the feedforward. It doesn't change the information's direction, just its volume — like an automatic gain control.

Most recent models use a lighter, cheaper variant called RMSNorm, and they place the norm before each component (pre-norm) rather than after, because that arrangement trains far more stably at depth. The exact placement is a small choice with a big effect — get it wrong and a 70-layer model simply won't converge.

RMSNorm rescales each vector by its root-mean-square so updates stay at a sane scale — cheaper than full layer norm.

Telling the model what order things came in

One subtlety we glossed over: attention is order-blind. Because it compares every word with every other in parallel, swapping two words gives the model no signal that anything moved — *"dog bites man"* and *"man bites dog"* would look identical. Word order obviously matters, so we have to inject it explicitly. That injection is positional encoding.

Early transformers added a fixed wave-like signal to each word's vector to stamp its position. Most current LLMs instead use rotary position embedding (RoPE), which rotates the query and key vectors by an angle proportional to position. The elegant payoff: when two words' queries and keys are compared, the rotation makes the score depend only on their relative distance, so the model naturally handles *"five words apart"* the same way wherever it appears in a long document.