Stacking the Engine: Depth, Context Mixing, and Why It Scales

One block, on repeat

We now have a complete transformer block: multi-head attention, then a feedforward network, each wrapped with normalization and added into the residual stream. A full model is just this identical block stacked many times — a small model might stack 12, a frontier model 80 or more. Tokens enter at the bottom as embeddings and flow upward, every block refining the running vectors a little more.

Because every generative LLM is built from masked blocks like this with nothing bolted on top, the whole family is called the decoder-only architecture. From GPT to Llama to Claude, the skeleton is the same; what differs is size, the norm and position details from the last guide, and the training.

# a decoder-only transformer, end to end
x = embed(tokens) + positional_info        # tokens -> vectors

for block in range(n_layers):              # e.g. 12 ... 80+
    x = x + attention(norm(x), causal=True)  # mix across positions
    x = x + feedforward(norm(x))             # think per position

x = norm(x)
logits = x @ embed.weight.T                # vectors -> next-token scores

Depth turns mixing into understanding

Why repeat the block at all? Because each layer does only one round of context mixing — one chance for words to exchange information. Stacking lets information travel and combine in stages, and a rough hierarchy emerges that researchers see again and again.

1. Early layers handle surface structure — token identity, nearby words, basic grammar. This is where positional and previous-token heads dominate.
2. Middle layers assemble meaning — resolving what it refers to, tracking who did what to whom, binding facts together.
3. Late layers sharpen the prediction — converting the accumulated understanding into a concrete guess about the very next token.

A neat way to watch this happen is the logit lens: decode the residual stream into a next-token guess at every layer, not just the last. You can literally watch the model's prediction firm up as the vector climbs the stack — vague at the bottom, confident at the top. Combined with attention pattern visualization from guide 3, it gives a surprisingly readable picture of the model's reasoning.

Why this engine, of all engines, scales

The transformer didn't win only because attention is clever. It won because it scales like almost nothing before it. Three properties stack up:

1. Parallelism. With no recurrence, every position is computed at once, so the architecture saturates modern GPUs and TPUs. You can train on trillions of tokens in a reasonable time.
2. Stable depth. The residual stream and pre-norm let gradients survive through enormous depth, so adding layers keeps helping instead of breaking training.
3. A uniform unit. The block is one repeated shape, easy to spread across thousands of chips — the same diagram, just wider and taller.

These properties are what make the famous scaling laws possible: across many orders of magnitude, a transformer's loss falls smoothly and predictably as you add parameters, data, and compute. That predictability is why labs are willing to spend fortunes on bigger runs — they can forecast roughly what they'll get. The transformer is, in a real sense, the first language-model architecture that rewards being made huge.

A scaling law: loss falls as a smooth power law in model size N — the regularity that makes bigger models predictably better.

What you can now read

Step back and look at what you can now trace end to end. Tokens become vectors; attention with queries, keys and values mixes context; multiple heads specialize while a causal mask keeps the model honest; the feedforward thinks; residuals and normalization keep the whole thing stable; position is encoded; and the block repeats until shallow patterns have become deep understanding. That is the engine inside every chatbot you have ever used.