the transformer block stack

A Transformer gets its power from repetition. One block — attention followed by a feed-forward network, each wrapped in normalization and a residual add — is useful but limited. Stack dozens of identical blocks on top of each other and something richer emerges: early layers handle surface patterns like word forms and local grammar, middle layers assemble phrases and meanings, and late layers shape the abstract features that drive the final prediction.

Every block reads from the residual stream, computes its update, and adds it back, so the same vector slot is refined again and again as it rises through the stack. The blocks are structurally identical but learn different weights, forming a kind of processing pipeline. Depth, how many blocks, and width, how big each vector is, are two of the main knobs that set a model's size, alongside the number of attention heads.

This uniform, repeatable design is part of why Transformers scale so smoothly: adding capacity is mostly a matter of stacking more of the same block. It also means there is no single layer 'where understanding happens' — meaning is built incrementally, and interpretability research often traces how a feature sharpens layer by layer.