normalization placement (pre-norm vs. post-norm)

Every transformer block wraps a residual connection around its attention and feed-forward sub-layers, and the only question is where the normalization sits relative to that skip path. Post-norm, the original design, normalizes after adding the sub-layer's output back to the residual. Pre-norm, now standard, normalizes the input before the sub-layer and leaves the residual highway untouched. The difference looks tiny but decides whether very deep stacks train at all.

Pre-norm wins because it keeps a clean, unnormalized path running straight from input to output, so gradients flow down dozens of layers without exploding or vanishing, and training is stable even without a delicate learning-rate warmup. Post-norm can reach slightly better final quality when it trains successfully, but it is fragile at depth. Some recent models add extra norms, for instance around the attention output, to capture both benefits; the placement choice pairs naturally with the move to RMSNorm.