layer norm placement

Layer normalization is a small housekeeping step that rescales the numbers flowing through each token so they have a consistent, well-behaved range — not too big, not too small — before the next computation. Without it, the values passing up through a deep stack of layers can balloon or shrink uncontrollably, and training falls apart. "Placement" refers to a seemingly tiny design choice that turns out to matter a great deal: where in each layer you put this normalization step.

There are two main options. In post-norm, the original design, normalization comes after the attention or feed-forward block and after its residual connection — the layer does its work, adds the input back, then normalizes. In pre-norm, normalization comes first, before the block, so the residual path runs clean and unnormalized from bottom to top. That seems like a trivial reshuffling, but it changes how gradients flow during training in a way that becomes decisive once you stack many layers.

The practical upshot: pre-norm is far easier to train deep. Post-norm models often need delicate warm-up schedules and careful tuning or they simply diverge, whereas pre-norm models train stably even when very deep, which is why most large modern Transformers use it. The honest nuance is that this stability isn't free — post-norm, when you can tame it, sometimes reaches slightly better final quality. It is a genuine trade-off, and the fact that such a minor-looking placement decision can make or break a billion-dollar training run is a humbling reminder of how much of this field is empirical engineering, not clean theory.