JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

The Modern Block, Part by Part

RMSNorm, rotary positions, and the SwiGLU feed-forward — the three upgrades that turned the 2017 block into the 2024 block.

Same skeleton, sharper organs

Every modern block still does mix, then transform. But the original parts — absolute positional encodings, LayerNorm, a ReLU feed-forward — have each been quietly replaced. None of these swaps changed the shape of the computation; they changed the quality of each piece. Together they are why a 2024 model trained on the same data as a 2020 model is simply better.

The modern block keeps the same skeleton — norm, attention, residual, feed-forward — while each organ gets quietly upgraded.

Diagram of a transformer block: normalization, attention, a residual connection, then a feed-forward network.

RMSNorm: just the scale

A normalization layer keeps the numbers flowing through a deep network from drifting too large or too small. RMSNorm does the bare minimum that works: it computes the root-mean-square of a token's vector and divides by it, then multiplies by one learned gain per dimension. No mean subtraction, no bias. That is one fewer reduction over the vector and a handful fewer parameters per layer — small per block, but multiplied across 80 blocks and trillions of forward passes, it is real.

\text{RMSNorm}(x)=\frac{x}{\sqrt{\dfrac{1}{n}\sum_{i=1}^{n} x_i^2+\epsilon}}\odot g

RMSNorm does the bare minimum: divide each vector by its root-mean-square, then rescale by a learned gain g — no mean subtraction.

RoPE: rotate, don't add

Attention is order-blind: shuffle the tokens and the math is the same, so the model needs to be told about position. The 2017 trick was to add a positional encoding vector to each token. The modern trick is RoPE — rotary position embeddings — which instead rotates each query and key vector by an angle proportional to its position before computing attention.

The 2017 trick added a fixed positional vector to each embedding; RoPE instead rotates the query and key.

Diagram showing a positional encoding vector added on top of a word embedding vector.

The beauty is what falls out: when you rotate a query at position 10 and a key at position 7, their dot product depends only on the difference, 3. So attention naturally sees relative distance, not absolute index — exactly what language wants. And because position is baked into the angles rather than an input you add once, you can stretch the rotation frequencies to handle sequences far longer than training, the foundation of most long-context tricks.

# RoPE, in spirit: rotate Q and K by an angle tied to position m
q_m = rotate(q, angle = m * freqs)   # query at position m
k_n = rotate(k, angle = n * freqs)   # key   at position n
score = dot(q_m, k_n)                # depends only on (m - n)
Rotary positions make the attention score a function of relative distance.

SwiGLU: a smarter feed-forward

After attention mixes tokens, the feed-forward network transforms each token on its own — and it holds about two-thirds of a model's parameters. The original used a single hidden layer with a ReLU. Modern models use SwiGLU: a gated unit with two parallel projections, where one branch passes through a smooth Swish activation and then multiplies the other branch element-by-element. The gate lets the network learn, per dimension, how much signal to let through.

\text{SwiGLU}(x)=\big(\text{Swish}(xW_1)\odot xV\big)W_2,\quad \text{Swish}(z)=z\,\sigma(z)

SwiGLU gates one linear projection by a Swish-activated twin — three weight matrices replacing the old two-matrix ReLU feed-forward.

Why these particular swaps and not a hundred others? Because each survived the only test that matters at this scale: train two models that differ in exactly one part, and keep the one with lower loss. RMSNorm, RoPE, and SwiGLU are the choices that won those bake-offs across many labs — which is why models from competing companies look so alike inside.