Why bigger models get better: the scaling-law idea

A regularity nobody had to invent

Train a large language model on next-word prediction, then ask how well it does — its cross-entropy loss, the average surprise per token. Now train a model that is ten times larger, or on ten times more text, or with ten times the compute. The loss does not wobble around unpredictably. It falls, and it falls along a strikingly clean curve. This empirical pattern is what we call neural scaling laws: model quality improves smoothly and predictably as you scale up the ingredients.

That last word — predictable — is the whole reason scaling laws matter. They turn the question "will a bigger model be better?" from a gamble into something close to engineering. You can measure a handful of small, cheap runs and read off, in advance, roughly how good a model a hundred times larger will be.

The straight line on log-log paper

The reason the curve looks so clean is that it is a power law. A power-law loss curve means loss behaves like one quantity raised to a (negative) power of another — for example, loss falls roughly as the model's parameter count raised to about −0.07. Plot it on ordinary axes and you get a vague downward swoosh. Plot it with logarithmic axes on both sides, and that swoosh straightens into a near-perfect line.

L(N) ≈ (N_c / N)^a   +   L_inf
         \_________/       \____/
          falls as N grows   the floor

A typical fit: loss L as a function of parameter count N. The first term shrinks as you add parameters; the second never moves.

Three knobs: parameters, data, compute

Scaling laws are usually written for three resources, and each gets its own power law. Parameter-count scaling asks what happens as you add weights to the model. Data scaling asks what happens as you feed it more tokens of training text. And compute scaling combines the two: the pretraining compute you spend, measured in FLOPs, is roughly proportional to parameters times tokens.

Add parameters but keep data fixed, and loss eventually stalls — the model memorises and stops generalising.
Add data but keep the model tiny, and loss also stalls — a small model can only absorb so much.
Grow both together, with enough compute to use them, and the clean power-law descent continues.

L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}

Each knob gets its own power-law term: parameters N and data D each push the loss down toward the floor E.

The floor you can't drill through

Notice the second term in the formula above — the one that never moves. That is the irreducible loss, the floor the curve approaches but never crosses. Human language is not perfectly predictable: even a flawless model cannot know which word you will pick next when several are equally reasonable. That residual uncertainty — the true entropy of the text — sets a hard floor. No amount of scale buys it back.

L(N) = L_{\infty} + \left(\frac{N_c}{N}\right)^{\alpha_N}

Loss splits into an irreducible floor L∞ that scale can't touch and a reducible term that shrinks as N grows.

Why this changed how labs build models

Before scaling laws were charted, building a frontier model meant guessing at a size and hoping. Afterward, teams could justify enormous, expensive runs on paper before spending a cent of the real budget, because the small-scale runs told them where the big one would land. The next guides turn this from a picture into a recipe: how to split a fixed budget between size and data (Chinchilla), how to read curves to plan a frontier run, and whether the new abilities people report are genuinely new — or just a trick of the measuring stick.