Chinchilla: spending a compute budget wisely

The budget is fixed — now what?

Imagine someone hands you a fixed amount of pretraining compute — say, the FLOPs you can afford in a month on your cluster. You can spend it on a bigger model that sees less data, or a smaller model that sees more. Every choice along that trade-off uses the same total compute, because, to a good approximation, compute equals about six times parameters times tokens.

C  ≈  6 · N · D

C = compute (FLOPs)
N = parameters
D = training tokens

The budget constraint: training FLOPs are roughly six times parameters N times tokens D, so for fixed C the two trade off.

Finding the sweet spot on that trade-off — the split of size versus data that reaches the lowest loss for a given budget — is the problem of compute-budget allocation, and a model trained at that sweet spot is called compute-optimal.

The plot twist: most big models were undertrained

The first influential scaling study (Kaplan and colleagues, 2020) concluded that if you get more compute, you should spend most of it on parameters and only modestly increase data. The field took the advice and built a wave of enormous models trained on comparatively little text.

In 2022 a DeepMind team revisited the question more carefully and found the opposite balance. Their result — now known as Chinchilla-optimal scaling — said that parameters and data should grow roughly in step. To prove it, they trained a 70-billion-parameter model called Chinchilla on far more tokens than was then fashionable, and it beat a 280-billion-parameter model (Gopher) that had used the same compute. The larger model had simply been starved of data.

Walking the allocation by hand

Here is the loop a team runs to size a compute-optimal model from scratch.

1. Decide the budget C in FLOPs — the most you can or want to spend.
2. Train a ladder of small models at several (N, D) splits that each use a slice of compute, and record the final loss of each.
3. Fit a surface to those points; the valley of the surface tells you the optimal N and D for any larger C.
4. Read off N and D at your real budget, then launch the one big run with confidence.

The Chinchilla loss surface fit to the ladder of runs: an irreducible term E plus terms that shrink with parameters N and tokens D — its valley gives the compute-optimal N and D.

When you should deliberately ignore the optimum

Chinchilla minimises training cost for a target loss. But a model that ships is run millions of times, so its inference cost matters too — and a smaller model is cheaper to run forever. That is why labs increasingly practise overtraining small models: deliberately feeding a smaller-than-optimal model far more tokens than Chinchilla recommends. You pay extra at training time to get a compact model that is cheap to serve. The popular open-weight models trained on trillions of tokens are doing exactly this.