JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Reading and extrapolating scaling curves

How teams fit a power law from cheap runs and project a frontier model's loss before training it — and where that clean line bends or snaps.

Fitting the line from a handful of cheap runs

Extrapolation is the superpower scaling laws grant. Because a power-law loss curve is a straight line in log-log space, you only need a few points to pin it down. Train a series of small, affordable models — each a different size or token count — measure their loss, and fit the line through them. Scaling-curve extrapolation then extends that line out to sizes you have never trained, predicting the loss of a model far beyond your sample.

In log-log space the loss falls as a straight power law, so a handful of cheap runs pin down the line you extrapolate.

A log-log plot of loss versus model scale, with data points lying on a single descending straight line.

Forecasting a frontier run before it exists

This is not just theory. Modern frontier labs report predicting a flagship model's final loss from a ladder of runs thousands of times smaller, and hitting the prediction closely. That is what makes it sane to commit tens of millions of dollars to a single training run: the scaling law said, with quantified confidence, roughly where the model would land.

  1. Train a ladder of small models spanning a wide range of sizes or token counts.
  2. Fit the power law plus floor to their losses in log-log space.
  3. Extrapolate to the frontier size and read off the predicted loss — with an honest error bar.
  4. Launch the big run; compare the outcome to the forecast to recalibrate next time.
L(N) = L_{\infty} + \left(\dfrac{N_c}{N}\right)^{\alpha}

Fit both halves: a falling power-law term riding on the flat irreducible-loss floor L∞, then read off the loss at the frontier size N.

Where the clean line bends or breaks

A single straight line is a beautiful idealisation, but reality sometimes has joints. Broken scaling laws describe curves that follow one power law for a while, then bend to a different slope — sometimes improving faster, sometimes stalling, sometimes a skill that was flat suddenly catching fire. A naive line fit through one regime can badly mispredict the next.

Loss is the means; capability is the end

Even a perfect loss forecast leaves a gap, because nobody ships a low loss — they ship a model that does things. Translating predicted loss into predicted usefulness is the hard, still-open problem of downstream-versus-loss scaling. Loss extrapolation is reliable; turning that loss into a benchmark or product score is far shakier, which is why careful evaluation never disappears just because the loss curve looks tame.