JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Beyond pretraining: inference scaling and the data wall

As fresh text runs short and pretraining returns flatten, two new frontiers open: reusing data carefully, and spending compute at answer time instead of training time.

Running into the data wall

Chinchilla scaling wants more tokens as models grow — but the supply of high-quality human text is finite. As frontier runs reach for trillions of tokens, teams increasingly bump into a data wall: there simply is not enough fresh, clean text to stay compute-optimal. The naive fix, training for more epochs over the same data, risks memorisation rather than learning, especially after deduplication strips the easy redundancy out.

Scaling when data, not compute, is the bottleneck

This reframes the whole problem as data-constrained scaling: how to make the most of a fixed pool of tokens. Careful studies show repetition is not worthless — reusing data for a handful of epochs recovers most of the value of fresh tokens, but the benefit decays fast, and after several passes extra epochs add almost nothing. So when you cannot get more data, the smart moves are to spend that scarce data wisely, add compute in other forms, and lean on overtraining only as far as it still pays.

L(N,D)=E+\dfrac{A}{N^{\alpha}}+\dfrac{B}{D^{\beta}}

The Chinchilla loss law: with the token pool D fixed, only the parameter term keeps shrinking — so data, not parameters, becomes the binding constraint.

Architecture offers another lever. A mixture-of-experts model holds a huge parameter count but activates only a slice per token, buying more capacity per unit of compute — a partial answer to scaling when both data and serving budgets bite.

Spend the compute at answer time instead

If you cannot scale training further, scale thinking. Inference-compute scaling lets a fixed model get better answers by spending more compute when it runs — generating a long chain of thought, sampling many candidate solutions and selecting the best, or revising its own draft. Test-time compute has its own scaling curve: accuracy on hard problems rises smoothly as you let the model deliberate longer, and reasoning models are built specifically to ride that curve.

Inference-compute scaling spends more at answer time — generating longer chains of tokens before committing to a final answer.

An autoregressive loop feeding each generated token back as input to produce the next.

Two budgets, one decision

Now you hold two dials, and scaling inference versus training asks how to set both. Pour everything into a bigger pretrained model and every query is cheap but the training bill is colossal. Keep the model modest and let it think longer per query, and training is cheap but each answer costs more. The right mix depends on how often the model will be queried and how hard the questions are.

  1. For a rarely-asked but extremely hard question, lean on inference: let a modest model deliberate at length.
  2. For a high-volume, easy-to-medium workload, lean on training: a stronger base model makes every cheap call better.
  3. In practice, blend both — a well-trained model that also knows when to think harder.