scaling inference vs. training

You have a budget and one goal: better answers. Two very different levers can deliver them. Lever one, scale training — build a bigger model or feed it more, so it is smarter the instant it replies. Lever two, scale inference — keep the model as is but let it think longer on each question. Choosing between them is one of the central engineering trade-offs of modern LLMs.

Training compute is spent once and amortised over every future query; inference compute is spent again on every single call. So the right lever depends on volume and difficulty. For a model serving enormous traffic, a smaller-but-thinks-longer setup is cheaper overall only up to a point — heavy per-query thinking eventually costs more than baking the ability in. For rare, very hard problems, letting a modest model deliberate is a bargain. Recent results show test-time compute can substitute for parameters, but not without limit, and the exchange rate shifts with task type.

It is not either-or. The frontier increasingly combines both — train a capable base, then scale its thinking at inference — and the optimal split is itself an open, actively studied scaling question.