compute-budget allocation

You rarely get to choose data and model size freely — you get a compute budget, a fixed number of floating-point operations the hardware can afford. Compute-budget allocation is the act of dividing that single number between a bigger model and more training tokens. Spend it all on size and you starve the model of reading; spend it all on data and the network is too small to absorb it.

Because training compute is about six times N times D, fixing the budget C pins you to a curve: pick N and D is forced, or vice versa. The scaling law tells you the loss at each point on that curve, and the lowest point is the compute-optimal split — Chinchilla's equal-growth answer. Shift the data's quality or the exponents and the sweet spot slides, so each lab re-fits the laws for its own setup before committing a flagship run.

This optimises the training bill alone. Factor in millions of future inferences and you may rationally pick a smaller, longer-trained model that sits off the training-optimal curve but wins on total lifetime cost.