compute-optimal training

Imagine a fixed pot of compute — say, a month on a cluster — and one question: how do I spend it to get the lowest possible loss? You can pour it into a bigger model, or into more training steps over more data, but not both without limit. Compute-optimal training is the recipe for the split that wrings the most capability out of that fixed pot.

Training compute is roughly six times parameters times tokens, so a budget C traces a curve of trade-offs: every choice of model size N forces a token count D. Fit a scaling law over small runs, then find the N and D on that budget line that sit at the bottom of the loss surface. Chinchilla's result is the canonical answer — grow N and D together — but the optimal split shifts with architecture, data quality, and the exponents you measure.

Compute-optimal means cheapest to train, not cheapest to use. The whole framing ignores inference, which is why deployment teams often choose a deliberately sub-optimal, smaller model.