overtraining small models

Chinchilla says a small model should stop reading once it has had its roughly 20 tokens per parameter — past that, you are wasting training compute. But training is paid once; serving is paid forever. If your model will answer billions of queries, it can be a brilliant bargain to keep cramming a small model with far more tokens than optimal, because every future query then runs on a cheaper, smaller network.

Deliberately over-training pushes a model far past its Chinchilla point — sometimes hundreds of tokens per parameter. The training loss improves only slowly in that regime, well past diminishing returns, but the payoff is a compact model whose inference cost is fixed and low. Modern small models that punch above their weight are typically over-trained on trillions of tokens for exactly this reason: shifting cost from every-inference to once-only training.

It is the rational answer once you count lifetime cost, not training cost — the clearest case where compute-optimal training is the wrong target. The limit is data: over-training devours tokens, and high-quality text can run short.