Chinchilla-optimal

Early big models were like a brilliant student who barely opened the textbook: enormous parameter counts trained on too little text. In 2022 DeepMind's Chinchilla experiment asked the cleaner question — given a fixed compute budget, what split of size versus reading maximises skill? The surprising answer was that most giant models were badly over-sized and under-fed. A smaller model trained on far more tokens beat them while costing the same to train.

Chinchilla swept hundreds of runs and found that, at the compute-optimal point, parameters and training tokens should grow in lockstep — roughly in equal proportion as the budget rises. A handy rule of thumb fell out: train on about twenty tokens for every parameter. A 70-billion-parameter model, by this recipe, wants on the order of 1.4 trillion tokens, not the few hundred billion its predecessors used.

It rewrote how labs spend money and made smaller, token-rich models the default. But the rule is optimal only for training cost; if a model will serve billions of queries, you may deliberately break it and over-train a smaller model to save on every inference.