training tokens budget

Given a fixed pile of compute, you face a choice: build a bigger model, or feed a smaller model more data. The training tokens budget is the second half of that decision — how many tokens to train on. For a long time people built huge models and under-fed them; the Chinchilla result showed that for a given compute budget, model size and token count should grow together, and many models had been badly out of balance.

A useful rule of thumb from that work is roughly twenty training tokens per parameter to be compute-optimal — so a model with ten billion parameters wants on the order of two hundred billion tokens. The budget interacts directly with compute through the relation that training cost is about six times parameters times tokens, so spending the budget on more tokens means you can afford fewer parameters, and vice versa.

In practice teams often blow past the compute-optimal token count on purpose. A smaller model trained on far more tokens than Chinchilla suggests is slower to train but cheaper to run forever after — an excellent trade when the model will serve billions of queries.