Three knobs, one budget
A pretraining run has three big dials: how many parameters the model has, how many tokens it trains on, and how much compute you are willing to spend. They are not independent — compute is roughly parameters × tokens — so once you fix a budget, choosing two of them fixes the third. The whole art of planning a run is spending a fixed budget wisely.
Let's define the budget precisely. Pretraining compute is measured in floating-point operations, or FLOPs, and there is a famous back-of-the-envelope rule: training a dense transformer costs about 6 × N × D FLOPs, where N is the parameter count and D is the number of training tokens. The factor of 6 bundles the forward and backward passes together. That tiny formula governs multi-hundred-million-dollar decisions.
compute (FLOPs) ~= 6 * N * D N = 7e9 params, D = 2e12 tokens compute ~= 6 * 7e9 * 2e12 = 8.4e22 FLOPs # on ~1000 GPUs at realistic efficiency, that's weeks of wall-clock time
The 6ND rule: training compute is roughly six FLOPs per parameter per token.
Scaling laws: bigger is predictably better
The reason any of this is plannable is neural scaling laws: across many orders of magnitude, the model's loss falls as a smooth power law in parameters, data, and compute. Plot loss against compute on log-log axes and you get a startlingly straight line. That predictability is what lets a team train a few small models, fit the curve, and forecast the loss of a giant model before spending a cent on it.
Log-log plot of model loss decreasing along a straight power-law line as scale grows.
Scaling laws also explain emergent abilities — skills like multi-step arithmetic that are absent in small models and appear, sometimes abruptly, past a size threshold. Whether that abruptness is real or an artifact of how we measure is debated, but the engineering takeaway is solid: scale buys capability, and the curve tells you roughly how much.
The Chinchilla lesson: feed the model more tokens
Early giant models were under-fed: enormous parameter counts trained on too few tokens. The Chinchilla finding corrected this. For a fixed compute budget, loss is minimized when parameters and tokens are scaled together — roughly in proportion. The headline guideline that came out of it: aim for about 20 training tokens per parameter. A 7-billion-parameter model wants on the order of 140 billion tokens just to be compute-optimal.
Chinchilla's loss model: an irreducible floor plus separate parameter and data terms, minimized at one best (N, D) pair for a fixed compute budget.
This is the idea of compute-optimal training: given your FLOP budget, there is a single best (N, D) pair, and overshooting either knob wastes money. But there is a crucial twist for real products — see the callout.
Spending the token budget across sources
Once you know your training-token budget — say 2 trillion tokens — you must decide how to spend it across the clean source piles from the last guide. That is data mixture weighting again, now with numbers attached. You set a percentage for each source, and those percentages directly shape the model's personality: more code makes a better coder, more multilingual text makes a better translator, more math makes a better reasoner.
data mixture (share of 2T total tokens): web (cleaned) 60% 1.20T code 18% 0.36T books & papers 12% 0.24T curated Q&A / wiki 7% 0.14T math 3% 0.06T
Some piles are small and precious (high-quality math, rare languages). You might repeat them a few times — a controlled exception to the no-duplicates rule — but only a little, since repeating too often re-introduces the memorization problem dedup was meant to fix. Picking the mixture is itself an experiment: train small proxy models on candidate mixtures and keep the one that scores best on the skills you care about.
Putting a plan on paper
Here is how a team actually sequences these decisions before launching a run:
- Fix the compute budget from your GPU fleet and deadline — that is the hard constraint.
- Fit scaling-law curves from a handful of small training runs to predict loss at scale.
- Choose N and D: start from Chinchilla-optimal, then bias toward a smaller N if inference cost matters.
- Design the data mixture by training small proxies on candidate recipes.
- Forecast cost and time, get sign-off, and only then commit the full cluster.
With the budget, the data, and the mixture decided, the model knows what to learn and how much. The next two guides cover how the learning actually happens: the optimizer and schedule that move the weights, and the distributed machinery that runs the whole thing across thousands of chips.