pretraining compute

Pretraining compute is the total amount of arithmetic spent to train a model, usually counted in floating-point operations (FLOPs). It is the dominant cost of building an LLM — frontier runs reach tens of millions of dollars and require thousands of GPUs working in concert for weeks or months. Compute is the budget around which almost every other decision is planned.

There is a strikingly simple estimate: training a dense transformer costs about six FLOPs per parameter per token — roughly two for the forward pass and four for the backward pass. So a model's total training compute is close to six times its parameter count times the number of training tokens. This little formula lets teams price a run in advance and decide how to split a fixed budget between a bigger model and more data.

Compute is the lever behind scaling laws: across many orders of magnitude, loss falls predictably as compute grows. But the relationship is one of diminishing returns — each further drop in loss costs exponentially more — and raw FLOPs say nothing about whether they were spent efficiently or wasted on poor data and instability.