data-constrained scaling

Every scaling story so far assumed text was effectively unlimited. It isn't. The supply of high-quality human-written text — books, code, careful web pages — is large but finite, and the biggest models are already trained on a sizeable fraction of it. Data-constrained scaling asks what happens when you want to keep growing but the world is running out of fresh, good tokens to feed you.

One option is to repeat data: studies find that re-using tokens for a few epochs is nearly as good as fresh ones, but the value of each extra pass decays, and after many repeats added compute buys almost nothing. Other levers squeeze more from less — aggressive filtering and deduplication so every token counts, mixing in code or other modalities, or generating synthetic data with care. Each pushes the wall back rather than removing it, and synthetic data risks feeding the model its own distribution.

This reframes the whole field's trajectory: compute and parameters can keep growing, but if unique high-quality text is the true bottleneck, future gains hinge on data efficiency and new sources, not just bigger clusters.