neural scaling laws

Train a family of language models — tiny ones, medium ones, huge ones — and plot how well they predict text against the resources you spent. The points don't scatter randomly; they fall along a smooth, gently bending curve. Make the model ten times bigger, or feed it ten times more text, or spend ten times more compute, and the prediction error drops by a steady, predictable fraction. That regularity is a scaling law: a quantitative promise that more resources buy more capability in a way you can chart in advance.

Concretely, the test loss falls as a power law in three knobs — the parameter count N, the training tokens D, and the compute C — each contributing a term like A over N raised to some power. Power laws look like straight lines on log-log axes, so a few small runs let you fit the slope and forecast a giant run. The laws say nothing about which architecture you use; they hold remarkably across model sizes, which is why labs treat them as a planning tool rather than a curiosity.

The catch: a falling loss is not the same as a useful skill, and every law eventually meets a floor it cannot cross. Scaling laws guide the budget; they do not guarantee the model can reason.