power-law loss curve

Plot a model's loss against how big it is, and on ordinary axes you see a curve that plunges fast then flattens into a long, discouraging tail. Switch both axes to logarithmic and something striking happens: that messy curve straightens into a clean line. A straight line on log-log paper is the signature of a power law — the relationship behind almost every scaling result in this field.

A power law means loss falls like one over the resource raised to some exponent: L is proportional to N to the minus alpha. Taking logs of both sides turns it into log L equals a constant minus alpha times log N — the equation of a line whose slope is the exponent. That slope is the prize: measure it from a handful of cheap runs and you can read off how much loss each tenfold increase in size, data, or compute will buy, far past where you can afford to test.

The line is rarely perfectly straight forever. It can bend near an irreducible floor, or kink when a new ability switches on — reminders that the tidy law is a local approximation, not a law of nature.