scaling-curve extrapolation

Training the flagship model can cost millions and weeks of cluster time — you cannot afford to guess and find out. So labs do something closer to engineering than alchemy: they train a ladder of small, cheap models, measure how loss falls, and extend the line to forecast the giant they have not built yet. That forecast is scaling-curve extrapolation, and it is how a team commits hundreds of GPUs with confidence.

Fit a power law — or the fuller form with an irreducible floor — to several small runs spanning a range of sizes, then evaluate it at the target scale. Done well, the predicted loss of a model a hundred times larger lands remarkably close to reality. Getting it right needs care: consistent data and tuning across the ladder, enough spread in sizes to pin the slope, and ideally hyperparameters transferred so the big run behaves like a scaled-up sibling rather than a different beast.

Extrapolation is trustworthy for pretraining loss but slippery for downstream skills, which can jump or stall. The further past your data you reach, the more a hidden bend or a broken law can blindside the forecast.