emergent abilities

Some skills seem to arrive all at once. A model with a billion parameters cannot do multi-step arithmetic or follow a tricky instruction at all — it scores near zero, run after run. Scale it up past some threshold and, almost abruptly, it can. These sudden, hard-to-predict capabilities that small models lack and large ones possess are called emergent abilities: the loss curve slid smoothly, but the skill seemed to switch on.

Classic examples include doing three-digit multiplication, answering with chain-of-thought reasoning, and using in-context examples to learn a new task on the fly. On benchmarks plotted against scale, the score hugs the floor across many model sizes and then shoots up in a narrow band. The phenomenon is real in the sense that the capability genuinely was absent and then present; what is debated is whether the sharpness of the jump is intrinsic or an artefact of how we measure.

Emergence makes scaling exciting and unnerving: it means new abilities — useful or hazardous — can appear in a bigger model that no one trained for or anticipated.