Composing skills into new ones
The dream of reasoning is reuse: master two skills, then combine them on a problem that needs both at once. That is compositional generalization — handling novel combinations of familiar parts, like applying a unit conversion inside a multi-step word problem the model never saw in that exact shape. When it works, the model feels genuinely intelligent; when it fails, you see a model that aced each piece in isolation but cannot bolt them together.
Compositional generalization is also the clearest test of whether an emergent skill is real understanding or a memorized shortcut. Memorization handles the cases it saw; composition handles the cases nobody could have pre-stored.
Three small plots: an underfit model, a well-fit model that generalizes, and an overfit model that memorizes the training points.
Reasoning about other minds
A subtle, much-debated capability is theory of mind: reasoning about what another agent knows, believes, or intends — including false beliefs, as in 'Sally thinks the ball is in the basket, but it was secretly moved.' Strong models often pass such tests, which matters for assistants that must track what a user already knows or what a character in a story would expect.
Reasoning or recall? The contamination trap
Every capability claim runs into the reasoning-versus-memorization question, and the practical danger is benchmark contamination: if a test's questions and answers leaked into the training data, a high score measures recall, not reasoning. A model that 'solves' a famous puzzle set may simply have read the solutions. This silently inflates scores and is the single most common way people overestimate a model's reasoning.
Good reasoning benchmarks fight back: held-out or freshly written problems, private test sets, and tasks generated from templates so each instance is new. The strongest evidence of reasoning is performance on problems that provably could not have been memorized.
Data divided into separate train, validation, and test portions.
Where it breaks
An honest map of capability includes its edges. Common reasoning failures include: confidently wrong arithmetic, plausible-looking proofs with an invalid step, brittleness to surface rewording, losing track over long contexts, and 'shortcut' answers that pattern-match the question's look instead of its logic. None of these mean the reasoning is fake — they mean it is uneven, and the unevenness is predictable enough to plan around.
- Probe with rewordings and fresh numbers, not just the canonical phrasing, to separate reasoning from recall.
- Force visible working on hard items so failures are diagnosable rather than hidden.
- Verify final answers with an independent check — run the code, recompute, or cross-question.
- Prefer held-out or private tasks; treat any public benchmark score as a possibly-contaminated ceiling.