reasoning vs. memorization

When a model nails a hard problem, there are two very different explanations, and telling them apart is one of the central puzzles of the field. Did it reason, working from principles to an answer it could reach on a genuinely new case? Or did it memorize, having seen that exact problem, or a near-twin, somewhere in its enormous training data and simply recalling the answer? From the outside both look like success, but only the first would survive a fresh variation it had never encountered.

This matters because benchmarks can be quietly contaminated: if test questions leaked into the training set, high scores measure recall, not reasoning, and overstate what the model can really do. Researchers probe the difference by perturbing problems, changing the numbers, renaming the characters, altering the surface while keeping the logic, and watching whether performance holds or collapses. A model that only memorized stumbles on the twist; one that reasoned sails through. In reality models do both at once, leaning on recall where they can and reasoning where they must, which is exactly why clean evaluation is so hard.