prompt brittleness

You can ask the same question two ways that mean exactly the same thing to a human and get two very different answers from a model. Reorder a few words, add a polite please, change a bullet list to a sentence, or swap one example, and the quality can swing from excellent to useless. The model is sensitive to surface form in ways our intuitions do not predict, because tiny wording changes shift the statistical patterns it keys on.

This brittleness is awkward for two reasons. It makes results hard to reproduce — a prompt that worked last week may falter after a model update — and it makes evaluation tricky, since a benchmark score can hinge on phrasing rather than ability. The practical response is to test prompts on several inputs, prefer clear and explicit instructions over clever ones, and treat a single lucky prompt as a starting point rather than a finished solution. Robustness comes from systematic testing, not from one magic incantation.