prompt sensitivity

Prompt sensitivity is the unsettling fact that small, seemingly harmless changes to a prompt can noticeably swing the model's output. Rewording a question, reordering a list of options, adding or dropping a space or a line break, changing whether the examples come before or after the instruction, any of these can move the answer or its quality. Two prompts that mean the same thing to a human can land differently for the model.

This happens because the model responds to the exact sequence of tokens, not to an abstract idea of your meaning. Wording and structure shift which continuation looks most likely, and there is no built-in guarantee that meaning-preserving edits preserve behaviour. It is a real reliability concern: a prompt tuned to work beautifully on your examples may quietly degrade when the phrasing or the inputs change a little.

The practical response is to not trust a single lucky prompt. Test prompts across many inputs and a few rewordings, prefer clear and explicit phrasing that leaves less room for chance, and where accuracy matters use techniques like self-consistency that average over variation. Honest evaluation reports performance across prompt variants, not just the best one you happened to find.