Brittle by Design: Prompts, Context & Hard Limits

The same question, two different answers

Type a question one way and get a good answer; rephrase it trivially — add 'please', reorder two clauses, change a synonym — and the answer can change in quality or even flip. This is prompt brittleness, and its close cousin prompt sensitivity: the model's output depends on surface features of the wording that should be irrelevant. A robust reasoner would not care about politeness or word order; a pattern-matcher does, because different phrasings nudge it toward different regions of the text it learned from.

There is only so much room

Everything a model considers at once — your instructions, the conversation so far, any documents you pasted, and the answer it is writing — has to fit inside its context window, measured in tokens (chunks of text, roughly a few characters each). When the conversation grows past that budget you hit context length limits: the oldest turns are dropped or truncated, and the model genuinely cannot 'see' what scrolled off. It does not warn you; it just quietly loses the early material and answers as if it were never there.

n_{\text{prompt}} + n_{\text{history}} + n_{\text{docs}} + n_{\text{output}} \le N_{\text{ctx}}

Everything shares one budget: prompt, conversation history, pasted documents, and the answer all count toward the context window.

Modern models advertise very large windows, which helps — but a bigger window is not free. More context costs more money and time per call, and as we will see, the model does not use a long context evenly. 'It fits' is not the same as 'it was used well'.

Lost in the middle

A striking, well-documented quirk is the lost-in-the-middle effect: when key information sits in the middle of a long context, the model is markedly worse at using it than when the same fact sits near the beginning or the end. Attention favours the edges. So you can paste a long report containing exactly the answer, and the model still misses it — not because the fact was absent, but because it was buried in the low-attention zone.

Put the most important instructions and facts at the start or end of your prompt, not buried in the middle.
Do not assume a model 'read' a long document just because it fit in the window; ask targeted questions and check.
When precision matters, retrieve and paste only the relevant passages instead of dumping the whole corpus.

How limits compound the failures

These limits do not stay in their lane — they amplify the earlier failures. A fact that scrolled out of the window forces the model back onto memory, where hallucination lives. A long, cluttered context makes reasoning more likely to drift. And through all of it the model keeps its confident tone, so a truncated or middle-buried context produces an answer that is missing information yet sounds complete. The limits are quiet, and that quiet is the hazard.