long-context evaluation

Advertising a million-token context window is easy; using it faithfully is not. Long-context evaluation asks whether a model can actually find and reason over information buried deep in a very long input, rather than only attending to the start and end. The canonical probe is needle-in-a-haystack: hide a specific fact (the needle) at a controlled depth inside a long distractor text (the haystack) and test exact retrieval.

Varying needle depth and total length sweeps out a two-dimensional heatmap of retrieval accuracy, which typically exposes a lost-in-the-middle valley — accuracy dips for facts placed in the central region. Because single-needle retrieval is too easy for current models, harder suites raise the bar: multi-needle and multi-hop retrieval, aggregation over many positions, and reasoning benchmarks like RULER, LongBench, and infinity-Bench that require synthesizing dispersed evidence rather than copying one span.

Effective context length — the length at which performance stays reliable — is usually far shorter than the advertised maximum, and synthetic needle tests overstate real capability because a flagged out-of-place sentence is unnaturally easy to spot. Genuine long-document reasoning, where the relevant evidence blends into the surrounding text, remains substantially harder than the headline window suggests.