long-context benchmarks

These benchmarks test whether a model that claims to read a hundred thousand words can actually use all of them, not just the start and end. The simplest probe is needle-in-a-haystack: hide one specific fact somewhere inside a huge document and ask the model to retrieve it. Sweep the needle to every position and every input length, and you get a heat-map of where the model truly attends versus where it quietly forgets.

Harder versions go beyond single-fact lookup: multiple needles that must be combined, questions that require reasoning across distant parts of the text, or tracking a variable that changes throughout a long story. These reveal that a large context window on the spec sheet is not the same as effective use of it.

The recurring finding is a lost-in-the-middle pattern: models reliably recall information at the very beginning and very end of a long input but grow unreliable in the middle, and accuracy decays as inputs lengthen. So long-context benchmarks measure not just whether a model accepts a long prompt, but how gracefully its real comprehension holds up across the whole span.