context length

Context length is the maximum amount of text a model can hold in mind at once — its working memory, measured in tokens (roughly word-pieces). Everything inside this window is available to attention; everything that falls outside it might as well not exist. If a conversation, document, or codebase exceeds the context length, the oldest material gets dropped or must be summarized away. A model with a 4,000-token context is reading a few pages; one with a million-token context can, in principle, hold an entire book.

The limit exists for two linked reasons. First, the model was trained on sequences up to some length, and its sense of position can degrade beyond that. Second, standard attention compares every token to every other, so doubling the context roughly quadruples the attention work and memory — which makes very long contexts genuinely expensive, both to build and to run. Recent models stretch context dramatically using cheaper attention variants and better positional schemes, but every extension is a fight against that quadratic cost.

Here is the crucial honesty: a large advertised context length is not the same as reliable use of it. Models frequently suffer from a "lost in the middle" effect, paying sharp attention to the beginning and end of a long input while glossing over the middle, where a buried fact can be missed entirely. A 200,000-token window does not guarantee the model will actually find and use what sits at token 100,000. Always judge long-context claims by tested recall at depth, not by the headline number.