long-context architectures

A model's context window is how many tokens it can read at once, and pushing it from a few thousand to hundreds of thousands runs into two walls: attention cost that grows with the square of the length, and position encodings that were never trained on such distances. Long-context architecture is the bag of tricks that gets past both, so a model can ingest a whole book or codebase in one pass.

On the position side, the most common move is scaling RoPE's frequencies so the same rotations cover a far wider range, sometimes followed by a short fine-tune on long examples. On the cost side, designers reach for sliding-window or other sparse attention, grouped-query attention and KV-cache compression to fit the memory, and serving tricks like paged and ring attention. The honest caveat is that a large window does not guarantee good use of it; models often retrieve well near the ends but lose detail in the middle.