WTF is Context Window?
The maximum number of tokens a model can take into account in one request, prompt and response included.
Also called: context length, context limit
TL;DR
Think of it as the model's working memory for a single call. Everything the model should consider (instructions, documents, chat history, tool results) has to fit inside it. Windows have grown from a few thousand tokens to a million or more, but bigger doesn't mean the model uses all of it equally well.
How it works
The limit covers the input plus generated output. Cost and latency scale with how much of it you fill, and quality can degrade as it fills (context rot), so production agents manage it actively: retrieval instead of pasting whole corpora, compaction of old turns, subagents that keep exploration out of the main context, and prompt caching for stable prefixes.
Origin
- Coined / introduced by
- Transformer architecture
- Story
- A consequence of models attending over a fixed maximum sequence length.