Context Window
Context Window
The context window is the maximum number of tokens a model can process in a single forward pass. It limits how much input text the model can "see" when generating output.
Importance
- Long documents — Context window determines if full documents fit in one pass
- Conversations — Longer context = longer chat history
- Code — Large codebases require big context windows
- RAG — More retrieved documents can be included in context
- Agentic workflows — Agents with long context handle complex multi-step tasks
Context Window Sizes
| Model | Context Window |
|---|---|
| Gemini 2 Pro | 1,048,576 (1M) |
| Claude 4 | 200K-500K |
| GPT-4o | 128K |
| DeepSeek V3 | 128K |
| Llama 4 | 128K-256K |
| Mistral Large | 128K |
| qwen-2.5 | 128K |
Challenges
Computational Cost
Attention computation scales O(N²) with sequence length (quadratic), making long contexts expensive.
Memory Cost
KV cache scales O(N) linearly with context length, consuming significant GPU memory.
"Lost in the Middle"
Models tend to perform worse on information in the middle of long contexts — they recall beginning and end better.
Optimization Techniques
- Flash Attention — IO-aware O(N²) attention that's 2-4x faster
- Ring Attention — Distributes attention across GPUs for near-infinite context
- Sliding window attention — Only attends within a local window
- Sparse attention — Selectively attends to subsets
- YaRN / NTK-aware — Extends context via RoPE interpolation
Related
- KV Cache — Memory cost of long contexts
- Attention Mechanism — The O(N²) bottleneck
- Transformer Architecture — The architecture with quadratic attention
- See context-window-comparison for side-by-side comparisons