Context Engineering

Self-attention computes a score between every pair of tokens, so both compute and memory for the attention matrix scale as $O(n^2)$ in sequence length $n$ (for a fixed model width). Doubling the context roughly quadruples the attention cost — which is why simply 'stuffing more context in' has a real, nonlinear cost, not just a linear one, and why efficient-attention and KV-cache techniques exist as separate engineering efforts.