Self-attention computes a weighted average over value vectors, and that average is a set operation — it looks at every token, scores it against every other token, and sums. Shuffle the input tokens and shuffle the output the same way, and attention produces the identical result. Formally, attention is permutation-equivariant: it has no built-in concept of 'first', 'second', or 'next to'.
Compare this to an RNN, which reads tokens one at a time in a fixed left-to-right order — position is implicit in the recurrence itself, baked into the architecture for free. Transformers threw away recurrence to get parallelism, and positional encoding is the price of admission: some explicit signal has to be injected so the model can tell tokens apart by location, not just by content.
The fix, at the highest level
Token embedding
"dog" → vector, same everywhere
+ position signal
vector that encodes "I'm token #3"
Combined vector
now carries identity AND location
Into attention
Q/K/V can now distinguish position
There isn't one right way to build that fingerprint — there are four major families, and the field has cycled through most of them in under a decade. Each answers a different version of the same question: how do you inject 'where am I' without breaking 'what am I', and without falling apart when the model is asked to handle longer sequences than it ever saw in training?
- Sinusoidal (absolute) — original Transformer paper. Fixed, hand-designed waves added to the embedding. No learned parameters.
- Learned absolute — a lookup table of position vectors, trained like any other embedding (GPT-2, early BERT). Simple, but breaks past the trained length.
- ALiBi — no positional vector at all. Instead, attention scores are penalized by a distance-proportional bias, baked directly into the softmax.
- RoPE (Rotary) — rotates the query and key vectors by an angle proportional to position. The dominant choice in modern open LLMs (LLaMA, Mistral, Qwen).
Why anyone bothers with four different schemes
The practical pressure that drove this evolution is context length. A model trained on 2k-token sequences that must later serve 32k or 128k tokens needs a positional scheme that either generalizes gracefully or can be deliberately stretched. Sinusoidal and learned absolute encodings degrade sharply past their training length; RoPE and ALiBi were specifically designed (or later patched) to extrapolate, which is why virtually every long-context model released since 2023 uses one of them.