This layer derives the one property that made RoPE the default choice: that after rotation, the attention score is provably a function of relative position alone.
Layer 4 · Foundations
Positional Encoding
The rotation matrix behind RoPE, why it produces a genuinely relative dot product, and the arithmetic of extending context.
14 min read180 XP