Layer 4 · Foundations

Attention

The full derivation of the scaling term, attention's exact FLOP and memory cost, and why FlashAttention's arithmetic-intensity argument works on real hardware.

13 min read180 XP

Restate the attention equation precisely, with shapes tracked:

Q, K, V \in R^{n \times d_{k}}, Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V \in R^{n \times d_{k}}