Attention · L2 Mechanics

The exact scaled dot-product formula, why the scaling factor exists, and how masking and multi-head splitting actually work mechanically.

X
input embeddings, (n, d)
Q, K, V
three learned linear projections
QKᵀ
raw similarity scores, (n, n)
÷√d_k, mask, softmax
turn scores into a weight distribution
× V
weighted sum of values

Scaled dot-product attention, mechanically, step by step.

Given an input matrix $X$ of shape $(n, d)$ — $n$ tokens, each a $d$ -dimensional embedding — three learned weight matrices $W^{Q}, W^{K}, W^{V}$ project it into Queries, Keys, and Values: $Q = X W^{Q}$ , $K = X W^{K}$ , $V = X W^{V}$ . Attention is then a single formula that turns those three matrices into an output the same shape as $Q$ .