Attention

The mechanism that lets every token look at every other token and decide what matters — the core idea underneath every transformer.

Each row is a token deciding how much to 'look at' every other token.

Before transformers, sequence models (RNNs) processed text one token at a time, squeezing everything seen so far into a single fixed-size memory vector — a bottleneck that made long-range dependencies fade fast. Attention replaces that bottleneck with a direct connection: every token can look at every other token in the sequence and decide, freshly, how relevant each one is *for this specific comparison*.

Queries, Keys, and Values

Every token produces three different vectors from its own embedding: a Query (what am I looking for?), a Key (what do I offer, as something others might look for?), and a Value (the actual content I'll contribute if someone attends to me). To compute a token's new representation, its Query is compared against every token's Key; the resulting scores become weights over every token's Value.

Query
"what am I looking for?"
compare vs Keys
score every other token
softmax
turn scores into weights (sum to 1)
weighted sum of Values
blend in proportion to relevance

One token's attention output, end to end.

Take the sentence "The animal didn't cross the street because *it* was too tired." To resolve what "it" refers to, the model needs "it"'s Query to strongly match "animal"'s Key (and weakly match "street"'s Key). That match is learned entirely from data — nobody hand-coded a coreference rule; the Query/Key vectors were shaped by training so that this kind of match happens to work.

Multiple heads, multiple perspectives

One attention computation can only capture one *kind* of relevance at a time. Multi-head attention runs several attention operations in parallel, each with its own learned Q/K/V projections, so different heads can specialize: one head might track subject-verb agreement, another might learn to copy the most recent occurrence of the current word, another might attend to punctuation boundaries. Their outputs are concatenated and mixed together at the end.

Head 1

tracks nearby syntax

Head 2

copies repeated tokens

Head 3

attends to sentence start

Head 4

tracks punctuation

Real heads in real models specialize like this, discovered by training, not designed by hand.

Causal masking: no peeking at the future

A language model predicts the next token, so at training time it must not be allowed to attend to tokens that come *after* the one it's predicting — that would be cheating, like being handed the answer. Causal masking enforces this by zeroing out (setting to $- \infty$ before the softmax) every attention score from a token to any position after it, guaranteeing token $i$ 's prediction only ever depends on tokens $1 \dots i$ .

The cost that scales badly: $O(n^2)$

Every token computes a score against every other token, so a sequence of length $n$ requires on the order of $n^{2}$ score computations. Double the context length and attention's cost roughly quadruples — this quadratic wall is the single biggest reason long-context models are hard and expensive, and it's the motivation behind an entire sub-field (sparse attention, linear attention, FlashAttention) aimed at taming it.