Layer 4 · Foundations

The Transformer

The full parameter and FLOP accounting for a dense transformer, and why arithmetic intensity makes autoregressive decoding fundamentally different from training.

13 min read180 XP

Fix a dense decoder-only transformer with $L$ layers, hidden size $d$ , vocabulary size $V$ , and (for this derivation) an MLP expansion ratio of 4. The full parameter count, ignoring the small normalization weight vectors:

N \approx embedding (tied w/ output) V d + L blocks L (attention 4 d^{2} + MLP 8 d^{2}) = V d + 12 L d^{2}