The block from Layer 1 has one crucial design choice left unspecified: exactly *where* normalization sits relative to the residual connection. This single placement decision determines whether a 96-layer model trains at all.
Post-LN (original 2017 Transformer)
- x ← Norm(x + Sublayer(x))
- Normalizes the residual stream itself
- Deeper stacks are unstable to train
- Needs careful learning-rate warmup
Pre-LN (GPT-2 onward, current default)
- x ← x + Sublayer(Norm(x))
- Residual stream itself is never normalized
- Gradients flow cleanly through many layers
- Trains stably even at 100+ layers