The Transformer

Attention (previous star) answers one question: *which other tokens matter to me, and how should I blend their information in?* But attention alone, stacked, is just repeated weighted averaging — it needs a partner that can do genuine per-token computation, and needs plumbing that lets very deep stacks of both actually train. The transformer block is that complete, self-contained unit: attention, a small per-token neural network, and the wiring that makes stacking dozens of them possible.

The four ingredients

The MLP (sometimes called the feed-forward network) is deliberately simple: expand each token's vector to a wider dimension, apply a nonlinearity, then project back down. It has no notion of other tokens at all — it's the same function applied independently, in parallel, to every position. This is where most of a model's raw capacity to store and recombine facts actually lives; attention only decides what information *flows*, the MLP is where it gets *transformed*.

Residual connections: why depth doesn't break training

Instead of a sublayer's output replacing its input, the input is simply *added back*: output = input + sublayer(input). This residual, or 'skip', connection means gradients during backpropagation always have a direct, unobstructed path all the way back to the very first layer, alongside whatever path runs through all the intervening sublayers. Without residuals, stacking more than a handful of layers reliably makes training unstable or impossible — this single addition is what made genuinely deep networks trainable at all.

Normalization: keeping the notepad legible

As a residual stream accumulates contributions from dozens of blocks, its scale can drift wildly. Normalization (LayerNorm historically, RMSNorm in most current models) rescales activations back to a consistent range before each sublayer processes them, which keeps training numerically stable across very deep stacks.

Stacking: depth is just repetition

"GPT-4 has more layers than GPT-2" is a statement about depth, not about a fundamentally different architecture — the block itself has barely changed since 2018. Nearly all of the visible progress in frontier LLM architecture is scale (more layers, wider layers, more data, more compute) layered onto small, well-understood refinements (better normalization placement, better position encodings, better attention variants) rather than a wholesale redesign.