The single most consequential empirical result in modern deep learning is not an architecture — it's a curve. Plot a transformer's test loss against the amount of compute spent training it, put both axes on a log scale, and you get an almost perfectly straight line, stretching across seven orders of magnitude. That line is a power law, and once you trust it, you can predict the loss of a model you haven't built yet.
Three knobs, one law each
- N — parameters. More weights, more capacity to represent patterns.
- D — training tokens. More data, less overfitting to any one example.
- C — compute. Roughly floating point operations for one training run, so N and D trade off against a fixed compute budget.
Holding one of N or D fixed and growing the other, loss falls smoothly and predictably — until the other variable becomes the bottleneck and the curve flattens. The interesting question, and the one that split the field into two camps, is: for a *fixed* compute budget, what's the best split between N and D?
Kaplan et al. (2020, OpenAI)
- Bigger models are dramatically more sample-efficient
- Prescription: scale parameters fast, data slower
- GPT-3 (175B params, ~300B tokens) followed this recipe
- Optimizer/LR schedule details later shown to bias the fit
Hoffmann et al. (2022, DeepMind, 'Chinchilla')
- Re-ran the experiment with fairly tuned LR schedules per run
- Prescription: scale parameters and data ~equally
- Chinchilla (70B params, 1.4T tokens) beat Gopher (280B, 300B)
- Became the industry-standard 'compute-optimal' rule of thumb
Emergent abilities: real phase transition or measurement artifact?
Some capabilities — multi-step arithmetic, certain reasoning benchmarks — appear to switch on abruptly at a scale threshold rather than improving gradually, and this was branded 'emergence'. A well-known 2023 critique argued that much of this apparent discontinuity is an artifact of using discontinuous metrics (exact-match accuracy) on an underlying quantity (per-token log-likelihood) that actually improves smoothly. Swap to a continuous metric and many 'emergent' jumps turn back into smooth curves — but not all of them; some genuine threshold effects survive the swap.
Train tiny models
cheap, many configs
Fit power law
loss vs N, D, C
Extrapolate
predict loss at target scale
Pick N, D split
compute-optimal or inference-aware
Spend the big budget
with much less risk