Scaling Laws · L2 Mechanics

The machinery behind the curve: the parametric loss form, why Chinchilla's fit differs from Kaplan's, and how the compute-optimal allocation falls out of it.

Fix a compute budget C
Train many (N, D) pairs at that C
Read off the lowest-loss pair
Repeat across many budgets
Fit N*(C), D*(C) as power laws

This is literally the Chinchilla experiment: an IsoFLOP sweep, repeated at several compute levels.

The Chinchilla paper's core method is called an IsoFLOP profile. Pick a compute budget C. Train a whole family of models at that *same* C, varying how many parameters vs. how many tokens each one gets (bigger model, fewer steps; smaller model, more steps — same total FLOPs either way). Plot loss against N for that fixed C: it's a parabola-ish curve with a minimum. That minimum tells you the best N (and hence D, since C fixes their product) for that budget. Repeat at several budgets, and the minima themselves trace out two more power laws: $N^{*} (C) \propto C^{a}$ and $D^{*} (C) \propto C^{b}$ .