The Model Landscape · L4 Foundations

Scaling laws, compute-optimal training, and the arithmetic behind 'how big should this model be'.

Loss falls predictably as a power law in compute, parameters, and data — until it doesn't.

Empirically, pretraining loss $L$ follows a power law in model size $N$ (parameters), dataset size $D$ (tokens), and compute $C$ , roughly $C \approx 6 N D$ FLOPs for a dense transformer trained with standard forward+backward passes. Scaling laws (Kaplan et al. 2020, then refined by Hoffmann et al. 2022 — 'Chinchilla') quantify how loss decreases as you scale these up.