Training compute is approximated as FLOPs (2 for the forward pass per parameter per token, doubled again for backward, i.e. roughly 6 total — a standard estimate for dense transformers, ignoring attention's quadratic term which is usually small relative to the linear projections at typical context lengths). The compute-optimal problem is:
Layer 4 · Foundations
Scaling Laws
The full parametric loss surface, the Lagrangian derivation of the compute-optimal split, and why loss extrapolation has hard limits.
10 min read180 XP