Large Language Models
Pretraining
Next-token prediction over a large slice of the internet.
You can jump straight in, but this star assumes The Transformer. Not recommended as a first stop.
Four layers of depth
Each layer ends with a quiz. Finish layer 4 and you own this concept.
- L1IntuitionThe single objective — predict the next token — that, scaled up enough, produces everything an LLM can do.7m
40 XP - L2MechanicsThe training loop mechanically: loss, batching, learning rate schedules, checkpointing, and how a run is actually managed for months.11m
70 XP - L3CodeA real (small-scale but structurally complete) pretraining loop: AdamW, warmup+cosine schedule, gradient accumulation, checkpointing.15m
110 XP - L4FoundationsCross-entropy and perplexity derived properly, the Chinchilla compute-optimal arithmetic, and where the FLOPs actually go on real hardware.15m
180 XP
Where this leads
33 stars in the atlas.