Large Language Models
Mixture of Experts
Huge parameter count, small compute per token.
You can jump straight in, but this star assumes The Transformer. Not recommended as a first stop.
Four layers of depth
Each layer ends with a quiz. Finish layer 4 and you own this concept.
- L1IntuitionHow to get a model with a trillion parameters that costs about as much to run as one with a tenth of that — by only ever using a slice of it at a time.6m
40 XP - L2MechanicsThe router, top-k gating, why load balance is a training-stability problem, and what expert parallelism does to a compute cluster.9m
70 XP - L3CodeBuild a top-k router with a differentiable load-balancing loss in PyTorch, and watch what happens to expert utilization with and without it.11m
110 XP - L4FoundationsFLOPs accounting for sparse layers, the auxiliary loss objective in full, and the memory-bandwidth reality of running MoE at inference time.10m
180 XP
33 stars in the atlas.