Data parallel, mechanically
Each GPU: forward+backward
on its own micro-batch
All-reduce gradients
average across GPUs
Each GPU: optimizer step
identical update, independently
Layer 2 · Mechanics
Training at Scale
ZeRO's three stages, tensor and pipeline parallelism mechanically, collectives, and how 3D parallelism combines all of it.
11 min read70 XP