Training at Scale

Inference is memory-light: you need the weights and a KV cache. Training is memory-heavy: you need the weights, the gradients, and the optimizer's running statistics — for Adam, two extra copies of every parameter in fp32. That arithmetic alone rules out a single accelerator for any model bigger than a few billion parameters, long before you even ask how long training would take on one chip.

Distributed training is the set of techniques for splitting a training job — the data, the model, or both — across many accelerators that communicate over a network. There isn't one way to split it; there are four orthogonal axes, and real large-model training runs combine several of them at once.

The four axes, at a glance

• Data parallel (DP/DDP) — replicate the whole model on every GPU; split the batch. Simple, scales throughput, but every GPU must still fit the whole model.
• ZeRO / FSDP — data parallel, but shard the *optimizer states, gradients, and even weights* across GPUs instead of replicating them. Same math, dramatically less memory per GPU.
• Tensor parallel (TP) — split individual weight matrices across GPUs, so a single matmul runs as a set of smaller matmuls that exchange partial results. Needed when even one layer doesn't fit.
• Pipeline parallel (PP) — put different *layers* on different GPUs, and stream micro-batches through the resulting assembly line.

Why this isn't just 'buy more GPUs'

GPUs that never talk to each other don't help: if replica A finishes its gradient and replica B is still computing, A either waits or drifts out of sync. The entire discipline of distributed training is about designing the split so that (a) everything needed fits in memory and (b) the unavoidable communication overlaps with computation instead of blocking it. A cluster of 10,000 GPUs badly configured can train slower, per dollar, than 1,000 well-configured ones.

The rest of this star works through each axis mechanically, then implements a real FSDP training loop, then derives the memory and communication math that explains *why* the industry converged on this particular toolbox.