A model's weights are typically trained and stored as 16-bit floating point numbers (fp16/bf16). Quantization converts them to a lower-precision format — most commonly 8-bit or 4-bit integers — cutting memory footprint by 2x or 4x and, because moving less data is usually the bottleneck (see Layer 4), often making inference meaningfully faster too. The surprising part is how *little* quality is typically lost: a well-quantized 4-bit 70B model usually beats an unquantized 13B model on most benchmarks, despite using less memory than the 13B one.
Why this works at all
Neural network weights are, for the most part, redundant and noise-tolerant — the network was trained with dropout, noisy gradients, and stochastic optimization, so it already learned to be somewhat robust to small perturbations in its own parameters. Quantization exploits exactly this: a small, structured rounding error introduced into every weight is, in aggregate, similar in character to the noise the model already tolerated during training.
fp16 (16 bits/weight)
- Full training precision, no quality loss
- 70B model needs ~140GB of memory
- Requires multiple high-end GPUs
- The default output of a training run
int4 (4 bits/weight)
- Small, usually near-imperceptible quality drop
- 70B model needs ~35GB of memory
- Fits on a single consumer GPU
- Requires a separate quantization step, not free
Two very different ways to get there
- Post-training quantization (PTQ) — take an already-trained fp16 model and convert its weights afterward, with no further training. Cheap, fast (minutes to hours), the overwhelmingly common approach for open-weight models you download and run yourself.
- Quantization-aware training (QAT) — simulate quantization *during* training or fine-tuning, so the model's own gradients adapt around the rounding error before it's ever actually applied. More expensive, better quality at very low bit-widths, used more by labs shipping their own official low-precision releases.