Batch GD
- Uses the entire dataset per step
- Very stable gradient estimate
- Extremely slow, often infeasible
Mini-batch SGD
- Uses a small random subset per step
- Noisier but far more updates/sec
- The universal practical choice
Computing the exact gradient over millions of examples before taking a single step is wasteful. Stochastic gradient descent (SGD) instead estimates the gradient from a small random batch (say, 32 or 512 examples), takes a step, and repeats. The noise this introduces is actually often *beneficial* — it helps escape shallow local minima.