Gradient descent's update rule follows directly from a first-order Taylor expansion: near , the loss behaves approximately like , which decreases fastest by moving in the direction .
Layer 4 · Foundations
Training & Optimization
The convex optimization theory behind gradient descent, convergence guarantees, and why loss landscapes for neural nets are so much harder.
15 min read180 XP