Recurrent Networks · L4 Foundations

Backpropagation through time, why vanishing gradients are provably worse for recurrence, and the LSTM's gating math.

A vanilla RNN's recurrence is $h_{t} = tanh (W_{hh} h_{t - 1} + W_{x h} x_{t} + b)$ . Training requires backpropagation through time (BPTT): unroll the recurrence into a $T$ -step computation graph and apply the ordinary chain rule across it.

\frac{\partial L}{\partial h _{k}} = \frac{\partial L}{\partial h _{T}} t = k + 1 \prod T \frac{\partial h _{t}}{\partial h _{t - 1}} = \frac{\partial L}{\partial h _{T}} t = k + 1 \prod T W_{hh}^{⊤} diag (tanh^{'} (z_{t}))