Vanilla RNN
- One hidden state update per step
- Simple, cheap
- Struggles to remember >10-20 steps back
- Suffers vanishing gradients over long sequences
LSTM / GRU
- Adds gates controlling what to keep/forget
- Separate 'cell state' highway for long-term memory
- Much better at long-range dependencies
- More parameters, still sequential
At each timestep , a vanilla RNN computes — the same weights, applied over and over. The problem: backpropagating through many timesteps multiplies the same matrix repeatedly, causing the exact vanishing/exploding gradient issue seen in very deep feedforward networks, except here 'depth' equals sequence length. A 200-word sentence effectively creates a 200-layer-deep computation.