The general vanishing gradient problem from Vanishing Gradient Problem hits RNNs with particular severity — because, as established in Unrolling RNN, sequence length plays the same role "depth" does in a feedforward network, and sequences are routinely far longer than any feedforward network is deep.
The Mechanism, Specific to RNNs
From Backpropagation Through Time, the gradient reaching an early time step involves a product of many \(\frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_{t-1}}\) terms, one per intervening step:
Each factor involves both the shared weight matrix \(\mathbf{W}_{hh}\) and tanh's derivative (at most 1, per Tanh Function, and typically much smaller). If the eigenvalues of \(\mathbf{W}_{hh}\) (see Eigenvalues) have magnitude less than 1, or if tanh is operating in its saturating region for much of the sequence, this repeated multiplication shrinks the gradient exponentially in sequence length \(T\).
Numerical Example
For a sequence of 50 time steps, with each step's local gradient factor averaging just \(0.7\):
The gradient reaching the first time step is essentially zero by the time it arrives — meaning the network effectively cannot learn dependencies spanning more than a few dozen time steps at most, no matter how relevant an early input might genuinely be to a much later output.
The Practical Consequence: Short-Term Memory Only
This is exactly why plain RNNs are often described as having only "short-term memory" — they can pick up on dependencies spanning a handful of nearby time steps reasonably well, but struggle badly to learn relationships between elements that are far apart in a long sequence, even when the network's hidden state could theoretically carry that information forward, because the gradient signal needed to actually learn to carry it never survives the backward journey intact.
Code — Observing Gradient Magnitude Decay
import torch
import torch.nn as nn
rnn = nn.RNN(input_size=1, hidden_size=1, nonlinearity='tanh')
with torch.no_grad():
rnn.weight_hh_l0.fill_(0.5) # a sub-1 recurrent weight, encouraging shrinkage
sequence = torch.ones(1, 50, 1, requires_grad=True) # 50 time steps
output, _ = rnn(sequence)
loss = output[:, -1, :].sum() # loss depends only on the LAST time step's output
loss.backward()
# Inspect how much gradient reaches inputs at different points in the sequence
print(sequence.grad[0, 0, 0].item()) # gradient reaching the FIRST input -- typically tiny
print(sequence.grad[0, 45, 0].item()) # gradient reaching a NEARBY input -- typically much largerWhy This Motivates LSTM and GRU
This exact limitation — gradients vanishing over long sequences, preventing the network from learning long-range dependencies — is the direct historical and mathematical motivation for the LSTM architecture, covered in the very next category. LSTM introduces a separate "cell state" pathway with a mostly-additive (rather than repeatedly-multiplicative) update rule, specifically designed to let gradients flow across many time steps without this exponential shrinkage.
Common Mistakes
- Assuming a larger hidden size fixes vanishing gradients — hidden size affects the RNN's representational capacity, but doesn't change the fundamental multiplicative structure of BPTT that causes gradient shrinkage over long sequences.
- Attributing an RNN's failure to learn long-range patterns to "not enough training data" before checking whether vanishing gradients over the relevant sequence length are the actual bottleneck.
Interview Relevance
Q: "Why do plain RNNs struggle to learn dependencies between elements that are far apart in a long sequence?" Backpropagation through time computes gradients as a product of local derivatives across every intervening time step. If those local gradients have magnitude less than 1 (common with tanh's saturating derivative and sub-1 recurrent weight eigenvalues), the product shrinks exponentially with sequence length — by the time the gradient signal reaches an early time step in a long sequence, it's often too small to meaningfully update the weights based on that step's true relevance to a much later output.
Practice Question
If a sequence has 100 time steps and the average local gradient factor is 0.9, roughly how much smaller is the gradient reaching step 1 compared to step 100? (Compute \(0.9^{100}\) approximately, or reason about the order of magnitude.)
