Just as with vanishing gradients, RNNs are especially prone to the exploding gradient problem from Exploding Gradient Problem — the same repeated-multiplication mechanism, but with local gradient factors greater than 1 instead of less than 1.
The Mechanism, Specific to RNNs
Reusing the product formula from RNN Vanishing Gradient:
If \(\mathbf{W}_{hh}\)'s eigenvalues have magnitude greater than 1, this same product grows exponentially with sequence length instead of shrinking — producing enormous gradients that can destabilize training entirely.
Numerical Example
For a sequence of 30 time steps, with each step's local gradient factor averaging \(1.4\):
A weight update scaled by a gradient this large, even with a modest learning rate, can move weights by an enormous amount in a single step — frequently producing NaN loss values and effectively crashing training.
Why RNNs Are More Exposed to This Than Typical Feedforward Networks
A standard \(L\)-layer feedforward network has \(L\) distinct weight matrices, each potentially well-conditioned individually even if the network overall is deep. An RNN reuses the exact same \(\mathbf{W}_{hh}\) at every one of potentially hundreds or thousands of time steps — so even a modest amount of "instability" in that one shared matrix compounds identically at every single step, with no opportunity for different matrices to partially offset each other's effects the way distinct per-layer weights in a feedforward network sometimes can.
The Standard, Nearly Mandatory Fix
This is exactly why Gradient Clipping is close to a default requirement when training any RNN, LSTM, or GRU — capping the gradient's norm before each weight update directly prevents any single BPTT-computed gradient from producing a destabilizing update, regardless of how large the raw computed value happens to be.
Code
import torch
import torch.nn as nn
rnn = nn.RNN(input_size=10, hidden_size=20, batch_first=True)
optimizer = torch.optim.Adam(rnn.parameters(), lr=0.001)
sequence = torch.randn(1, 100, 10) # a long sequence -- 100 time steps
output, _ = rnn(sequence)
loss = output.sum()
loss.backward()
# Standard practice for RNNs: clip BEFORE the optimizer step
torch.nn.utils.clip_grad_norm_(rnn.parameters(), max_norm=1.0)
optimizer.step()Common Mistakes
- Training an RNN on long sequences without gradient clipping and attributing resulting
NaNlosses to a data or architecture problem — always check for exploding gradients (and add clipping) before investigating other causes. - Setting the clipping threshold as an afterthought rather than based on observed gradient norm behavior during early training — a threshold that's a reasonable order of magnitude for the specific model and data tends to work better than an arbitrary default.
Interview Relevance
Q: "Why is gradient clipping considered close to mandatory for training RNNs, more so than for typical feedforward networks?" An RNN reuses the same recurrent weight matrix at every time step of potentially very long sequences, so any instability in that shared matrix compounds multiplicatively and identically across every step — making exploding gradients both more likely and more severe than in a typical feedforward network with distinct per-layer weights. Gradient clipping directly caps the resulting gradient magnitude before it can produce a destabilizing weight update.
Practice Question
Two RNNs are trained on sequences of length 20 and length 200 respectively, with otherwise identical architectures and weight initializations. Which is more likely to suffer from exploding gradients, and why?
