Why LSTM

The LSTM (Long Short-Term Memory) network, introduced by Hochreiter and Schmidhuber in 1997, was designed to directly solve the exact problem closed out in the previous category: plain RNNs cannot learn long-range dependencies because gradients vanish across long sequences.

The Core Insight

Recall from RNN Vanishing Gradient that a plain RNN's hidden state is recomputed from scratch at every step through a full \(\tanh\) non-linearity: \(\mathbf{h}_t = \tanh(\mathbf{W}_{xh}\mathbf{x}_t+\mathbf{W}_{hh}\mathbf{h}_{t-1}+\mathbf{b}_h)\). Every single time step forces the carried information through a squashing non-linearity and a matrix multiplication — repeated many times, this is exactly what causes gradients to shrink (or grow) exponentially.

LSTM's key idea: add a second pathway — the cell state — that carries information forward with a much gentler update rule, dominated by element-wise multiplication and addition rather than being forced through a new non-linear transformation at every single step. Gradients can flow along this pathway across many time steps far more easily than through a plain RNN's repeatedly-squashed hidden state.

The "Conveyor Belt" Analogy

How LSTM Achieves This: Gates

Rather than blindly overwriting information at every step, an LSTM cell uses learned gates — small neural network components (each a sigmoid-activated layer, producing values in \((0,1)\)) that control precisely how much information to forget, add, or expose at each step. Each gate answers one specific question: how much of the old memory should we keep? How much new information should we add? How much of the current memory should we reveal as this step's output? These gates are covered individually in the next four notes.

Code — A Preview of the API

import torch.nn as nn

lstm = nn.LSTM(input_size=10, hidden_size=20, batch_first=True)
# Notice LSTM returns TWO states per step -- hidden state AND cell state --
# unlike a plain RNN's single hidden state; this reflects the dual-pathway design

Common Mistakes

• Assuming LSTM eliminates vanishing/exploding gradients entirely — it substantially mitigates the vanishing gradient problem specifically, making much longer-range dependencies learnable in practice, but gradient clipping is still standard practice for exploding gradients, and extremely long sequences can still pose challenges.
• Thinking "LSTM" is an unrelated architecture from RNN — it's specifically a more sophisticated type of recurrent cell; it's still fundamentally an RNN in the broad sense (sequential processing, shared weights across time), just with a more carefully engineered internal update rule.

Interview Relevance

Q: "What's the core architectural idea that lets LSTM avoid the vanishing gradient problem that limits plain RNNs?" A separate cell state pathway that's updated mostly through element-wise multiplication and addition (controlled by learned gates), rather than being fully recomputed through a non-linearity at every step like a plain RNN's hidden state. This near-additive update path lets gradients flow across many more time steps with much less exponential shrinkage.

Practice Question

Why does a purely additive update path (versus one that squashes values through tanh at every step) help preserve gradient magnitude across many time steps?

Related DL Notes

• RNN Vanishing Gradient
• LSTM Architecture
• LSTM Cell State