This note maps out the complete LSTM cell's structure at a glance — every component and how they connect — before the next several notes examine each piece individually in depth.
The Five Components
| Component | Role |
|---|---|
| Cell state \(\mathbf{C}_t\) | The long-term memory pathway — carries information across time with minimal distortion |
| Hidden state \(\mathbf{h}_t\) | The short-term, "working" output — a filtered view of the cell state, exposed to the next layer and passed to the next time step |
| Forget gate \(\mathbf{f}_t\) | Decides what fraction of the old cell state to discard |
| Input gate \(\mathbf{i}_t\) | Decides how much new information to add to the cell state |
| Output gate \(\mathbf{o}_t\) | Decides how much of the cell state to expose as this step's hidden state |
A sixth component, the candidate state \(\tilde{\mathbf{C}}_t\), computes what new information could be added — the input gate then decides how much of it actually gets incorporated.
Diagram — The Full LSTM Cell
The cell state flows along the top, modified only by the forget gate (multiply) and input gate (add). The hidden state is derived from the cell state, filtered by the output gate.
What Every Gate Has in Common
Every gate (\(\mathbf{f}_t, \mathbf{i}_t, \mathbf{o}_t\)) is computed the same structural way: a sigmoid-activated linear layer taking the concatenation of the previous hidden state \(\mathbf{h}_{t-1}\) and the current input \(\mathbf{x}_t\):
Sigmoid's (0,1) output range (see Sigmoid Function) is exactly what makes it interpretable as "how much to let through" — 0 means block entirely, 1 means let everything through, and values in between are learned, soft decisions.
Code — Every Gate's Weight Shape
import torch.nn as nn
lstm = nn.LSTM(input_size=10, hidden_size=20, batch_first=True)
# PyTorch packs all 4 gates' weights into one combined tensor for efficiency:
print(lstm.weight_ih_l0.shape) # torch.Size([80, 10]) -- 80 = 4 gates x hidden_size(20)
print(lstm.weight_hh_l0.shape) # torch.Size([80, 20]) -- same idea, from the hidden state
# The 4 gates packed together are: input, forget, candidate (cell), output -- in that orderCommon Mistakes
- Assuming the cell state and hidden state are the same thing — they're two genuinely distinct vectors with different roles: the cell state is the protected long-term memory pathway, the hidden state is a filtered, exposed "working" view of it.
- Forgetting that every gate takes both the previous hidden state and the current input as its input — a gate's decision is context-dependent on both what's happened so far and what's arriving right now.
Interview Relevance
Q: "How many distinct learned components does one LSTM cell have, and what does each control?" Four gates/states computed from \([\mathbf{h}_{t-1}, \mathbf{x}_t]\): the forget gate (how much old cell state to discard), the input gate (how much new candidate information to add), the candidate state (what new information could be added), and the output gate (how much of the cell state to expose as the hidden state) — plus the cell state and hidden state themselves as the two carried memory pathways.
Practice Question
Why does PyTorch's weight_ih_l0 for an LSTM with hidden size 20 have shape \((80, \text{input\_size})\) rather than \((20, \text{input\_size})\)?
