The forget gate is the first of LSTM's three gates — it decides, for every element of the cell state, what fraction of the old memory to keep versus discard.
Formula
\([\mathbf{h}_{t-1}, \mathbf{x}_t]\) denotes concatenating the previous hidden state and current input into one vector, which is then passed through a sigmoid-activated linear layer. Since sigmoid outputs values in \((0,1)\), each element of \(\mathbf{f}_t\) is interpretable directly as "keep this fraction of the corresponding cell state element."
Reading the Gate's Values
| \(f_t\) Value (per element) | Meaning |
|---|---|
| Close to 1 | "Keep this piece of memory almost entirely" — this element of the cell state is preserved across this time step |
| Close to 0 | "Discard this piece of memory" — this element gets nearly wiped out and must be re-learned from scratch by future input if needed again |
| Around 0.5 | A partial, soft retain/discard decision |
A Concrete Intuition
Imagine an LSTM processing a document, tracking the current sentence's subject in one dimension of the cell state. When a new sentence begins (signaled by, say, a period followed by a new word), the forget gate for that specific dimension could learn to output a value close to 0 — deliberately "forgetting" the previous sentence's subject, since it's no longer relevant, freeing that part of memory for the new sentence's subject.
Numerical Example
\(\mathbf{h}_{t-1}=[0.2, -0.1]\), \(\mathbf{x}_t=[1.0]\), concatenated: \([0.2,-0.1,1.0]\). With \(\mathbf{W}_f=[0.5, -0.3, 0.8]\), \(b_f=0\):
This forget gate value (\(\approx0.717\)) would retain about 72% of the corresponding cell state element.
Code
import torch
h_prev = torch.tensor([0.2, -0.1])
x_t = torch.tensor([1.0])
combined = torch.cat([h_prev, x_t])
W_f = torch.tensor([0.5, -0.3, 0.8])
b_f = torch.tensor(0.0)
z_f = torch.dot(W_f, combined) + b_f
f_t = torch.sigmoid(z_f)
print(f_t) # tensor(0.7171) -- matches the hand-worked exampleCommon Mistakes
- Assuming the forget gate is a single scalar shared across the whole cell state — it's actually a full vector, one value per cell-state dimension, letting the network forget different pieces of information at different rates independently.
- Confusing "forget gate outputs near 0" with "the network made a mistake" — a low forget-gate value can be exactly the correct, learned behavior when old information genuinely becomes irrelevant (like the new-sentence example above).
Interview Relevance
Q: "Why does the forget gate use sigmoid specifically, rather than tanh or ReLU?" Sigmoid's output range, \((0,1)\), maps directly onto "what fraction of this memory to retain" — 0 means fully discard, 1 means fully retain, and values in between represent a soft, learned partial retention. Tanh's range \((-1,1)\) or ReLU's unbounded range wouldn't have this clean multiplicative "fraction to keep" interpretation.
Practice Question
If a specific cell-state dimension's forget gate value is consistently around 0.99 across many time steps, what does that suggest about how the network is using that dimension of memory?
