LSTM Hidden State

While the cell state is LSTM's protected long-term memory, the hidden state \(\mathbf{h}_t\) is its short-term "working" output — a filtered, bounded view of the cell state, exposed to whatever comes next (the next layer, or the next time step's gate computations).

The Formula

The current cell state is squashed through \(\tanh\) (bringing it into the bounded range \((-1,1)\)), then scaled element-wise by the output gate \(\mathbf{o}_t\) — deciding how much of that squashed cell state to actually reveal as this step's output.

Why Two Separate States, Not Just One

Separating these lets the network preserve information in the cell state that isn't immediately useful for the current step's output, without being forced to discard it — the hidden state can be a selectively filtered, task-relevant "view," while the cell state keeps carrying everything the network has decided is worth remembering longer-term.

Numerical Example

Continuing the scalar example from LSTM Cell State, where \(C_t=2.25\): with output gate \(o_t=0.6\):

Code

import torch
import torch.nn as nn

lstm = nn.LSTM(input_size=5, hidden_size=8, batch_first=True)
x = torch.randn(1, 3, 5)

output, (h_final, c_final) = lstm(x)
print(output.shape)    # (1, 3, 8) -- the hidden state at every time step
print(h_final.shape)   # (1, 1, 8) -- final hidden state (same as output[:, -1, :])
print(c_final.shape)   # (1, 1, 8) -- final CELL state, a separate tensor entirely

Common Mistakes

• Assuming output from nn.LSTM is the cell state — it's the hidden state at every time step; the cell state is returned separately and typically only its final value is exposed, not its full history across time steps.
• Feeding the raw cell state directly into a downstream classification/output layer instead of the hidden state — the hidden state is specifically the bounded, gate-filtered representation intended for consumption outside the cell.

Interview Relevance

Q: "What's the functional difference between an LSTM's cell state and its hidden state?" The cell state is a largely unfiltered, long-term memory pathway updated through gentle, mostly-additive operations, protected from the repeated non-linear squashing that causes vanishing gradients. The hidden state is a bounded, output-gate-filtered "view" of the cell state — the task-relevant information actually exposed at this time step, both for external use and for computing the next step's gates.

Practice Question

With \(C_t=-1.0\) and output gate \(o_t=0.8\), compute the hidden state \(h_t\).

Related DL Notes

• LSTM Cell State
• Output Gate
• LSTM Equations