The output gate is the last of LSTM's three gates — it decides how much of the (now-updated) cell state gets exposed as this time step's hidden state, the value actually used downstream.
Formula
Structurally identical to the forget and input gates — sigmoid-activated, taking the concatenated previous hidden state and current input — with its own independent weights \(\mathbf{W}_o, \mathbf{b}_o\).
How It Produces the Hidden State
Recall from LSTM Hidden State: \(\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{C}_t)\). The freshly updated cell state \(\mathbf{C}_t\) is squashed into \((-1,1)\) via tanh, then the output gate decides, element by element, how much of that squashed value to actually reveal.
Why Separate the Cell State From What Gets Exposed
This separation lets the network maintain information in the cell state that isn't immediately relevant for the current step's output, without being forced to expose it prematurely. A concrete example: an LSTM tracking whether a sentence is a question might maintain that information in the cell state throughout the sentence, but the output gate might only "reveal" it (strongly influence the hidden state) at the very end of the sentence, near the question mark — at earlier positions, that specific dimension of the cell state could remain present but largely gated out of the hidden state.
Numerical Example
Same input setup as the previous gates: \(\mathbf{h}_{t-1}=[0.2,-0.1]\), \(\mathbf{x}_t=[1.0]\), with output-gate weights \(\mathbf{W}_o=[0.1, -0.2, 0.5]\), \(b_o=-0.1\):
Using the cell state \(C_t\approx1.321\) computed in Candidate State:
Code — The Complete Chain, All Four Gates Together
import torch
h_prev = torch.tensor([0.2, -0.1])
x_t = torch.tensor([1.0])
combined = torch.cat([h_prev, x_t])
W_o = torch.tensor([0.1, -0.2, 0.5])
b_o = torch.tensor(-0.1)
z_o = torch.dot(W_o, combined) + b_o
o_t = torch.sigmoid(z_o)
print(o_t) # tensor(0.6082)
C_t = torch.tensor(1.321) # from the previous notes' worked example
h_t = o_t * torch.tanh(C_t)
print(h_t) # tensor(0.5271) -- the complete forward pass for one step, assembled piece by pieceCommon Mistakes
- Applying the output gate directly to the raw cell state instead of \(\tanh(\mathbf{C}_t)\) — the tanh squashing is a required part of the formula, ensuring the hidden state stays in the bounded \((-1,1)\) range.
- Assuming the output gate affects what gets stored in the cell state — it only affects what gets exposed as the hidden state; the cell state itself is already fully updated (via the forget and input gates) before the output gate is even applied.
Interview Relevance
Q: "What's the purpose of the output gate, given that the cell state has already been fully updated by that point in the computation?" The output gate controls what part of the (already updated) long-term memory is relevant to expose right now, as this step's working output — separating "what the network remembers" (the cell state) from "what's immediately useful to reveal at this specific point in the sequence" (the hidden state). This lets information persist quietly in memory without necessarily influencing every single step's visible output.
Practice Question
If the output gate for a specific dimension is close to 0 at a given time step, what does the corresponding hidden-state dimension look like at that step, regardless of the cell state's actual value?
