This note runs a complete LSTM forward pass, for one full time step, entirely by hand with real numbers — assembling every equation from LSTM Equations into one continuous worked example, then verifying every value against PyTorch.
The Setup
Combined input for every gate: \([\mathbf{h}_{t-1}, x_t] = [0.1, 0.2, 1.0]\). For simplicity, this example uses scalar weight rows shared identically for both cell-state dimensions (a genuine LSTM would have independent weights per dimension — this simplification keeps the hand-computation tractable while preserving the exact mechanism).
Step 1 — Forget Gate
Step 2 — Input Gate
Step 3 — Candidate State
Step 4 — Cell State Update (applied per dimension of \(\mathbf{C}_{t-1}\))
Step 5 — Output Gate
Step 6 — Hidden State
Code — Verifying Every Value
import torch
h_prev = torch.tensor([0.1, 0.2])
C_prev = torch.tensor([0.5, -0.3])
x_t = torch.tensor([1.0])
combined = torch.cat([h_prev, x_t])
W_f, b_f = torch.tensor([0.4, -0.2, 0.3]), torch.tensor(0.1)
W_i, b_i = torch.tensor([0.2, 0.5, -0.1]), torch.tensor(0.0)
W_C, b_C = torch.tensor([-0.3, 0.4, 0.6]), torch.tensor(0.2)
W_o, b_o = torch.tensor([0.5, 0.1, -0.2]), torch.tensor(-0.1)
f_t = torch.sigmoid(torch.dot(W_f, combined) + b_f)
i_t = torch.sigmoid(torch.dot(W_i, combined) + b_i)
C_candidate = torch.tanh(torch.dot(W_C, combined) + b_C)
C_t = f_t * C_prev + i_t * C_candidate
o_t = torch.sigmoid(torch.dot(W_o, combined) + b_o)
h_t = o_t * torch.tanh(C_t)
print("f_t:", f_t.item()) # 0.5987
print("i_t:", i_t.item()) # 0.5050
print("C_candidate:", C_candidate.item()) # 0.6911
print("C_t:", C_t) # tensor([0.6484, 0.1694])
print("o_t:", o_t.item()) # 0.4427
print("h_t:", h_t) # tensor([0.2527, 0.0743]) -- matches every hand-computed valueCommon Mistakes
- Applying the forget and input gates as if they were single scalars rather than per-dimension vectors when the cell state has more than one dimension — each element of \(\mathbf{C}_t\) is scaled independently by the corresponding element of \(\mathbf{f}_t\) and \(\mathbf{i}_t\).
- Losing track of the order of operations — the cell state must be fully updated (Step 4) before the output gate is applied to it (Steps 5–6); the output gate never influences the cell-state update itself.
Interview Relevance
Q: "Walk through one complete LSTM time step with real numbers." This exact worked example — computing the forget gate, input gate, candidate state, updated cell state, output gate, and finally the hidden state, in that specific order, with real numbers at each step — is exactly the kind of exercise a strong candidate should be able to reproduce fluently for a sequence-modeling interview.
Practice Question
Using the same weights, compute one more time step, now with \(\mathbf{h}_t \approx [0.2527, 0.0743]\) and \(\mathbf{C}_t \approx [0.6484, 0.1694]\) as the new "previous" values, and a new input \(x_{t+1}=-0.5\). (You only need to compute the forget gate for this practice.)
