The update gate is GRU's merged replacement for LSTM's separate forget and input gates — it controls, in a single decision, how much of the old hidden state to keep versus how much to replace with new candidate content.
Formula
Structurally the same pattern as every gate seen so far — a sigmoid-activated linear layer over the concatenated previous hidden state and current input.
How One Gate Does the Work of Two
Recall LSTM's cell-state update: \(\mathbf{C}_t = \mathbf{f}_t\odot\mathbf{C}_{t-1}+\mathbf{i}_t\odot\tilde{\mathbf{C}}_t\), with \(\mathbf{f}_t\) and \(\mathbf{i}_t\) as two independently learned gates. GRU's hidden-state update (full formula in GRU Equations) instead uses a single \(\mathbf{z}_t\) for both roles simultaneously:
Notice \((1-\mathbf{z}_t)\) and \(\mathbf{z}_t\) are exact complements — if \(\mathbf{z}_t\) is close to 1, almost all of the new candidate is used and almost none of the old state is kept; if \(\mathbf{z}_t\) is close to 0, the old state is almost entirely preserved and almost none of the new candidate gets in. This couples the "how much to forget" and "how much to add" decisions into a single, complementary tradeoff — a genuine simplification (fewer independent parameters) compared to LSTM's fully independent forget and input gates, at the cost of losing LSTM's ability to, say, forget a lot AND add only a little at the same time.
Numerical Example
\(z_t = 0.7\): 70% of the new candidate is used, 30% of the old hidden state is retained. With \(h_{t-1}=0.4\), \(\tilde h_t = -0.2\):
Code
import torch
h_prev = torch.tensor(0.4)
z_t = torch.tensor(0.7)
h_candidate = torch.tensor(-0.2)
h_t = (1 - z_t) * h_prev + z_t * h_candidate
print(h_t) # tensor(-0.0200) -- matches the hand-worked exampleCommon Mistakes
- Assuming the update gate works identically to LSTM's input gate — LSTM's input gate scales only the new candidate's contribution, independently of the forget gate; GRU's update gate scales both the old and new contributions simultaneously, as complementary fractions that always sum to 1.
- Forgetting that this coupling is a genuine tradeoff, not a strict improvement — it reduces GRU's parameter count and can simplify training, but it does remove one degree of freedom LSTM has (independently controlling forget and input amounts).
Interview Relevance
Q: "How does GRU's update gate relate to LSTM's forget and input gates?" It merges their roles into a single gate: \(\mathbf{z}_t\) directly determines both how much of the new candidate to incorporate (weighted by \(\mathbf{z}_t\)) and how much of the old hidden state to retain (weighted by the complement, \(1-\mathbf{z}_t\)) — a coupled, complementary tradeoff rather than LSTM's two independently-learned gates.
Practice Question
If the update gate outputs exactly 0 for a specific dimension, what happens to that dimension of the hidden state at this time step?
