The GRU (Gated Recurrent Unit), introduced in 2014, is a simplified alternative to LSTM — it captures the same core benefit (gated, gradient-friendly memory) with a leaner structure: two gates instead of four, and a single state instead of two.
The Key Simplifications
| LSTM | GRU | |
|---|---|---|
| Number of states | Two — cell state \(\mathbf{C}_t\) and hidden state \(\mathbf{h}_t\) | One — just the hidden state \(\mathbf{h}_t\), which serves both roles |
| Number of gates | Three — forget, input, output | Two — update and reset (covered in the next two notes) |
| Parameter count | Higher (roughly 4x a plain RNN) | Lower (roughly 3x a plain RNN) |
GRU merges LSTM's separate cell state and hidden state into a single state vector, and merges the roles of the forget and input gates into one update gate — since in LSTM, "how much to forget" and "how much to add" are conceptually related decisions (if you're adding a lot of new information, it often makes sense to forget a correspondingly large amount of old information, though LSTM doesn't force this relationship).
Diagram — GRU's Simpler Structure
A single state pathway, blended by one update gate — visually and structurally leaner than LSTM's separate cell/hidden states and three-gate system.
Code — The API Comparison
import torch.nn as nn
lstm = nn.LSTM(input_size=10, hidden_size=20)
gru = nn.GRU(input_size=10, hidden_size=20)
print(sum(p.numel() for p in lstm.parameters())) # roughly 4x a plain RNN
print(sum(p.numel() for p in gru.parameters())) # roughly 3x a plain RNN -- fewer than LSTM
# GRU returns only ONE state per step (no separate cell state):
import torch
x = torch.randn(1, 5, 10)
output, h_final = gru(x) # note: just h_final, not (h_final, c_final) like LSTMCommon Mistakes
- Assuming GRU is strictly "worse" than LSTM because it's simpler — empirically, GRU often performs comparably to LSTM on many tasks, sometimes even matching or exceeding it, while training faster due to fewer parameters; neither is a strictly dominant choice across all tasks.
- Forgetting that GRU's
forward()call returns a different tuple structure than LSTM's — GRU has no separate cell state to return, which is a common source of code adapted from one to the other breaking.
Interview Relevance
Q: "What are the two main structural simplifications GRU makes compared to LSTM?" GRU merges LSTM's separate cell state and hidden state into a single state vector, and merges the forget and input gates into one combined update gate — reducing both the number of distinct states (from 2 to 1) and the number of gates (from 3 to 2), which reduces total parameter count by roughly 25% compared to LSTM at the same hidden size.
Practice Question
Roughly how many times more parameters does an LSTM have compared to a plain RNN with the same hidden size, and how does GRU compare to both?
