The context vector is the single fixed-size vector that carries all the information the encoder passes to the decoder — the encoder's final hidden state (and, for LSTM, cell state), standing in for the entire meaning of the input sequence.
Formula
\(T\) is the length of the input sequence — the context vector is simply the encoder's hidden state after processing the very last input token. Everything the decoder will ever know about the input passes through this one vector.
Why It's Called a "Bottleneck"
No matter how long the input sequence is — 5 words or 500 words — the context vector has exactly the same fixed size (the encoder's hidden dimension). This is precisely the same fixed-capacity limitation discussed for the hidden state generally in Hidden State, but now applied at the most extreme point: the entire input sequence's meaning must be compressed into this single vector, with nothing else surviving to help the decoder.
Numerical Intuition
Imagine translating a 4-word sentence versus a 40-word paragraph, both using an encoder with a 256-dimensional hidden state. Both get compressed into the exact same 256 numbers. Intuitively, a 40-word paragraph carries far more information than a 4-word sentence, yet both must fit through the identical bottleneck — this is exactly why translation quality for longer sentences degrades noticeably with this basic architecture, a limitation quantified further in Seq2Seq Limitations.
Code — Seeing the Compression Directly
import torch
import torch.nn as nn
encoder_rnn = nn.LSTM(input_size=50, hidden_size=256, batch_first=True)
short_input = torch.randn(1, 4, 50) # a 4-token sequence
long_input = torch.randn(1, 40, 50) # a 40-token sequence
_, (h_short, _) = encoder_rnn(short_input)
_, (h_long, _) = encoder_rnn(long_input)
print(h_short.shape) # torch.Size([1, 1, 256])
print(h_long.shape) # torch.Size([1, 1, 256]) -- IDENTICAL size, despite 10x more input informationCommon Mistakes
- Assuming a larger hidden size "solves" the bottleneck problem — it raises the ceiling somewhat, but doesn't eliminate the fundamental issue that a single fixed-size vector must represent inputs of arbitrarily varying length and complexity.
- Confusing the context vector with the decoder's own hidden state during generation — the context vector specifically refers to what's handed from encoder to decoder at the start; the decoder then maintains its own evolving hidden state as it generates each output token.
Interview Relevance
Q: "Why does basic encoder-decoder translation quality tend to degrade on longer sentences?" The entire input sentence, regardless of its length, must be compressed into one fixed-size context vector — the encoder's final hidden state. As sentences get longer, there's proportionally more information competing for the same fixed capacity, so more gets lost or blurred together, directly hurting the decoder's ability to generate an accurate translation.
Practice Question
If you doubled the encoder's hidden size, would that fully eliminate the context-vector bottleneck problem, or just shift where it becomes noticeable? Explain your reasoning.
