The 2017 paper "Attention Is All You Need" made a radical proposal: throw away recurrence entirely — no RNN, no LSTM, no GRU — and build a sequence model using only attention and feedforward layers. This single architectural bet became the foundation of virtually every modern large language model.
The Problem Being Solved
Recall the closing note of the RNN category, Limitations of RNN: even LSTM and GRU, despite fixing the vanishing gradient problem, never solved RNNs' fundamental limitation — sequential, non-parallelizable computation. Every recurrent architecture must process a sequence one step at a time, since each step's computation genuinely depends on the previous step finishing.
The Transformer's Core Bet
Self-attention, covered fully in the previous category, already showed that every pair of positions in a sequence can be related directly, in one computation, with no sequential dependency between positions — recall the diagram in Self-Attention where every token connects to every other token simultaneously. The Transformer's insight: if self-attention can already relate any two positions without recurrence, why keep the recurrent structure at all? Replace it entirely with stacked self-attention and feedforward layers.
Sequential vs Parallel Processing, Directly Compared
Removing recurrence entirely is what unlocks massive parallelization — the single biggest practical reason Transformers scale to today's enormous models.
Why Parallelization Matters So Much in Practice
Modern GPUs and TPUs are built for massively parallel computation. An architecture that must process a sequence strictly one step at a time (any RNN family) fundamentally cannot exploit this hardware capability for the sequence dimension — training time scales directly with sequence length, no matter how much parallel hardware is available. A Transformer, processing all positions simultaneously, can exploit that parallel hardware fully, which is a major reason it became practical to train models on vastly larger datasets and with vastly more parameters than recurrent architectures ever were.
What Still Needs to Be Solved
Removing recurrence isn't free — it introduces new problems this category resolves one at a time: self-attention alone has no notion of token order (addressed by Positional Encoding), training very deep stacks of attention layers needs help (addressed by Residual Connections and normalization), and a decoder generating text one token at a time still needs to avoid "cheating" by looking at future tokens (addressed by Masked Self-Attention).
Common Mistakes
- Assuming Transformers process sequences "faster" in every sense — training and encoding can be fully parallelized, but autoregressive generation (as covered in Seq2Seq Model) still produces one token at a time, since each new token genuinely depends on all previous tokens already having been generated.
- Thinking Transformers eliminated attention's computational cost entirely — self-attention's cost actually grows quadratically with sequence length (every position attends to every other position), a real tradeoff explored further in later categories on efficient inference.
Interview Relevance
Q: "What was the key architectural bet the original Transformer paper made, and why did it matter so much practically?" Removing recurrence entirely and relying solely on self-attention and feedforward layers to process sequences — since self-attention can relate any two positions directly without sequential dependency, the entire sequence can be processed in parallel. This let Transformers fully exploit modern parallel hardware (GPUs/TPUs) during training, unlike RNN-family architectures, which was a major factor enabling the scale of modern large language models.
Practice Question
Why can a Transformer encoder process an entire 500-token input sequence in a single parallel computation, while an LSTM encoder cannot?
