Neural networks and backpropagation are decades old. Deep learning's takeover of AI since 2012 came down to three things arriving together: data, compute, and a handful of algorithmic fixes that made deep networks actually trainable.
The Three Pillars
Remove any one of the three pillars and the 2012+ deep learning boom does not happen.
Pillar 1 — Data
Neural networks with millions of parameters need proportionally large datasets to learn generalizable patterns instead of memorizing noise. ImageNet (2009) provided over a million labeled images across 1,000 categories — the first dataset large enough for deep CNNs to show their advantage. The modern web supplies an even larger resource: billions of text tokens, images and videos that self-supervised pretraining can learn from without manual labeling (see Types of Learning).
Pillar 2 — Compute
| Era | Hardware | Practical Effect |
|---|---|---|
| Pre-2012 | CPUs | Training a deep CNN on a large dataset took weeks to months — impractical to iterate. |
| 2012 onward | GPUs (originally built for graphics rendering) | Matrix multiplications — the core operation of a neural network — parallelize extremely well on GPU cores, cutting training time from weeks to days. |
| 2016 onward | TPUs, multi-GPU clusters | Enabled training networks with billions of parameters (modern LLMs) across many chips at once. |
A neural network's forward and backward pass is dominated by matrix multiplication — exactly the operation GPUs were already optimized for. This is a large part of why AlexNet's authors trained on GPUs when almost no one else in computer vision was doing so in 2012.
Pillar 3 — Algorithmic Fixes
| Fix | Problem It Solved |
|---|---|
| ReLU activation | Sigmoid/tanh activations saturate and stop passing useful gradients in deep networks (vanishing gradients); ReLU doesn't saturate for positive inputs. |
| Better weight initialization (Xavier/He) | Random initialization that's too large or small causes activations to explode or vanish across many layers. |
| Dropout | Large networks overfit small datasets; dropout randomly disables neurons during training to force redundant, generalizable representations. |
| Batch normalization | Stabilizes and speeds up training by normalizing layer inputs, allowing higher learning rates. |
| Adam optimizer | Adapts the learning rate per parameter, converging faster and more reliably than plain gradient descent on deep, noisy loss surfaces. |
Each of these is covered in full depth later in this hub — ReLU, Dropout, Batch Normalization and Adam all get their own notes with formulas and code.
Common Mistakes
- Crediting a single cause ("GPUs made deep learning work") — it took data, compute and algorithmic fixes together; any one alone was insufficient, as the 1986–2012 gap shows.
- Assuming today's largest models succeed purely by scale — architectural choices (attention, normalization, residual connections) still matter enormously at every scale.
Interview Relevance
Q: "What three factors explain deep learning's rise since 2012?" Large labeled/self-supervised datasets, GPU/TPU-scale parallel compute, and algorithmic fixes (ReLU, dropout, batch norm, Adam, better initialization) that made very deep networks actually trainable. All three were necessary; none alone was sufficient.
Practice Question
Explain, in one or two sentences, why a GPU trains a neural network faster than a CPU of similar cost, in terms of the operation a neural network spends most of its time doing.
