Deep learning is not a new idea — the mathematical building blocks are over 60 years old. What changed recently is data, compute and a handful of algorithmic fixes that finally let those old ideas work at scale.
Timeline of Key Milestones
Two AI winters separate the early theory from the deep learning boom that started in 2012.
The Milestones, Explained
| Year | Breakthrough | Why It Mattered |
|---|---|---|
| 1958 | Rosenblatt's Perceptron | First trainable artificial neuron — could learn a linear decision boundary from examples. |
| 1969 | Minsky & Papert's Perceptrons | Proved a single perceptron can't learn XOR (non-linearly separable problems) — funding for neural nets collapsed, starting the first AI winter. |
| 1986 | Backpropagation popularized (Rumelhart, Hinton, Williams) | Showed how to train multi-layer networks efficiently, which can solve XOR-like problems — but compute and data were still too limited for major impact. |
| 1998 | LeNet-5 (Yann LeCun) | A working convolutional neural network for handwritten digit recognition — proved CNNs work in practice, though adoption stayed niche. |
| 2006 | Deep Belief Networks (Hinton) | Showed deep networks could be trained layer-by-layer, reviving interest ahead of the 2012 breakthrough. |
| 2012 | AlexNet wins ImageNet | A deep CNN trained on GPUs crushed the ImageNet classification benchmark by a huge margin over hand-crafted-feature methods — the moment deep learning went mainstream. |
| 2014 | GANs (Goodfellow et al.) | Introduced generative adversarial training — two networks competing to generate realistic data. |
| 2015 | ResNet | Residual connections made it possible to train networks hundreds of layers deep without degrading. |
| 2017 | "Attention Is All You Need" — the Transformer | Replaced recurrence with self-attention, enabling massive parallel training — the architecture behind every modern LLM. |
| 2018–2020 | BERT, GPT-2, GPT-3 | Showed that pretraining huge Transformers on internet-scale text produces broadly capable language models. |
| 2020–2022 | Diffusion models, Stable Diffusion, ChatGPT | Brought generative AI (text-to-image, conversational assistants) into mainstream daily use. |
Two AI Winters — Why Funding Dried Up
Both AI winters shared the same root cause: the theory was ahead of the available data and compute. The 1969 winter followed a real mathematical limitation (perceptrons can't learn XOR). A second, milder winter in the late 1980s/early 1990s followed inflated expectations about expert systems and neural nets that the hardware of the time couldn't deliver on. Neither collapse meant the underlying ideas were wrong — it meant the ecosystem around them (data, GPUs, better initialization/optimizers) wasn't ready yet.
Common Mistakes
- Thinking deep learning is a 2010s invention — the perceptron is from 1958 and backpropagation from 1986. What's genuinely new is the scale of data and compute (see Why Deep Learning Became Successful).
- Assuming progress has been smooth and continuous — two multi-year AI winters show it was not; funding and interest can collapse when results don't match hype.
Interview Relevance
Q: "If backpropagation was known in 1986, why did deep learning only take off in 2012?" Because algorithms alone weren't the bottleneck — ImageNet-scale labeled datasets and GPU-scale parallel compute didn't exist yet. AlexNet in 2012 combined 1986-era backpropagation with 2012-era data and hardware.
Practice Question
Pick one milestone from the timeline and explain, in your own words, what specific limitation of the previous milestone it solved.
