AI Guide · Updated 14 November 2026
History
The deep learning revolution, 2012-2017
By drayz.ai · Updated · 11 min read

Between 2012 and 2017, deep learning went from a niche research interest to the engine of the entire technology industry. Neural networks with many layers, trained on huge datasets using graphics chips, suddenly beat every other method at recognising images, understanding speech, translating languages and playing games. London's DeepMind became world-famous when AlphaGo beat one of the best Go players in history. And in 2017 a short paper from Google introduced the Transformer, the design behind today's chatbots. This chapter tells the story.
What deep learning is
A neural network is a mathematical model made of layers of simple units, each of which multiplies its inputs by adjustable numbers called weights, adds them up and passes the result through a simple function. Deep learning means using networks with many such layers, sometimes dozens or hundreds.
The key advantage of deep networks is that they learn their own features. Earlier machine learning needed engineers to decide what to measure: edges, colours, keywords. A deep network trained on raw pixels discovers for itself that edges matter in its first layers, that combinations of edges form textures and parts in the middle layers, and that combinations of parts form whole objects in its final layers. This representation learning is why deep learning scaled so well: give it more data and more computing power, and it finds better features by itself.
Deep networks are trained with backpropagation and stochastic gradient descent: show the network an example, measure its error, calculate how much each weight contributed to that error, and nudge every weight slightly in the direction that reduces it. Repeat millions of times.
The ideas were decades old. What changed around 2012 was the arrival of enough data (the internet and ImageNet), enough computing power (GPUs) and a set of practical tricks that made deep networks train reliably.
AlexNet and the ImageNet moment, 2012
The ImageNet Large Scale Visual Recognition Challenge asked programs to classify images into 1,000 categories, from "tabby cat" to "container ship". In 2010 and 2011 the best systems used hand-engineered features and had a "top-5" error rate (the right answer not among the program's top five guesses) of around 26-28%.
In 2012 a team from the University of Toronto, Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, entered a deep convolutional neural network later known as AlexNet. It had about 60 million parameters and eight learned layers, and it was trained for about a week on two Nvidia gaming GPUs in Krizhevsky's bedroom. It achieved a top-5 error rate of about 15.3%, against 26.2% for the runner-up. In a field where improvements were usually measured in fractions of a percentage point, it was a shock.
AlexNet combined several ideas that became standard:
- convolutional layers, which slide small learned filters across an image, building on Yann LeCun's work from the 1980s and 1990s;
- the ReLU activation function, which made deep networks train much faster;
- dropout, a technique developed in Hinton's group that randomly switches off parts of the network during training to stop it memorising the training data;
- data augmentation, creating extra training examples by flipping and cropping images;
- and, crucially, GPU training.
Within two years almost every serious computer vision team had switched to deep learning. Hinton, Krizhevsky and Sutskever formed a tiny company, DNNresearch, which Google acquired in 2013 after a bidding war.
Rapid progress in vision
After AlexNet, results improved every year:
- 2014: Oxford University's Visual Geometry Group produced VGG, a simple and very deep network that became one of the most widely used in research. Google's GoogLeNet (Inception) won the competition.
- 2015: Microsoft Research's ResNet (residual networks), by Kaiming He and colleagues, used "skip connections" to train networks over 150 layers deep. Its error rate of about 3.6% was below commonly quoted estimates of human performance on the task (about 5%). Residual connections are now used in almost every large neural network, including language models.
By 2017 the ImageNet challenge was effectively solved and ended. Computer vision moved into products: photo search on phones, face unlock, medical image analysis, quality control in factories, and the perception systems of self-driving cars.
Speech and translation
Deep learning transformed speech recognition at the same time. In 2012 researchers from Microsoft, Google, IBM and the University of Toronto published a joint paper showing that deep neural networks substantially cut error rates in speech recognition. Within a few years, voice assistants, dictation and automatic captions became far more accurate.
Language translation followed. In 2014 Ilya Sutskever, Oriol Vinyals and Quoc Le at Google introduced sequence-to-sequence learning, using LSTM networks to read a sentence in one language and generate it in another. In the same year Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio introduced the attention mechanism, which let the model focus on the relevant words in the source sentence while generating each word of the translation. In 2016 Google switched Google Translate to Google Neural Machine Translation, reporting that it cut translation errors by around 60% on several major language pairs compared with the previous system.
Words as numbers: word2vec
In 2013 Tomas Mikolov and colleagues at Google released word2vec, which learned to represent each word as a list of numbers (a vector) from the contexts in which it appears. Words with similar meanings ended up close together, and the vectors captured relationships: famously, the vector for "king" minus "man" plus "woman" came out close to "queen". These embeddings became a standard building block of language AI, and the same idea, representing meaning as points in a high-dimensional space, underlies today's AI search and retrieval systems.
Machines that imagine: GANs
In 2014 Ian Goodfellow and colleagues at the University of Montreal introduced generative adversarial networks (GANs). A GAN pits two networks against each other: a generator that creates fake images, and a discriminator that tries to tell fakes from real ones. As each improves, the generator learns to produce increasingly realistic images. By 2018-2019, GANs such as Nvidia's StyleGAN could generate photorealistic faces of people who do not exist. GANs launched the field of modern generative AI, and also the problem of deepfakes: the term appeared in 2017, when face-swapped videos began circulating online.
Other generative approaches developed in the same period included variational autoencoders (2013) and the first diffusion models (2015), which would later power image generators such as Stable Diffusion and DALL-E 2.
DeepMind: London's AI giant
DeepMind was founded in London in 2010 by Demis Hassabis, a former chess prodigy, video game designer and neuroscientist, Shane Legg, a machine learning researcher focused on artificial general intelligence, and Mustafa Suleyman. Its stated mission was to "solve intelligence" and use it to solve everything else.
Learning to play Atari
In 2013 DeepMind published a paper showing that a single program, the Deep Q-Network (DQN), could learn to play dozens of classic Atari 2600 video games, such as Breakout and Space Invaders, directly from the pixels on the screen and the score, with no game-specific programming. It combined deep learning with reinforcement learning. A 2015 paper in Nature reported that DQN reached or exceeded human-level performance on many of 49 games. Its discovery that the best strategy in Breakout was to dig a tunnel through the side of the wall and bounce the ball behind it became a famous example of AI finding creative solutions.
In January 2014 Google acquired DeepMind, for a price reported at around £400 million, one of the largest acquisitions of a European AI company at the time. DeepMind remained in London, and its founders insisted on an ethics and safety board as part of the deal.
AlphaGo
The ancient board game Go had long been seen as a "grand challenge" for AI. It has far more possible positions than chess (more than the number of atoms in the observable universe), and good play relies on intuition that is hard to express as rules. In 2014 many experts thought a computer would not beat a top professional for another decade.
DeepMind's AlphaGo combined deep neural networks, trained first on records of human expert games and then by playing millions of games against itself, with a search technique called Monte Carlo tree search. In October 2015 it beat the European champion Fan Hui 5-0, the first time a program had beaten a professional Go player on a full-size board without a handicap. The result was published in Nature in January 2016.
In March 2016 AlphaGo played Lee Sedol, one of the strongest players of the previous decade, in Seoul. More than 200 million people watched worldwide. AlphaGo won 4-1. In game two it played Move 37, a move on the fifth line that commentators first thought was a mistake, and which turned out to be brilliant; AlphaGo's own estimate was that a human player would choose it with a probability of about one in ten thousand. In game four, Lee Sedol found his own brilliant "Move 78", the only game he won. In 2019 Lee retired from professional Go, saying that AI was an entity that could not be defeated.
AlphaGo's victory was a cultural shock, particularly in East Asia, and it helped prompt China's 2017 national plan to become a world leader in AI by 2030.
AlphaGo Zero and AlphaZero
In 2017 DeepMind went further. AlphaGo Zero learned Go entirely by playing against itself, starting from random play and with no human game records, and within days surpassed all previous versions. Later that year AlphaZero used the same general approach to master Go, chess and shogi (Japanese chess), beating the world's strongest chess program, Stockfish, after a few hours of self-play training. Grandmasters praised its aggressive, creative style.
The lesson was striking: in well-defined domains, a general learning algorithm with enough computing power could discover knowledge that surpassed thousands of years of human expertise.
The tools that democratised deep learning
Deep learning spread so quickly partly because the tools became free and easy to use:
- Theano (Montreal, 2007), Caffe (Berkeley, 2013), TensorFlow (Google, 2015) and PyTorch (Facebook, 2016) let researchers and engineers build and train networks without writing low-level GPU code. PyTorch became the dominant research framework.
- Keras (2015) made building networks almost as simple as stacking blocks.
- Research papers were posted openly on arXiv, often with code on GitHub, so new ideas spread around the world within days.
- Online courses, such as Andrew Ng's machine learning course (2011) and fast.ai (2016), taught deep learning to hundreds of thousands of people.
The AI talent race and the founding of OpenAI
As deep learning proved its value, the big technology companies competed fiercely for talent. Google hired Hinton (2013) and bought DeepMind (2014). Facebook hired Yann LeCun to lead its new AI research lab in 2013. Baidu hired Andrew Ng. Salaries for top researchers soared.
In December 2015 OpenAI was founded in San Francisco as a non-profit research lab by a group including Sam Altman, Elon Musk, Ilya Sutskever and Greg Brockman, with stated pledges of $1 billion. Its mission was to ensure that artificial general intelligence benefits all of humanity. In 2016 it released OpenAI Gym, a toolkit for reinforcement learning research, and in 2017 it began working on training game-playing agents for the complex online game Dota 2.
Early warnings: bias, deepfakes and safety
Deep learning's success also raised concerns that are now central to AI policy:
- Bias. In 2015 Google Photos labelled photos of Black people as "gorillas", an error that exposed how training data and testing could fail some groups. In 2018 the Gender Shades study by Joy Buolamwini and Timnit Gebru found that commercial face analysis systems were far less accurate for darker-skinned women than for lighter-skinned men.
- Adversarial examples. In 2013-2014 researchers showed that tiny, invisible changes to an image could fool a neural network into misclassifying it with high confidence, for example making it see a panda as a gibbon. This revealed how differently AI "sees" from people.
- Deepfakes. Face-swapping tools appeared in 2017, raising fears about fake videos and non-consensual imagery.
- Long-term safety. Oxford philosopher Nick Bostrom's 2014 book Superintelligence argued that advanced AI could pose existential risks. In 2015 thousands of researchers signed an open letter calling for research into making AI robust and beneficial, and in 2016 researchers from Google and OpenAI published "Concrete Problems in AI Safety". The UK's Leverhulme Centre for the Future of Intelligence (Cambridge, 2016) and other institutes began studying the societal impact of AI.
2017: Attention Is All You Need
In June 2017 eight researchers at Google, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin, published a paper titled "Attention Is All You Need". It introduced the Transformer, a neural network design for processing sequences such as sentences.
Previous language models, based on recurrent networks such as LSTMs, read text one word at a time, which made them slow to train and prone to forgetting earlier words. The Transformer dropped recurrence and relied entirely on self-attention: every word in a passage looks at every other word at once and works out which ones are most relevant to its meaning. Because all positions are processed in parallel, Transformers could be trained far faster on GPUs and scaled to enormous sizes.
The paper was written to improve machine translation, and it did, setting new records on English-to-German and English-to-French benchmarks. But its real impact was far greater. The Transformer became the foundation of BERT, GPT, Claude, Gemini, Llama and almost every other major language model, as well as many image, audio, video, protein and robotics models. Several of the paper's authors went on to found AI companies, including Cohere (Aidan Gomez) and Character.AI (Noam Shazeer); Llion Jones, who is Welsh, co-founded Sakana AI in Tokyo. The title itself became one of the most imitated in computer science.
Why this era matters
Between 2012 and 2017, deep learning:
- overturned decades of computer vision and speech research in a few years;
- made AI a top priority for every major technology company and many governments;
- made London a global AI centre through DeepMind;
- showed, through AlphaGo and AlphaZero, that learning systems could surpass human expertise in complex domains;
- created modern generative AI, and with it deepfakes;
- surfaced the problems of bias, robustness and safety that dominate AI policy today;
- and produced the Transformer, which made the large language model era possible.
Key takeaways
- Deep learning uses many-layered neural networks that learn their own features from data.
- AlexNet's 2012 ImageNet win (15.3% vs 26.2% error) started the revolution, powered by GPUs.
- ResNet (2015) surpassed estimated human accuracy on ImageNet; deep learning transformed speech recognition and translation.
- GANs (2014) launched modern generative AI and, later, deepfakes.
- London's DeepMind, bought by Google in 2014, built AlphaGo, which beat Lee Sedol 4-1 in 2016.
- The 2017 Transformer paper, "Attention Is All You Need", underpins nearly all of today's large language models.
Previous chapter: The machine learning era, 1993-2011 · Next chapter: Transformers and large language models, 2018-2022