AI Guide · Updated 14 November 2026
History
Transformers and large language models, 2018-2022
By drayz.ai · Updated · 10 min read

Between 2018 and 2022, a single neural network design, the Transformer, turned language AI upside down. Models trained on huge amounts of text began to write, summarise, translate and answer questions far better than anything before them. Scientists used similar methods to crack the 50-year-old protein folding problem, and image generators learned to paint from a sentence. This chapter explains how large language models (LLMs) were born, and why they led to ChatGPT.
The big idea: pre-train, then fine-tune
Before 2018, most AI systems were trained from scratch for one job: one model for translation, another for sentiment analysis, another for answering questions. Each needed its own large set of labelled examples.
The breakthrough of 2018 was transfer learning for language. First, pre-train a large model on a vast amount of ordinary text, with a simple self-supervised task such as predicting missing or next words. In doing so, the model absorbs grammar, vocabulary, facts and patterns of reasoning. Then fine-tune that same model on a small amount of labelled data for a specific task. Because the model already "knows" a great deal about language, it needs far fewer examples to learn the new job, and it does it much better.
Computer vision had used this trick since AlexNet, by pre-training on ImageNet. In 2018 it arrived for language, and the results were dramatic.
2018: ELMo, GPT and BERT
Three models in 2018 showed the power of pre-training:
- ELMo (Allen Institute for AI, early 2018) produced word representations that changed depending on context, so "bank" in "river bank" and "bank account" got different meanings.
- GPT (OpenAI, June 2018), short for Generative Pre-trained Transformer, pre-trained a Transformer to predict the next word on a collection of about 7,000 unpublished books, then fine-tuned it for different tasks.
- BERT (Google, October 2018), short for Bidirectional Encoder Representations from Transformers, was trained to fill in masked words using context from both sides of a sentence. BERT smashed records on a wide range of language understanding benchmarks. In 2019 Google began using BERT in Search, calling it one of the biggest leaps forward in the history of Search, because it helped understand the meaning of queries rather than just matching keywords.
Within months, "BERT-style" models were everywhere: in search engines, chatbots, document classification, sentiment analysis and translation. A new benchmark called GLUE, and then a harder one called SuperGLUE, were quickly approached and then surpassed by these models.
2019: GPT-2 and "too dangerous to release"
In February 2019 OpenAI announced GPT-2, with 1.5 billion parameters, trained on about 40 gigabytes of text from web pages linked from Reddit. Given a short prompt, GPT-2 could write several paragraphs of surprisingly coherent prose, from news stories to fiction.
OpenAI initially withheld the full model, saying it was concerned about misuse, such as generating fake news or spam at scale. It released progressively larger versions over the year, and the full model in November 2019, saying it had seen no strong evidence of misuse. The decision was controversial: critics called it a publicity stunt, while others saw it as a responsible first experiment in staged release. Either way, it started a debate about open versus closed AI models that continues today.
Other important 2019 models included Google's T5, which framed every language task as converting one piece of text into another, and Facebook's RoBERTa, which showed BERT could be improved simply by training it longer on more data.
Scaling laws: bigger really is better
In January 2020 researchers at OpenAI, led by Jared Kaplan, published "Scaling Laws for Neural Language Models". They found that a language model's performance improved in a smooth, predictable way as three things increased: the number of parameters, the amount of training data and the amount of computing power used. Make the model bigger, feed it more data and train it with more compute, and it gets better, along a curve you can forecast in advance.
This was hugely important. It meant AI labs could invest in ever larger training runs with some confidence about the results. In 2022 DeepMind's Chinchilla paper refined the recipe, showing that many models were too large for the amount of data they were trained on, and that a smaller model trained on more data could beat a larger one. Scaling became the central strategy of the leading AI labs, and the main reason AI training costs and data centre investments have grown so dramatically since.
2020: GPT-3 and in-context learning
In May 2020 OpenAI described GPT-3, with 175 billion parameters, more than a hundred times larger than GPT-2, trained on hundreds of billions of words from the web, books and Wikipedia.
GPT-3's most striking ability was few-shot or in-context learning. Instead of fine-tuning the model for a new task, you could simply describe the task in the prompt and give it a few examples, and it would continue the pattern. Show it three English-to-French translations and it would translate the fourth. Show it two examples of turning a description into code and it would write code. Nobody had explicitly trained it to do these things.
OpenAI made GPT-3 available through a paid API rather than releasing the model, and thousands of start-ups began building on it: copywriting tools, chatbots, coding assistants, summarisers. In 2019 Microsoft had invested $1 billion in OpenAI, and in 2020 it obtained an exclusive licence to GPT-3's underlying technology. In 2021, OpenAI's Codex, a GPT model fine-tuned on code, powered GitHub Copilot, the first widely used AI coding assistant.
GPT-3 also showed the weaknesses of pure language models. It could produce fluent nonsense, repeat biases and toxic language from its training data, and make up facts. A 2021 paper by Emily Bender, Timnit Gebru and colleagues, "On the Dangers of Stochastic Parrots", argued that ever larger language models carried environmental costs and risks of encoding bias, and that their fluency could mislead people into thinking they understood more than they did. The paper became central to debates about AI ethics.
AlphaFold: AI solves a grand challenge of biology
Proteins are the molecular machines of life, and their function depends on the complex three-dimensional shapes they fold into. Working out a protein's shape from its sequence of amino acids, the protein folding problem, had challenged scientists for 50 years. Experimental methods could take months or years and cost a great deal for a single protein.
In 2018 DeepMind's first AlphaFold won the CASP13 competition, which tests protein structure prediction. In November 2020, AlphaFold 2 stunned the CASP14 competition, predicting many structures with accuracy comparable to experiments. The competition's organisers declared that the problem had, in large part, been solved. AlphaFold 2 used an attention-based architecture related to the Transformer.
In 2021 DeepMind and the European Bioinformatics Institute (EMBL-EBI), based at the Wellcome Genome Campus near Cambridge, launched a free public database of predicted structures. By 2022 it contained over 200 million structures, covering almost every catalogued protein known to science, and it has been used by millions of researchers around the world, for example to study malaria, antibiotic resistance and plastic-eating enzymes. In 2024 Demis Hassabis and John Jumper of DeepMind shared the Nobel Prize in Chemistry with David Baker, who designs new proteins computationally.
AlphaFold showed that AI could make fundamental scientific discoveries, not just power consumer apps.
Transformers everywhere
The Transformer turned out to work on far more than text:
- Vision Transformers (Google, 2020) split images into patches and treated them like words, matching or beating convolutional networks.
- CLIP (OpenAI, January 2021) learned to connect images and text from 400 million image-caption pairs from the internet, so it could match pictures to descriptions it had never been trained on.
- Speech: OpenAI's Whisper (September 2022), trained on 680,000 hours of audio, transcribed and translated speech in many languages with high accuracy and was released openly.
- Code, music, molecules and robotics all saw Transformer-based models. DeepMind's Gato (2022) used a single Transformer to play Atari games, caption images, chat and control a robot arm.
The idea of one huge model, pre-trained on broad data and adaptable to many tasks, got a name in 2021 from Stanford researchers: foundation models.
Image generation: from text to pictures
Generative image models advanced just as quickly.
- DALL-E (OpenAI, January 2021) generated images from text descriptions, producing memorable examples such as "an armchair in the shape of an avocado".
- Diffusion models overtook GANs. They learn to generate images by reversing a process of gradually adding noise: start with random static and repeatedly "denoise" it, guided by a text description, until a picture emerges.
- DALL-E 2 (April 2022), Google's Imagen (May 2022) and Midjourney (open beta July 2022) produced strikingly realistic and artistic images.
- Stable Diffusion (August 2022), developed by researchers at LMU Munich and Runway with funding and compute from the London-based start-up Stability AI, was released with open model weights, so anyone could run it on a home computer with a decent graphics card. It unleashed a wave of creativity, and also immediate controversy.
Artists objected that their work had been used for training without permission or payment, and that the tools could imitate their styles. In early 2023 Getty Images sued Stability AI in the UK and US over the use of its photos. In the UK case, the High Court gave judgment in November 2025, largely rejecting Getty's main claims, but the wider questions of AI and copyright remain unresolved in law and policy, in the UK and around the world.
Making language models helpful: instruction tuning and RLHF
A raw language model is trained to continue text, not to follow instructions or be helpful. Ask GPT-3 a question and it might answer, or it might write more questions, or continue in an unexpected direction.
Researchers developed ways to turn these models into assistants:
- Instruction tuning: fine-tuning the model on many examples of instructions paired with good responses, so it learns to follow requests. Google's FLAN (2021) showed this greatly improved performance on new tasks.
- Reinforcement learning from human feedback (RLHF): people compare pairs of model responses and pick the better one. A separate "reward model" learns to predict these human preferences, and the language model is then trained with reinforcement learning to produce responses the reward model rates highly. RLHF built on earlier research by OpenAI and DeepMind in 2017.
In January 2022 OpenAI described InstructGPT, a version of GPT-3 trained with human feedback. Human evaluators preferred its outputs to those of the original GPT-3, even though it was more than a hundred times smaller. It was also more truthful and less toxic, though far from perfect.
Other labs pursued similar goals. Anthropic, founded in 2021 by former OpenAI researchers including siblings Dario and Daniela Amodei with a focus on AI safety, developed Constitutional AI (2022), in which a model is trained to critique and revise its own responses according to a written set of principles, reducing the amount of human labelling needed. DeepMind built Sparrow, and Google built LaMDA, a conversational model. In June 2022 a Google engineer, Blake Lemoine, claimed LaMDA was sentient; Google rejected the claim and he was dismissed. The episode showed how persuasive conversational AI had become, the ELIZA effect on a new scale.
Chain-of-thought and emergent abilities
As models grew, researchers noticed abilities that seemed to appear suddenly at larger scales, sometimes called emergent abilities, such as multi-step arithmetic. (Later research argued some of these jumps were partly an artefact of how abilities were measured.)
In 2022 Google researchers showed that chain-of-thought prompting, asking a model to "think step by step" and write out its reasoning before giving an answer, greatly improved performance on maths and logic problems. This simple idea would later become the basis of "reasoning models" that are trained to think at length before answering.
Open models and the race heats up
By 2022 a vigorous ecosystem of large models existed:
- Google's PaLM (April 2022), with 540 billion parameters, could explain jokes and solve reasoning problems.
- DeepMind's Gopher (2021) and Chinchilla (2022).
- Meta's OPT (2022), released to researchers.
- BLOOM (2022), a 176-billion-parameter open model built by the BigScience collaboration of over 1,000 researchers, coordinated by the French-American company Hugging Face and trained on a French public supercomputer.
- EleutherAI, a grassroots collective, released open alternatives to GPT-3, such as GPT-J and GPT-NeoX.
Hugging Face's platform became the place where the world shared open models and datasets, playing a role for AI similar to GitHub's for software.
The road to ChatGPT
By late 2022, all the ingredients were in place:
- a powerful pre-trained model (OpenAI's GPT-3.5 series, trained on text and code);
- instruction tuning and RLHF to make it helpful, conversational and safer;
- a simple chat interface anyone could use, free of charge.
On 30 November 2022 OpenAI released ChatGPT as a "research preview". The company reportedly did not expect it to be a big deal. Within five days it had more than a million users. The generative AI era had begun. That story is told in the next chapter.
Key takeaways
- From 2018, language models were pre-trained on huge amounts of text and then adapted to tasks: BERT and GPT led the way.
- Scaling laws (2020) showed models improve predictably with more parameters, data and compute, driving the race to build ever larger models.
- GPT-3 (2020, 175 billion parameters) could learn new tasks from a few examples in a prompt.
- AlphaFold 2 (2020) largely solved protein structure prediction, earning a 2024 Nobel Prize.
- Transformers spread to images, speech, code and robotics; diffusion models made text-to-image generation mainstream in 2022.
- Instruction tuning and RLHF turned raw language models into helpful assistants, leading to ChatGPT on 30 November 2022.
Previous chapter: The deep learning revolution, 2012-2017 · Next chapter: The generative AI boom, 2022-2024