Improving Language Understanding by Generative Pre-Training
GPT-1 — pre-train once, fine-tune for anything
In 2018, most language AI was built one task at a time: to do sentiment analysis you built a sentiment model, for question answering a question-answering model, each needing its own pile of hand-labeled examples — which are scarce and expensive. This paper (the original GPT) showed a better recipe. First, take a single Transformer and train it on a mountain of ordinary unlabeled text with one dead-simple goal: predict the next word, over and over, across 7,000+ books. That teaches it a surprising amount about grammar, facts, and reasoning — for free, no labels needed. Then, to specialize it for a real task, just keep that same model almost entirely unchanged and fine-tune it briefly on a small labeled set, feeding structured inputs (like a premise and hypothesis) as one plain sequence with a divider token. This one general model beat purpose-built, task-specific architectures on 9 of 12 benchmarks. It's the blueprint every GPT since has followed.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever
9 / 12
State-of-the-art on
One general model beat task-specific architectures on 9 of the 12 datasets studied.
+8.9%
Commonsense (Story Cloze)
Absolute improvement on the Stories Cloze Test — the largest single-task gain.
+5.7%
Question answering (RACE)
Absolute improvement on the RACE reading-comprehension benchmark.
+5.5%
GLUE benchmark
Improvement on the multi-task GLUE benchmark; +1.5% on MultiNLI entailment.
12
Transformer layers
A 12-layer decoder-only Transformer, 768-dim states, 12 attention heads, 3072-dim feed-forward.
~117M
Parameters
Tiny by today's standards — yet enough to beat bespoke task-specific models.
7,000+ books
Pretraining data
BooksCorpus — long stretches of contiguous text, letting the model learn long-range structure.
−14.8%
Remove pre-training
Ablation: without generative pre-training, performance drops 14.8% across tasks — the pre-training is doing the heavy lifting.
The one-line version
The paper that taught AI to learn from raw text
This is GPT-1 — the first “Generative Pre-Trained” Transformer, and the direct ancestor of every GPT since. Its idea is simple enough to state in a sentence: train one model to predict the next word across a huge pile of unlabeled text, then lightly fine-tune that same model for whatever specific task you need. That recipe — pre-train once, adapt for anything — is now the foundation of modern AI. Let's unpack why it works.
The problem
Labeled data is scarce; raw text is everywhere
In 2018, the dominant approach trained a fresh model for each task, from scratch, using hand-labeled examples. But labels are expensive and rare — someone has to sit and tag thousands of sentences. Meanwhile, the internet and libraries are overflowing with unlabeled text that no one was really using to teach models deep language skills.
The question this paper answers: can we soak up general language ability from all that free, unlabeled text first — and then only need a little labeled data to specialize?
Think of it like…
A broad education before a job. Instead of training someone from zero for one narrow role, you give them years of general reading first. When a specific job comes up, a short bit of on-the-job training is enough — because the general knowledge transfers. Pre-training is the education; fine-tuning is the on-the-job training.
Stage 1
Pre-training: just predict the next word
The model is trained on one beautifully simple objective: given the words so far, predict the next word. Do that billions of times across 7,000+ books, and to get good at it the model is forced to learn grammar, facts, and even bits of reasoning — because guessing the next word well often requires understanding the sentence.
Crucially, this needs no labels at all. The text itself is the answer key: the next word is always right there.
L₁(U) = Σᵢ log P( uᵢ | uᵢ₋ₖ, …, uᵢ₋₁ ; Θ )
The language-modeling objective: maximize the probability the model assigns to each actual next token uᵢ, given the previous k tokens (the context window). The conditional probability is modeled by a multi-layer Transformer decoder with masked self-attention. This is the entire pre-training signal — no human labels.
Predict the next word
Given “The”, the model predicts the next word:
Actual next word: “cat”✓ top guess
No labels anywhere — the correct next word is simply the one that comes next in the text. Repeat this billions of times across 7,000+ books and the model soaks up grammar, facts, and context. That's generative pre-training.
The model
A Transformer decoder that only looks left
The model is a 12-layer Transformer decoder — the same architecture from Attention Is All You Need, but keeping only the decoder half. Its masked self-attention means each position can only attend to earlier tokens, never future ones. That's exactly what you want for next-word prediction: no peeking at the answer.
Concretely: 768-dimensional states, 12 attention heads, a 3072-dim feed-forward inner layer, a 512-token context window, GELU activations, and a byte-pair-encoding vocabulary of 40,000 merges — about 117M parameters total.
GPT-1 at a glance
| Component | Choice |
|---|---|
| Architecture | 12-layer Transformer decoder (decoder-only) |
| Hidden size / heads | 768 dims / 12 heads |
| Feed-forward inner | 3072 dims |
| Context window | 512 tokens |
| Tokenizer | Byte-pair encoding, 40,000 merges |
| Pretraining data | BooksCorpus (7,000+ books) |
For the curious
“Decoder-only” means GPT drops the Transformer's encoder entirely and uses just the decoder stack with causal (look-left-only) masking. Later GPTs (2, 3, …) kept exactly this shape and mostly just made it bigger with more data — a testament to how right this design was.
Stage 2
Fine-tuning: same model, tiny tweak
Once pre-trained, adapting to a real task is remarkably light-touch. You add just one new layer — a linear classifier on top — and fine-tune on the task's labeled data. Almost every parameter is inherited from pre-training; the model barely changes.
One extra trick helps: during fine-tuning they keep the next-word prediction objective around as an auxiliary goal (weighted by λ). It improves generalization and speeds up convergence.
L₃(C) = L₂(C) + λ · L₁(C)
The fine-tuning objective: L₂ is the supervised task loss (predict the correct label y from the final token's representation via a new linear layer W_y), and λ·L₁ keeps the language-modeling objective as an auxiliary term. The only genuinely new parameters are W_y and the embeddings for delimiter tokens.
Pre-train once, fine-tune many
Once, expensive. The model reads mountains of raw text with a single goal — predict the next word — and absorbs general language ability. Slow and compute-heavy, but done only once.
The clever bit
Turning any task into one sequence of tokens
Here's the elegant part. Many tasks have structured inputs — a premise and a hypothesis, or a document, question, and several candidate answers. Older methods bolted on custom, task-specific architectures to handle each shape.
GPT instead flattens everything into a single sequence with special delimiter tokens, so the same unchanged model can read it. Entailment? Concatenate premise + $ + hypothesis. Multiple-choice QA? Make one sequence per answer (context + question + $ + answer) and let a softmax pick the best. Same model, just reshaped inputs.
How each task becomes a sequence
Does the hypothesis follow from the premise? Concatenate them with a delimiter.
Think of it like…
A universal intake form. Rather than building a different office for every kind of request, you convert every request — a complaint, a question, a comparison — into the same standard form, and one clerk who's read everything handles them all.
The payoff
One general model beats the specialists
The results were striking. This single task-agnostic model improved the state of the art on 9 of 12 datasets — beating models whose architectures had been hand-crafted for each specific task. Highlights (absolute gains): +8.9% on commonsense reasoning (Story Cloze), +5.7% on question answering (RACE), +5.5% on the GLUE benchmark, and +1.5% on textual entailment (MultiNLI).
And an ablation drove the point home: remove the generative pre-training and performance drops 14.8% across tasks. The free, unlabeled pre-training was doing the heavy lifting.
By the numbers
9 / 12
State-of-the-art on
One general model beat task-specific architectures on 9 of the 12 datasets studied.
+8.9%
Commonsense (Story Cloze)
Absolute improvement on the Stories Cloze Test — the largest single-task gain.
+5.7%
Question answering (RACE)
Absolute improvement on the RACE reading-comprehension benchmark.
−14.8%
Remove pre-training
Ablation: without generative pre-training, performance drops 14.8% across tasks — the pre-training is doing the heavy lifting.
Explore the gains
Absolute improvement over the previous state of the art — from a single pre-trained model, lightly fine-tuned:
Story Cloze
commonsense reasoning
RACE
question answering
GLUE
multi-task benchmark
MultiNLI
textual entailment
9 / 12
datasets improved to a new state of the art
−14.8%
drop if you remove the generative pre-training (ablation)
The pattern is consistent across categories, and the ablation makes the cause clear: the free, label-free pre-training is what carries the performance.
The diagnosis
Why does pre-training transfer so well?
The authors probed why the free pre-training helps. Two findings stand out.
More transferred layers help. Fine-tuning while carrying over more of the pre-trained layers steadily improves results — each layer holds useful, transferable knowledge, not just the early ones.
The Transformer's memory matters. Using a Transformer (rather than an LSTM) lets the model capture long-range structure, which is a big part of why it transfers so effectively. The pre-trained model even shows useful behavior on some tasks with zero fine-tuning — a hint of the “few-shot” abilities that later GPTs would make famous.
The lasting idea
Stop building a new model for every task. Train one model to predict the next word on oceans of unlabeled text, then adapt it with a light touch. That single insight — proven here on a ~117M-parameter model — scaled directly into GPT-2, GPT-3, and the assistants of today.
Key takeaways
- Two stages: unsupervised generative pre-training (predict the next word on lots of unlabeled text), then supervised fine-tuning on each task.
- Pre-training needs no labels — the text is its own answer key — which unlocks learning from vast, cheap, unlabeled corpora.
- The model is a 12-layer decoder-only Transformer with causal (look-left-only) masking — the shape every later GPT kept.
- Fine-tuning adds essentially one linear layer and keeps language modeling as an auxiliary objective (L₃ = L₂ + λ·L₁).
- Task-aware input transformations flatten structured inputs into one token sequence, so one model handles many tasks with no architecture changes.
- A single general model beat task-specific architectures on 9 of 12 benchmarks; removing pre-training cost 14.8% — it was doing the heavy lifting.
References
- Radford, Narasimhan, Salimans, Sutskever (2018). Improving Language Understanding by Generative Pre-Training. OpenAI
- Vaswani et al. (2017). Attention Is All You Need. NeurIPS
- Peters et al. (2018). Deep Contextualized Word Representations (ELMo)
Keep exploring
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Ask a big model to show its work — and it starts to reason
Big language models were surprisingly bad at multi-step problems — grade-school math, logic puzzles — often blurting a wrong final answer. This paper found a shockingly simple fix that requires no training at all: in the few examples you show the model (the “prompt”), don't just show it question-and-answer pairs — show it the *reasoning in between*. Write out the step-by-step thinking that leads to each answer. The model then imitates that pattern on new problems, producing its own “chain of thought” before answering — and accuracy leaps. The headline result: prompting a 540-billion-parameter model with just eight worked examples hit 56.9% on the GSM8K math benchmark — up from 17.9% with ordinary prompting — beating the previous best (a model specially fine-tuned for the task and paired with a separate verifier, at 55%). The twist: this only works in *large* models. In small ones, showing the steps does little or even hurts. Reasoning-by-prompting is an ability that *emerges* with scale — a discovery that reshaped how people use LLMs.
QLoRA: Efficient Finetuning of Quantized LLMs
Fine-tune a 65-billion-parameter model on one gaming GPU
LoRA already made fine-tuning cheaper by training tiny add-on matrices instead of the whole model. But there was still a wall: you had to fit the giant frozen base model in GPU memory just to run it — and a 65-billion-parameter model needs over 780 GB, far beyond any single GPU. QLoRA smashes that wall by squashing the frozen base model down to 4 bits per number (instead of the usual 16), roughly a 4× shrink, so the whole thing fits in under 48 GB — a single high-end GPU. It does this without hurting quality, using three clever tricks: a new 4-bit number format (NF4) tuned to how weights are actually distributed, “double quantization” that even compresses the compression bookkeeping, and “paged optimizers” that spill memory to the CPU during brief spikes instead of crashing. The tiny LoRA add-ons still train in full precision on top. To prove it works, they fine-tuned a chatbot called Guanaco in 24 hours on one GPU — and it reached 99.3% of ChatGPT's score on a popular benchmark. QLoRA is why hobbyists and small labs can now fine-tune enormous models at home.
Retrieval-Augmented Generation for Large Language Models: A Survey
RAG — giving a language model an open book to read from
A language model only knows what it saw during training. So it can confidently make things up (“hallucinate”), it can't know anything recent, and it can't see your private documents. Retrieval-Augmented Generation (RAG) fixes this with a simple, powerful idea: before the model answers, go fetch relevant text from an external library and hand it to the model along with the question — like turning a closed-book exam into an open-book one. The basic recipe has three steps: (1) Indexing — chop your documents into chunks and store them so they're searchable by meaning; (2) Retrieval — when a question comes in, pull the few most relevant chunks; (3) Generation — feed those chunks plus the question to the model so its answer is grounded in real sources. This survey organizes the whole field into three generations of increasingly sophisticated RAG — Naive, Advanced, and Modular — and breaks every system into three parts: what you retrieve, how you use it, and how the model generates. RAG is now one of the most widely used techniques in real-world AI, because it makes answers more accurate, up-to-date, and traceable to a source.