GPT-3
Language Models are Few-Shot Learners
People can learn a new task from just a couple of examples. If someone shows you two or three sentences and their French translations, you can usually translate the next one — no long training needed. Computers used to be bad at this. To teach a model a new task, you normally had to feed it thousands of labeled examples and retrain it. This paper asks a simple question: what if we just make the model really, really big? The authors built GPT-3, a text model with 175 billion “knobs” (parameters) — about 10 times bigger than anything like it before. Then they did something surprising: they gave it new tasks with only a few examples typed right into the prompt, and changed nothing inside the model. No retraining. It just read the examples and did the task. This is called few-shot learning, and GPT-3 turned out to be shockingly good at it — even at things like simple arithmetic and writing news articles people couldn't tell were fake.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al.
175B
Parameters
175 billion tunable “knobs” — about 10× bigger than any comparable model at the time.
0
Retraining per task
GPT-3 learns each new task from examples in the prompt alone — no weight updates, no fine-tuning.
10–100
Examples in a few-shot prompt
“Few-shot” means showing the model a handful of examples (typically 10 to 100) that fit in its context window.
300B
Training text
Trained on 300 billion tokens of text — mostly filtered Common Crawl, plus web text, books, and Wikipedia.
71.2%
TriviaQA (few-shot)
Open-domain trivia questions: 64.3% zero-shot → 68.0% one-shot → 71.2% few-shot — the last a new state of the art.
86.4%
LAMBADA (few-shot)
Predicting the final word of a passage that needs whole-paragraph understanding.
8
Model sizes studied
From GPT-3 Small (125M) up to the full 175B model — used to measure how ability grows with scale.
~52%
Human-fooling news
People told real news from GPT-3-written news only about as often as a coin flip.
The one-line version
What if the model could just learn from the examples you give it?
You already do this every day. Someone shows you one or two examples of a task, and you copy the pattern. No studying required.
Old AI models couldn't. To teach one a new task, you had to collect thousands of labeled examples and retrain it. This paper — the GPT-3 paper — showed a different way. Make the model huge, then just show it a few examples in the prompt. It reads them and does the task. Nothing inside the model changes.
The old way
The problem: every new task meant a new pile of data
The best models before GPT-3 worked in two steps. First, pre-train on lots of text. Then, fine-tune: retrain the model on thousands of labeled examples for the one task you care about.
That has two downsides. You need a big labeled dataset for every task — which is expensive and often doesn't exist. And it's not how people work. A person doesn't need 10,000 examples to start a new task. A few will do.
Think of it like…
Learning to fold a paper airplane. The old way is like taking a semester-long course before your first fold. The new way is like watching a friend fold two of them, then folding your own. Same result — far less fuss.
The key idea
Zero-shot, one-shot, few-shot: how many examples do you give?
The paper describes three ways to ask the model to do a task — all just by typing into the prompt, with no retraining:
- Zero-shot: you only describe the task. (“Translate English to French: cheese →”)
- One-shot: you describe it and give one example first.
- Few-shot: you give several examples — usually 10 to 100, as many as fit in the prompt — then ask.
More examples usually means better answers. And crucially, the model's “knowledge” never changes — it's all in the reading.
Zero-shot vs. one-shot vs. few-shot — build the prompt
Translate English to French:
sea otter = loutre de mer← example
cheese = fromage← example
hello = bonjour← example
peppermint = ▮← model answers this
Description + several examples (the paper uses 10–100).
It's all one prompt— the model reads the examples and completes the last line. Nothing inside the model changes; there's no retraining between settings.
This is “in-context learning”
The model learns the task inside a single reading of the prompt — the paper calls this in-context learning. It's the same thing you do when you use a chatbot today: you describe what you want (maybe with an example or two), and it just does it.
The big bet
The trick that made it work: sheer size
Why did this suddenly work when it hadn't before? Scale. GPT-3 has 175 billion parameters — the tiny adjustable numbers a model learns. That's roughly 10× bigger than any similar model that came before it.
The authors trained a whole family of models, from small to huge, and measured them. The pattern was clear: the bigger the model, the better it got at learning from the examples in its prompt. Few-shot learning wasn't just present in the big model — it got dramatically stronger as size grew.
Watch ability grow with size
Slide from small to huge: the few-shot bar pulls further ahead of zero-shotas the model grows. That's the paper's core finding — bigger models learn from their examples better. (Curve is illustrative of the trend.)
Think of it like…
A well-read person vs. a beginner. Show a beginner two examples of a puzzle and they may still be lost. Show the same two examples to someone who has read millions of books, and they instantly catch the pattern. GPT-3's size is that broad reading — it's what makes a couple of examples enough.
Under the hood (gently)
How it was trained: predict the next word, a lot
GPT-3's training is simple to say. Show it enormous amounts of text and have it play one game over and over: guess the next word. Do that across 300 billion words' worth of text — filtered web pages (Common Crawl), more web text, books, and Wikipedia.
That's it. No task labels, no human answers. Just “predict what comes next.” Doing this at massive scale forces the model to pick up grammar, facts, reasoning patterns, translation, and much more — all as a side effect of getting better at the guessing game.
Think of it like…
Your phone's autocomplete, but supercharged. Phone autocomplete guesses your next word from a little context. GPT-3 does the same thing, trained on a huge slice of the internet — so its “guesses” can be whole correct answers, translations, or paragraphs.
What happened
The results: good at things nobody trained it to do
With just a few examples in the prompt and no retraining, GPT-3 did well across a huge range of tasks — translation, answering trivia, filling in missing words, and more. On TriviaQA, it went from 64.3% (zero-shot) to 71.2% (few-shot) — a new record for that setting.
More striking were the tasks that need reasoning on the spot: doing 3-digit arithmetic, unscrambling words, and using a made-up word in a sentence. And it could write short news articles that people couldn't reliably tell from real ones — right around a coin-flip, ~52%.
By the numbers
175B
Parameters
175 billion tunable “knobs” — about 10× bigger than any comparable model at the time.
10–100
Examples in a few-shot prompt
“Few-shot” means showing the model a handful of examples (typically 10 to 100) that fit in its context window.
71.2%
TriviaQA (few-shot)
Open-domain trivia questions: 64.3% zero-shot → 68.0% one-shot → 71.2% few-shot — the last a new state of the art.
~52%
Human-fooling news
People told real news from GPT-3-written news only about as often as a coin flip.
See more examples help
TriviaQA: Open-domain trivia. Climbs with each added example — few-shot set a new record for this setting.
It wasn't magic
The paper is honest about the limits. GPT-3 still struggled on some tasks, sometimes contradicted itself over long passages, and could absorb biases from its web training data. And because it trained on so much of the internet, some test questions may have leaked into training. Impressive — but not flawless.
The lasting idea
You don't always need to retrain a model to teach it something new — if it's big enough, you can just show it. That single idea, in-context learning through scale, is why modern AI is something you talk to in plain language instead of program. Every chatbot you use today is built on it.
Figure 1.2 in the original paper
The paper's plot showing accuracy climbing with more in-context examples and larger models — the core evidence for few-shot learning.
Brown et al., 2020 · view on arXiv
Key takeaways
- GPT-3 learns new tasks from a few examples typed into the prompt — no retraining and no weight changes (this is “in-context learning”).
- Zero-shot = a task description only; one-shot = description + 1 example; few-shot = description + several examples (typically 10–100).
- The key ingredient is scale: 175 billion parameters, ~10× bigger than earlier models.
- Bigger models use their in-prompt examples more effectively — few-shot ability grows sharply with size.
- It was trained by one simple game — predict the next word — over 300 billion words of text, with no task labels.
- It reached strong results across many tasks (e.g. TriviaQA few-shot 71.2%) and even wrote news people couldn't reliably flag as fake.
References
- Brown et al. (OpenAI) (2020). Language Models are Few-Shot Learners. NeurIPS
- Radford et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2)
- Vaswani et al. (2017). Attention Is All You Need. NeurIPS
- Kaplan et al. (2020). Scaling Laws for Neural Language Models
Keep exploring
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Ask a big model to show its work — and it starts to reason
Big language models were surprisingly bad at multi-step problems — grade-school math, logic puzzles — often blurting a wrong final answer. This paper found a shockingly simple fix that requires no training at all: in the few examples you show the model (the “prompt”), don't just show it question-and-answer pairs — show it the *reasoning in between*. Write out the step-by-step thinking that leads to each answer. The model then imitates that pattern on new problems, producing its own “chain of thought” before answering — and accuracy leaps. The headline result: prompting a 540-billion-parameter model with just eight worked examples hit 56.9% on the GSM8K math benchmark — up from 17.9% with ordinary prompting — beating the previous best (a model specially fine-tuned for the task and paired with a separate verifier, at 55%). The twist: this only works in *large* models. In small ones, showing the steps does little or even hurts. Reasoning-by-prompting is an ability that *emerges* with scale — a discovery that reshaped how people use LLMs.
QLoRA: Efficient Finetuning of Quantized LLMs
Fine-tune a 65-billion-parameter model on one gaming GPU
LoRA already made fine-tuning cheaper by training tiny add-on matrices instead of the whole model. But there was still a wall: you had to fit the giant frozen base model in GPU memory just to run it — and a 65-billion-parameter model needs over 780 GB, far beyond any single GPU. QLoRA smashes that wall by squashing the frozen base model down to 4 bits per number (instead of the usual 16), roughly a 4× shrink, so the whole thing fits in under 48 GB — a single high-end GPU. It does this without hurting quality, using three clever tricks: a new 4-bit number format (NF4) tuned to how weights are actually distributed, “double quantization” that even compresses the compression bookkeeping, and “paged optimizers” that spill memory to the CPU during brief spikes instead of crashing. The tiny LoRA add-ons still train in full precision on top. To prove it works, they fine-tuned a chatbot called Guanaco in 24 hours on one GPU — and it reached 99.3% of ChatGPT's score on a popular benchmark. QLoRA is why hobbyists and small labs can now fine-tune enormous models at home.
Retrieval-Augmented Generation for Large Language Models: A Survey
RAG — giving a language model an open book to read from
A language model only knows what it saw during training. So it can confidently make things up (“hallucinate”), it can't know anything recent, and it can't see your private documents. Retrieval-Augmented Generation (RAG) fixes this with a simple, powerful idea: before the model answers, go fetch relevant text from an external library and hand it to the model along with the question — like turning a closed-book exam into an open-book one. The basic recipe has three steps: (1) Indexing — chop your documents into chunks and store them so they're searchable by meaning; (2) Retrieval — when a question comes in, pull the few most relevant chunks; (3) Generation — feed those chunks plus the question to the model so its answer is grounded in real sources. This survey organizes the whole field into three generations of increasingly sophisticated RAG — Naive, Advanced, and Modular — and breaks every system into three parts: what you retrieve, how you use it, and how the model generates. RAG is now one of the most widely used techniques in real-world AI, because it makes answers more accurate, up-to-date, and traceable to a source.