Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Ask a big model to show its work — and it starts to reason
Big language models were surprisingly bad at multi-step problems — grade-school math, logic puzzles — often blurting a wrong final answer. This paper found a shockingly simple fix that requires no training at all: in the few examples you show the model (the “prompt”), don't just show it question-and-answer pairs — show it the *reasoning in between*. Write out the step-by-step thinking that leads to each answer. The model then imitates that pattern on new problems, producing its own “chain of thought” before answering — and accuracy leaps. The headline result: prompting a 540-billion-parameter model with just eight worked examples hit 56.9% on the GSM8K math benchmark — up from 17.9% with ordinary prompting — beating the previous best (a model specially fine-tuned for the task and paired with a separate verifier, at 55%). The twist: this only works in *large* models. In small ones, showing the steps does little or even hurts. Reasoning-by-prompting is an ability that *emerges* with scale — a discovery that reshaped how people use LLMs.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, et al.
56.9%
GSM8K, PaLM 540B + CoT
Chain-of-thought prompting with 8 exemplars on the GSM8K math word-problem benchmark — a new state of the art.
17.9%
Same model, standard prompt
PaLM 540B on GSM8K with ordinary few-shot prompting (answers only, no reasoning). Chain-of-thought more than triples it — a +39-point gain.
55%
Previous best on GSM8K
A GPT-3 model fine-tuned on GSM8K and paired with a separate verifier (Cobbe et al., 2021). Chain-of-thought beat it with zero training.
8
Examples in the prompt
Just eight hand-written worked examples were enough — no gradient updates, no fine-tuning.
~100B params
Emerges around
Below roughly 100 billion parameters, chain-of-thought gives little or even negative benefit. The ability appears only at scale.
3
Models tested
GPT-3 / InstructGPT, LaMDA (137B), and PaLM (540B) — the effect showed up across all three model families at large scale.
The one-line version
Ask the model to show its work
Large language models could recite facts and write fluently, yet they stumbled on problems needing a few steps of reasoning — grade-school math, logic puzzles, word problems. This paper's fix is almost too simple: when you give the model example problems in the prompt, include the step-by-step reasoning, not just the answer. The model copies that habit — thinking out loud before answering — and its accuracy on hard problems jumps. No training required.
The setup
Prompting, and where it fell short
Since GPT-3, the standard way to steer a big model is few-shot prompting: show it a handful of example (question → answer) pairs, then ask your real question. The model infers the pattern and responds — no fine-tuning needed.
That works beautifully for one-step tasks. But on problems that need several steps — “she had 23, used 20, bought 6 more…” — the model tends to jump straight to a final number and gets it wrong. Scaling the model up barely helped. Reasoning looked like a wall that bigger models alone couldn't climb.
The idea
A “chain of thought” is just the steps in between
A chain of thought is a series of short intermediate steps that lead to an answer — exactly what a student writes when they show their work. The insight: if your prompt's examples demonstrate those steps, the model will produce them too.
So each example changes from (question → answer) to (question → reasoning steps → answer). That's the entire method. You're not teaching new facts; you're demonstrating a format for thinking.
Standard vs. chain-of-thought prompting
Example shown to the model
Q: Roger has 5 tennis balls. He buys 2 more cans of 3 balls each. How many does he have now? A: The answer is 11.
New question → model's answer
Q: The cafeteria had 23 apples. They used 20 to make lunch and bought 6 more. How many apples do they have?
With only answers as examples, a big model often blurts a wrong number — it never “shows its work,” so multi-step arithmetic slips.
Think of it like…
A math teacher who says “show your working, not just the answer.” A student who scribbles only a final number gets multi-step problems wrong more often; the same student, asked to write each step, catches their own mistakes and lands the right answer. Chain-of-thought is that instruction, given to a language model through examples.
What the model actually does
It generates the steps, then the answer
At inference the model, having seen worked examples, first writes out its own intermediate reasoning for your new question — then states the final answer. Those written steps aren't decoration: producing them gives the model room to break a hard problem into easy pieces, so each step is a small, reliable computation instead of one giant leap.
Walk through a chain of thought
Problem
A juggler has 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there?
…
…
…
The payoff
It beat a fine-tuned, verifier-equipped model — with 8 examples
The clearest demonstration is GSM8K, a benchmark of grade-school math word problems. With ordinary prompting, the 540-billion-parameter PaLM model scored just 17.9%. With chain-of-thought prompting — the same model, just eight worked examples — it jumped to 56.9% (a +39-point leap, more than tripling accuracy).
The kicker: the previous state of the art, 55%, came from a GPT-3 model that was fine-tuned on GSM8K and paired with a separate verifier model to rank answers. Chain-of-thought beat that with zero training — just a better prompt.
The GSM8K headline
17.9%
Same model, standard prompt
PaLM 540B on GSM8K with ordinary few-shot prompting (answers only, no reasoning). Chain-of-thought more than triples it — a +39-point gain.
55%
Previous best on GSM8K
A GPT-3 model fine-tuned on GSM8K and paired with a separate verifier (Cobbe et al., 2021). Chain-of-thought beat it with zero training.
56.9%
GSM8K, PaLM 540B + CoT
Chain-of-thought prompting with 8 exemplars on the GSM8K math word-problem benchmark — a new state of the art.
8
Examples in the prompt
Just eight hand-written worked examples were enough — no gradient updates, no fine-tuning.
GSM8K: prompting vs. fine-tuning
GSM8K grade-school math word problems — accuracy (%). Higher is better.
Prompting alone — 8 examples, zero training — beat a model that was specifically fine-tuned on GSM8K and paired with a verifier. That result is what put chain-of-thought on the map.
The twist
It only works when the model is big
Here's what makes the finding deep. Chain-of-thought is not a free win for every model. On small models, adding reasoning steps does little — and sometimes makes things worse, because the model produces fluent-looking but flawed reasoning.
The benefit appears only past roughly 100 billion parameters, and grows as models get larger. The authors call this an emergent ability: a skill that's essentially absent in small models and switches on at scale. You can't extrapolate it from smaller models — it simply isn't there until it is.
Reasoning emerges with scale
GSM8K accuracy as the model grows — standard vs chain-of-thought.
At 8B chain-of-thought is actually a hair worse than standard (4.9% → 4.1%). By 540B it more than triples accuracy (17.9% → 56.9%). The gain is an emergent property of scale.
How general is it?
Not just math
The paper tested three families of tasks and chain-of-thought helped across all of them at scale:
- Arithmetic — GSM8K, SVAMP, ASDiv, AQuA, MAWPS.
- Commonsense — CommonsenseQA, StrategyQA, date and sports understanding.
- Symbolic — toy tasks like concatenating the last letters of words, or tracking a sequence of coin flips.
The symbolic tasks made a further point: chain-of-thought helped the model generalize to longer problems than any it had seen in the prompt — evidence it had picked up a procedure, not just a surface pattern.
Two ways to prompt a model
| Standard prompting | Chain-of-thought | |
|---|---|---|
| Examples show | Question → answer | Question → steps → answer |
| Model outputs | Just the answer | Reasoning, then the answer |
| Training needed | None | None |
| GSM8K (PaLM 540B) | 17.9% | 56.9% |
| Helps small models? | — | Little / can hurt |
An honest caveat from the authors
A model writing a convincing chain of thought is not proof it is truly “reasoning.” The steps can look right and still be wrong, and the method only reliably helps at large scale. Chain-of-thought is a powerful, practical prompting technique — not a claim that models think like people.
The lasting idea
Sometimes you don't need a bigger model or more training — you need to ask better. Demonstrating the intermediate steps unlocks reasoning that was latent in a large model all along. That reframing — prompt as a way to elicit hidden abilities — reshaped how the field uses language models and kicked off a wave of reasoning techniques.
Figure 1 in the original paper
The paper's side-by-side example: standard prompting gets the math wrong, while chain-of-thought prompting (showing the reasoning steps) gets it right.
Wei et al., 2022 (CC-BY 4.0) · view on arXiv
Key takeaways
- Chain-of-thought prompting adds the intermediate reasoning steps to a prompt's examples, so the model produces its own steps before answering — no training required.
- On GSM8K, PaLM 540B went from 17.9% (standard) to 56.9% (chain-of-thought) — a +39-point jump that beat the prior state of the art of 55% from a fine-tuned GPT-3 with a verifier.
- Just ~8 worked exemplars were enough to unlock the gains across GPT-3/InstructGPT, LaMDA 137B, and PaLM 540B.
- The ability is emergent: below ~100B parameters chain-of-thought helps little or even hurts; the benefit appears and grows only at large scale.
- It generalized across arithmetic, commonsense, and symbolic reasoning — and on symbolic tasks extended to longer problems than those shown, suggesting a learned procedure.
- The authors caution that fluent reasoning steps aren't proof of true reasoning; it's a prompting technique, not a claim about how models think.
References
- Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le, Zhou (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS
- Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems (GSM8K)
- Brown et al. (2020). Language Models are Few-Shot Learners (GPT-3). NeurIPS
- Wei et al. (2022). Emergent Abilities of Large Language Models. TMLR
Keep exploring
QLoRA: Efficient Finetuning of Quantized LLMs
Fine-tune a 65-billion-parameter model on one gaming GPU
LoRA already made fine-tuning cheaper by training tiny add-on matrices instead of the whole model. But there was still a wall: you had to fit the giant frozen base model in GPU memory just to run it — and a 65-billion-parameter model needs over 780 GB, far beyond any single GPU. QLoRA smashes that wall by squashing the frozen base model down to 4 bits per number (instead of the usual 16), roughly a 4× shrink, so the whole thing fits in under 48 GB — a single high-end GPU. It does this without hurting quality, using three clever tricks: a new 4-bit number format (NF4) tuned to how weights are actually distributed, “double quantization” that even compresses the compression bookkeeping, and “paged optimizers” that spill memory to the CPU during brief spikes instead of crashing. The tiny LoRA add-ons still train in full precision on top. To prove it works, they fine-tuned a chatbot called Guanaco in 24 hours on one GPU — and it reached 99.3% of ChatGPT's score on a popular benchmark. QLoRA is why hobbyists and small labs can now fine-tune enormous models at home.
Retrieval-Augmented Generation for Large Language Models: A Survey
RAG — giving a language model an open book to read from
A language model only knows what it saw during training. So it can confidently make things up (“hallucinate”), it can't know anything recent, and it can't see your private documents. Retrieval-Augmented Generation (RAG) fixes this with a simple, powerful idea: before the model answers, go fetch relevant text from an external library and hand it to the model along with the question — like turning a closed-book exam into an open-book one. The basic recipe has three steps: (1) Indexing — chop your documents into chunks and store them so they're searchable by meaning; (2) Retrieval — when a question comes in, pull the few most relevant chunks; (3) Generation — feed those chunks plus the question to the model so its answer is grounded in real sources. This survey organizes the whole field into three generations of increasingly sophisticated RAG — Naive, Advanced, and Modular — and breaks every system into three parts: what you retrieve, how you use it, and how the model generates. RAG is now one of the most widely used techniques in real-world AI, because it makes answers more accurate, up-to-date, and traceable to a source.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
The paper that introduced RAG — a model with two memories
A big language model stores everything it knows inside its own weights — call that its “memory from studying.” That memory is fuzzy: the model can't quote a source, can't easily be corrected, and forgets nothing gracefully. This 2020 paper — the one that actually coined “RAG” — gives a model a *second*, separate memory: a searchable index of all of Wikipedia (about 21 million passages). When you ask a question, a **retriever** finds the most relevant passages, and a **generator** (a seq2seq model, BART) writes the answer using both its own internal knowledge *and* those retrieved passages. The whole thing is trained end-to-end. They test two flavors — one that uses a single retrieved passage for the whole answer (RAG-Sequence), and one that can draw on a different passage for each word (RAG-Token) — and set new records on open-domain question answering, while producing answers that are more specific, more factual, and *traceable to a source*. Best of all, you can update what the model knows just by swapping the index — no retraining.