Training Language Models to Follow Instructions with Human Feedback
InstructGPT — teaching AI to actually do what you ask
A giant language model is basically a super-powered autocomplete: it was only ever trained to guess the next word on random internet text. So when you ask it to do something, it doesn't really try to be helpful — it just continues the text in a plausible way, which can mean ignoring your request, making things up, or saying something rude. This paper fixes that with a three-step recipe. First, hire people to write good example answers and train the model to copy them. Second, have people rank several of the model's answers from best to worst, and train a second “judge” model to predict those human preferences. Third, let the main model practice, using the judge as a coach that rewards more-helpful answers. The result — called InstructGPT — follows instructions far better: people preferred the answers of a small 1.3-billion-parameter InstructGPT over the original model that was 100× bigger. It also made up facts less and was more truthful. This is the training recipe behind ChatGPT.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, et al.
100× smaller
1.3B beats 175B
People preferred answers from the 1.3-billion-parameter InstructGPT over the original 175-billion-parameter GPT-3 — a model 100× larger.
85%
Preferred over GPT-3
When compared head-to-head, labelers preferred the 175B InstructGPT's answers 85 ± 3% of the time (and 71 ± 4% over a carefully-prompted GPT-3).
21% vs 41%
Made-up facts
On tasks where the answer should stick to the given text, InstructGPT made things up about half as often as GPT-3 (21% vs 41%).
≈ 2×
Truthfulness (TruthfulQA)
InstructGPT gave truthful and informative answers about twice as often as GPT-3.
3
Training steps
Show examples (SFT) → learn human preferences (reward model) → practice with the reward as a coach (PPO).
~40
Human labelers
About 40 contractors, selected via a screening test, wrote demonstrations and ranked model outputs.
6B
Reward model size
The “judge” model that scores answers is 6B parameters — smaller 175B judges were unstable to train.
4–9
Answers ranked at once
Labelers ranked between 4 and 9 responses per prompt, which the reward model learns from as pairwise comparisons.
The one-line version
The paper that taught AI to be helpful
You've used ChatGPT. This is the paper that made something like it possible. Its idea is simple: a raw language model is trained only to guess the next word — it was never taught to actually do what you ask. This paper adds that missing piece, using feedback from real people. The method is called RLHF (reinforcement learning from human feedback), and it's now the standard way to turn a rough model into a helpful assistant.
The problem
A giant autocomplete isn't the same as a helpful assistant
A large language model is trained on a huge pile of internet text with one goal: predict the next word. That makes it a fantastically powerful autocomplete — but not necessarily a helpful one.
Ask it a question and it might continue your text in a plausible-sounding way rather than a helpful way. It can ignore your actual request, confidently make up facts, or produce rude or biased text. The paper's word for this gap is alignment: the model isn't aligned with what the user actually wants. Simply making the model bigger doesn't fix this.
Think of it like…
A brilliant new intern who has read the entire internet but has never been told what the job is. They know an enormous amount, but without guidance they'll answer the wrong question, ramble, or invent an answer to sound confident. What they need isn't more reading — it's coaching on what a good answer looks like.
Step 1 of 3
Show it good examples (supervised fine-tuning)
First, the team hired about 40 people to write high-quality example answers to a wide range of requests — the kind of answer a helpful assistant should give. Then they trained the model to imitate those examples.
This is like giving the intern a binder of model answers to study. It already helps a lot: the model starts responding in a helpful style instead of just continuing text. But hand-writing examples is slow and expensive, so it can only take you so far.
Step 2 of 3
Learn what people prefer (the reward model)
Writing perfect answers is hard. But judging which of a few answers is better is easy. So next, the model produces several answers to the same request, and a person simply ranks them from best to worst (they ranked between 4 and 9 answers at a time).
Those rankings are used to train a second model — a reward model, or “judge” — whose only job is to look at any answer and predict how much a human would like it, as a score. Now, instead of needing a person for every answer, we have an automatic stand-in for human taste.
Rank the answers — train the judge
You're the labeler. Put these answers in order, best at the top. The reward model learns from your ranking.
Sure — here's a 2-sentence summary, as you asked.
I can't help with that.
Here is a summary. Also, did you know penguins…
A clear, correct summary in exactly two sentences.
Your ranking becomes 6 comparison pairs:
Each pair says “this answer beats that one.” The reward model trains to give the winner a higher score — that's Eq. 1. Ranking a handful of answers is much easier than writing a perfect one from scratch.
loss(θ) = − log σ( r_θ(x, y_w) − r_θ(x, y_l) )
The reward model's training goal, in plain terms: for a preferred answer y_w and a worse answer y_l to the same prompt x, push the score r(y_w) to be higher than r(y_l). σ turns the score gap into a probability, and the loss rewards the model for ranking the human-preferred answer above the other. Ranking K answers gives “K choose 2” such comparison pairs.
Step 3 of 3
Let it practice with a coach (reinforcement learning)
Now the main model practices. It writes an answer, the reward model scores it, and the model adjusts to earn higher scores next time — nudging itself toward answers people prefer. This trial-and-reward loop uses a method called PPO (a reinforcement-learning algorithm).
Think of the reward model as a coach standing courtside: the player takes a shot, the coach scores it, and over many rounds the player learns what works — without a human grading every single attempt.
The three steps of RLHF
Step 1
Show examples (SFT)
About 40 people write high-quality answers. The model is trained to copy them, learning a helpful style.
Think of it like…
Learning to cook. Step 1: follow a recipe book (examples). Step 2: a friend tastes several of your dishes and ranks them, until you can guess their taste yourself. Step 3: you keep cooking and “taste-testing” against that learned preference, improving every meal — no need to bother your friend each time.
The payoff
Smaller, but far more helpful
The headline result is remarkable: people preferred the answers of the 1.3-billion-parameter InstructGPT over the original GPT-3 that was 100× bigger (175 billion). Alignment beat raw size.
Head-to-head, labelers preferred the 175B InstructGPT's answers 85% of the time over GPT-3 (and 71% over a carefully-prompted GPT-3). It also made up facts about half as often (21% vs 41% on tasks that should stick to the source) and was roughly twice as truthful on a truthfulness test — all while barely losing ground on standard benchmarks.
By the numbers
100× smaller
1.3B beats 175B
People preferred answers from the 1.3-billion-parameter InstructGPT over the original 175-billion-parameter GPT-3 — a model 100× larger.
85%
Preferred over GPT-3
When compared head-to-head, labelers preferred the 175B InstructGPT's answers 85 ± 3% of the time (and 71 ± 4% over a carefully-prompted GPT-3).
21% vs 41%
Made-up facts
On tasks where the answer should stick to the given text, InstructGPT made things up about half as often as GPT-3 (21% vs 41%).
≈ 2×
Truthfulness (TruthfulQA)
InstructGPT gave truthful and informative answers about twice as often as GPT-3.
Explore the human preferences
Head-to-head, labelers preferred the 175B InstructGPT's answers 85% of the time over GPT-3 (±3%), and 71% over a carefully-prompted GPT-3. 50% would be a coin-flip tie.
1.3B > 175B
The 1.3B InstructGPT was preferred over the 100×-larger 175B GPT-3 — alignment beat size.
A catch, and the fix
The “alignment tax” — and how they paid it down
There was a side effect. Tuning the model to be helpful made it slightly worse at some standard academic benchmarks — the authors call this the alignment tax.
Their fix: while practicing (step 3), occasionally mix back in the original “predict the next word” training. This version — PPO-ptx — keeps the model's broad skills sharp while still making it helpful, largely erasing the tax. It's a small tweak with a nice lesson: you can align a model without making it forget what it knew.
Honest limitations
InstructGPT is more helpful and truthful, but not perfect — it still makes simple mistakes, and it's aligned to the preferences of a specific small group of labelers and researchers, not humanity at large. Whose values a model should follow is a hard open question the paper is careful not to overclaim on.
The lasting idea
Don't just train a model to predict text — train it on what people actually prefer. That single shift, powered by a learned reward model and a bit of practice, is what turned raw language models into the helpful assistants we use every day.
Figure 2 in the original paper
The paper's own diagram of the three RLHF steps — supervised fine-tuning, reward-model training from human rankings, and PPO optimization.
Ouyang et al., 2022 · view on arXiv
Key takeaways
- Raw language models only learn to predict the next word — they aren't trained to be helpful, so they can ignore requests, make things up, or be rude.
- RLHF fixes this in three steps: imitate human-written examples, train a reward model on human rankings, then practice against that reward with PPO.
- Judging which answer is better is far easier for people than writing perfect answers — the reward model turns that easy judgment into an automatic score.
- Alignment can beat size: a 1.3B InstructGPT was preferred over the 100×-larger 175B GPT-3.
- InstructGPT was preferred ~85% of the time over GPT-3, made up facts about half as often (21% vs 41%), and was ~2× more truthful.
- The “alignment tax” (small drops on some benchmarks) is largely fixed by PPO-ptx — mixing pretraining back in during practice.
References
- Ouyang et al. (2022). Training language models to follow instructions with human feedback. NeurIPS
- Stiennon et al. (2020). Learning to summarize from human feedback
- Brown et al. (2020). Language Models are Few-Shot Learners (GPT-3)
Keep exploring
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Ask a big model to show its work — and it starts to reason
Big language models were surprisingly bad at multi-step problems — grade-school math, logic puzzles — often blurting a wrong final answer. This paper found a shockingly simple fix that requires no training at all: in the few examples you show the model (the “prompt”), don't just show it question-and-answer pairs — show it the *reasoning in between*. Write out the step-by-step thinking that leads to each answer. The model then imitates that pattern on new problems, producing its own “chain of thought” before answering — and accuracy leaps. The headline result: prompting a 540-billion-parameter model with just eight worked examples hit 56.9% on the GSM8K math benchmark — up from 17.9% with ordinary prompting — beating the previous best (a model specially fine-tuned for the task and paired with a separate verifier, at 55%). The twist: this only works in *large* models. In small ones, showing the steps does little or even hurts. Reasoning-by-prompting is an ability that *emerges* with scale — a discovery that reshaped how people use LLMs.
QLoRA: Efficient Finetuning of Quantized LLMs
Fine-tune a 65-billion-parameter model on one gaming GPU
LoRA already made fine-tuning cheaper by training tiny add-on matrices instead of the whole model. But there was still a wall: you had to fit the giant frozen base model in GPU memory just to run it — and a 65-billion-parameter model needs over 780 GB, far beyond any single GPU. QLoRA smashes that wall by squashing the frozen base model down to 4 bits per number (instead of the usual 16), roughly a 4× shrink, so the whole thing fits in under 48 GB — a single high-end GPU. It does this without hurting quality, using three clever tricks: a new 4-bit number format (NF4) tuned to how weights are actually distributed, “double quantization” that even compresses the compression bookkeeping, and “paged optimizers” that spill memory to the CPU during brief spikes instead of crashing. The tiny LoRA add-ons still train in full precision on top. To prove it works, they fine-tuned a chatbot called Guanaco in 24 hours on one GPU — and it reached 99.3% of ChatGPT's score on a popular benchmark. QLoRA is why hobbyists and small labs can now fine-tune enormous models at home.
Retrieval-Augmented Generation for Large Language Models: A Survey
RAG — giving a language model an open book to read from
A language model only knows what it saw during training. So it can confidently make things up (“hallucinate”), it can't know anything recent, and it can't see your private documents. Retrieval-Augmented Generation (RAG) fixes this with a simple, powerful idea: before the model answers, go fetch relevant text from an external library and hand it to the model along with the question — like turning a closed-book exam into an open-book one. The basic recipe has three steps: (1) Indexing — chop your documents into chunks and store them so they're searchable by meaning; (2) Retrieval — when a question comes in, pull the few most relevant chunks; (3) Generation — feed those chunks plus the question to the model so its answer is grounded in real sources. This survey organizes the whole field into three generations of increasingly sophisticated RAG — Naive, Advanced, and Modular — and breaks every system into three parts: what you retrieve, how you use it, and how the model generates. RAG is now one of the most widely used techniques in real-world AI, because it makes answers more accurate, up-to-date, and traceable to a source.