Learning to Summarize from Human Feedback
Teaching a model to write summaries people actually prefer
How do you train a model to write a good summary? The usual way is to have it copy human-written summaries and then score it with ROUGE — a metric that just counts overlapping words. But word-overlap isn't the same as quality; a summary can share lots of words and still be worse. This paper takes a different route: instead of optimizing a proxy, optimize for what people actually prefer. They showed humans two summaries of the same Reddit post and asked which is better — 64,832 times. From those comparisons they trained a “reward model” that predicts human preference, then used reinforcement learning to nudge a summarizer toward summaries that score high on it. The result: their 6.7-billion-parameter model's summaries were preferred over the human-written reference summaries about 65% of the time — and it beat much larger models trained the ordinary way. This is the RLHF recipe that later powered ChatGPT.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, et al.
~65%
Preferred over human references
Human labelers preferred the 6.7B RLHF model's summaries to the human-written reference summaries about 65% of the time (even after controlling for length).
64,832
Human comparisons collected
Pairwise judgments (“which summary is better?”) on the Reddit TL;DR dataset — the fuel for the reward model.
45%
Perfect 7/7 score
The 6.7B RLHF model earned a top overall score 45% of the time, versus 20% and 23% for supervised baselines.
1.3B / 6.7B
Model sizes
Two model sizes; the 6.7B human-feedback model beat much larger models trained with supervised learning alone.
0 fine-tuning
News transfer
The TL;DR-trained model summarized CNN/DailyMail news nearly as well as the human reference — with no news-specific training.
✓
Beats ROUGE-optimization
Optimizing the learned reward model produced better summaries (per humans) than directly optimizing the ROUGE metric.
The one-line version
Don't optimize a proxy — optimize what people actually want
To train a summarizer, the standard recipe is: copy human-written summaries, then measure quality with ROUGE, a score that counts how many words your summary shares with a reference. The trouble is that word-overlap isn't quality. A summary can overlap a lot and still be worse.
This paper asks: what if we optimize directly for human preference instead? The answer became the RLHF recipe behind modern chat assistants.
The problem
The metric is not the goal
There's a mismatch at the heart of how models are trained. We train them to maximize the likelihood of human-written text and score them with metrics like ROUGE — but what we actually want is high-quality output as judged by people. Those aren't the same thing.
Copying reference text teaches the model to imitate, not to be good. And ROUGE rewards surface word-overlap, not whether the summary is accurate, clear, or captures the point. Optimize a proxy hard enough and you drift away from the real goal.
Think of it like…
Studying to match an answer key word-for-word instead of understanding the topic. You might score well on the exact test, yet be worse at the real skill. ROUGE is the answer key; actual summary quality is the skill.
Why word-overlap misleads
The city council approved the new bike lane budget after a long public debate.
The city council approved the budget after a long debate about the new lane.
After public debate, the council funded a new bike lane.
Which is the better summary? Reveal the human judgment.
The method
The three-step recipe
The whole approach is three repeating steps:
1. Collect human comparisons. Show a person a post and two candidate summaries; ask which is better. They gathered 64,832 such judgments on Reddit TL;DR. 2. Train a reward model. Train a separate model to predict which summary a human would prefer — turning scattered judgments into a single quality score. 3. Optimize the policy with RL. Use that reward model as the goal, and fine-tune the summarizer with reinforcement learning (PPO) to produce summaries the reward model rates highly.
Then repeat: the improved model generates fresh summaries, humans compare those, and the reward model gets sharper.
Walk through the feedback loop
1 · Collect human comparisons
Show a person a Reddit post and two candidate summaries. They pick which is better. Repeat 64,832 times.
The steps repeat: the improved policy generates fresh summaries, humans compare those, and the reward model sharpens — a loop, not a one-shot pipeline.
Step 2, in detail
Turning “A is better than B” into a number
A single preference (“summary A beats summary B”) isn't a score — it's a comparison. The reward model learns to assign each summary a number r such that the human-preferred one gets the higher value. It's trained so that, across all the comparison pairs, the preferred summary's score beats the other's.
Once trained, this one model can score any new summary — becoming a stand-in for a human rater that the RL step can query millions of times.
loss = − log( σ( r_j − r_k ) ) where j is the human-preferred summary
The reward model is trained so the preferred summary j scores higher than the rejected summary k. σ is the logistic function, so this loss is small exactly when r_j is comfortably above r_k. Scores are normalized so reference summaries average 0.
Be the labeler
My landlord won't return my deposit even though I left the apartment spotless and gave 60 days notice. I have photos and the move-out checklist signed by the property manager.
Which summary is better?
Step 3, in detail
Reinforcement learning — with a leash
Now fine-tune the summarizer (the “policy”) to maximize the reward model's score, using PPO. But there's a danger: if the policy chases the reward too hard, it can find weird, degenerate summaries that fool the reward model without being good — and it can drift far from sensible language.
The fix is a KL penalty: a term that keeps the new policy from straying too far from the original supervised model. It acts like a leash — explore for higher reward, but don't run off into gibberish.
R(x, y) = r_θ(x, y) − β · log[ π_RL(y|x) / π_SFT(y|x) ]
The full RL reward. The first term r_θ is the learned reward-model score. The second is the KL penalty: it grows when the RL policy π_RL diverges from the original supervised model π_SFT, discouraging reward-hacking and mode collapse. β controls how tight the leash is.
Think of it like…
A dog on a long leash in a park. You want it to explore and find the good spots (high reward), but the leash (the KL penalty) stops it from bolting into the street (degenerate, un-language-like text). Loosen or tighten the leash with β.
The evidence
Preferred over the humans it learned from
The headline result is striking: humans preferred the 6.7B human-feedback model's summaries over the human-written reference summaries about 65% of the time — even after accounting for the fact that its summaries were a bit longer. It earned a perfect 7/7 quality score 45% of the time, versus 20% and 23% for supervised baselines.
It also beat much larger models trained the ordinary supervised way, and transferred to CNN/DailyMail news — writing summaries nearly as good as the reference with no news-specific training. And optimizing the learned reward beat optimizing ROUGE directly, per human judgment.
By the numbers
~65%
Preferred over human references
Human labelers preferred the 6.7B RLHF model's summaries to the human-written reference summaries about 65% of the time (even after controlling for length).
45%
Perfect 7/7 score
The 6.7B RLHF model earned a top overall score 45% of the time, versus 20% and 23% for supervised baselines.
64,832
Human comparisons collected
Pairwise judgments (“which summary is better?”) on the Reddit TL;DR dataset — the fuel for the reward model.
0 fine-tuning
News transfer
The TL;DR-trained model summarized CNN/DailyMail news nearly as well as the human reference — with no news-specific training.
Compare the approaches
Preferred vs. reference: How often humans preferred each model's summary over the human-written reference. Above 50% means it beats the human reference. The human-feedback model leads — and on preference it clears the 50% line, meaning it beats the very human summaries it learned from.
The lasting idea
If you can't write down what “good” means, learn it from human preferences and optimize that. This paper made RLHF work on a real generation task and handed the field the exact recipe — comparisons, reward model, RL with a KL leash — that later aligned InstructGPT and ChatGPT.
Figure 1 & 2 in the original paper
See the paper's own diagrams of the human-feedback training loop and the plot where the feedback-trained model's summaries beat the human reference.
Stiennon et al., 2020 · view on arXiv
Key takeaways
- Training to copy human text and scoring with ROUGE optimizes proxies, not the real goal: summary quality as judged by people.
- RLHF here is three steps: collect human comparisons → train a reward model to predict preference → fine-tune the policy with RL (PPO).
- The reward model turns pairwise “A beats B” judgments into a single score that can rate any new summary.
- A KL penalty keeps the RL policy close to the original model, preventing reward-hacking and gibberish.
- The 6.7B human-feedback model's summaries beat the human reference ~65% of the time, and beat much larger supervised models.
- This is the RLHF recipe that later powered InstructGPT and ChatGPT — and the lesson that the metric you optimize is not the thing you want.
References
- Stiennon et al. (OpenAI) (2020). Learning to summarize from human feedback. NeurIPS
- Christiano et al. (2017). Deep Reinforcement Learning from Human Preferences
- Ziegler et al. (2019). Fine-Tuning Language Models from Human Preferences
- Schulman et al. (2017). Proximal Policy Optimization Algorithms (PPO)
Keep exploring
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Ask a big model to show its work — and it starts to reason
Big language models were surprisingly bad at multi-step problems — grade-school math, logic puzzles — often blurting a wrong final answer. This paper found a shockingly simple fix that requires no training at all: in the few examples you show the model (the “prompt”), don't just show it question-and-answer pairs — show it the *reasoning in between*. Write out the step-by-step thinking that leads to each answer. The model then imitates that pattern on new problems, producing its own “chain of thought” before answering — and accuracy leaps. The headline result: prompting a 540-billion-parameter model with just eight worked examples hit 56.9% on the GSM8K math benchmark — up from 17.9% with ordinary prompting — beating the previous best (a model specially fine-tuned for the task and paired with a separate verifier, at 55%). The twist: this only works in *large* models. In small ones, showing the steps does little or even hurts. Reasoning-by-prompting is an ability that *emerges* with scale — a discovery that reshaped how people use LLMs.
QLoRA: Efficient Finetuning of Quantized LLMs
Fine-tune a 65-billion-parameter model on one gaming GPU
LoRA already made fine-tuning cheaper by training tiny add-on matrices instead of the whole model. But there was still a wall: you had to fit the giant frozen base model in GPU memory just to run it — and a 65-billion-parameter model needs over 780 GB, far beyond any single GPU. QLoRA smashes that wall by squashing the frozen base model down to 4 bits per number (instead of the usual 16), roughly a 4× shrink, so the whole thing fits in under 48 GB — a single high-end GPU. It does this without hurting quality, using three clever tricks: a new 4-bit number format (NF4) tuned to how weights are actually distributed, “double quantization” that even compresses the compression bookkeeping, and “paged optimizers” that spill memory to the CPU during brief spikes instead of crashing. The tiny LoRA add-ons still train in full precision on top. To prove it works, they fine-tuned a chatbot called Guanaco in 24 hours on one GPU — and it reached 99.3% of ChatGPT's score on a popular benchmark. QLoRA is why hobbyists and small labs can now fine-tune enormous models at home.
Retrieval-Augmented Generation for Large Language Models: A Survey
RAG — giving a language model an open book to read from
A language model only knows what it saw during training. So it can confidently make things up (“hallucinate”), it can't know anything recent, and it can't see your private documents. Retrieval-Augmented Generation (RAG) fixes this with a simple, powerful idea: before the model answers, go fetch relevant text from an external library and hand it to the model along with the question — like turning a closed-book exam into an open-book one. The basic recipe has three steps: (1) Indexing — chop your documents into chunks and store them so they're searchable by meaning; (2) Retrieval — when a question comes in, pull the few most relevant chunks; (3) Generation — feed those chunks plus the question to the model so its answer is grounded in real sources. This survey organizes the whole field into three generations of increasingly sophisticated RAG — Naive, Advanced, and Modular — and breaks every system into three parts: what you retrieve, how you use it, and how the model generates. RAG is now one of the most widely used techniques in real-world AI, because it makes answers more accurate, up-to-date, and traceable to a source.