Summarization is easy to describe and difficult to specify. Accuracy, coverage, coherence, and brevity all matter, yet a next-token objective does not directly represent any of them. OpenAI’s 2020 study tested whether human preferences could supply that missing objective, and became an early, concrete demonstration of reinforcement learning from human feedback for language models.
01
The gap between imitation and quality
Language models are commonly trained to predict text written by people. That objective can produce fluent imitation, but it does not guarantee a useful answer. Training data may contain weak summaries, factual errors, bias, or stylistic choices that do not match the user’s purpose.
Summarization makes this mismatch visible because there is rarely one correct output. A good summary must preserve important facts, omit distractions, remain coherent, and use an appropriate length. Human judgment can compare those tradeoffs more naturally than a fixed lexical metric or a single reference answer.
02
A four-stage preference-learning method
The researchers began with pretrained GPT-style models and supervised fine-tuning on human-written Reddit TL;DR summaries. They then asked people to choose the better of two model-generated summaries. Those pairwise choices trained a reward model to predict which output a person would prefer.
Finally, the summarization policy was optimized against the learned reward using proximal policy optimization. A KL penalty kept the policy near its supervised starting point, limiting how far it could move while searching for outputs that earned a higher predicted score.
- Train an initial summarization policy with supervised examples.
- Collect pairwise human preferences between candidate summaries.
- Fit a reward model that predicts those preferences.
- Optimize the policy with reinforcement learning while constraining drift.
03
Human feedback is a data-quality system
The main models used roughly 65,000 comparison judgments, though the study reported useful results with substantially fewer. OpenAI hired about 80 contractors and paid by time rather than by the number of labels, reducing pressure to rush through decisions.
The team also invested in onboarding, a dedicated labeling interface, shared communication, individual conversations, and ongoing agreement checks between labelers and researchers. The process illustrates a central lesson of preference learning: a reward model inherits the standards, blind spots, and inconsistencies of the people and instructions behind its data.
04
Preference optimization beat scale alone
Human evaluators preferred summaries from a 1.3-billion-parameter model trained with feedback over those from a 12-billion-parameter model trained only with supervision. Both the 1.3B and 6.7B feedback-trained systems were also preferred to the dataset’s original human-written TL;DRs, although those references were uneven and did not represent expert-level summaries.
Some of the gain came from length preference: labelers favored longer summaries, and the policies learned to use the maximum allowed length. Controlling for length reduced the reported preference for the 6.7B system, but most of the advantage remained. This is a useful example of how models learn the full evaluation signal, including unintended incentives.
05
The learned behavior transferred to news
The feedback and reinforcement-learning stages used Reddit posts, but the resulting models were evaluated without additional fine-tuning on CNN/DailyMail news articles. Those articles were longer and stylistically different from the training task.
The models still produced strong short summaries, and the 6.7B feedback-trained system outscored the dataset’s reference summaries after accounting for length. That result suggested the policy had learned a broader notion of summary quality rather than a narrow recipe tied only to Reddit posts.
06
A reward model remains an imperfect proxy
Optimizing a policy against a learned reward works only while that reward remains aligned with real human judgment. The study varied optimization strength and found that pushing too far eventually reduced summary quality even as predicted reward increased. The best outputs sat near the high end of the reward model’s familiar training distribution rather than far beyond it.
This is an early example of reward overoptimization. A model can exploit errors or gaps in the proxy when optimization pressure becomes strong enough. Constraints, fresh human evaluation, and tests outside the reward model’s training distribution are therefore part of the training method, not optional checks added afterward.
07
What the experiment could not decide
Human feedback can optimize a defined preference, but it cannot determine whose preference should define good behavior. Summarization has relatively legible criteria; higher-stakes domains contain legitimate disagreement and affect groups whose perspectives may differ from those of researchers or contractors.
The models could still produce inaccurate or harmful summaries, and the Reddit data contained offensive and biased material. Training also required substantial computation, while the comparison did not test whether the same budget spent on expert demonstrations and supervised learning might perform better.
The broader research direction is scalable oversight: better tools for human evaluators, richer feedback such as edits and explanations, and methods for judging work that would otherwise require more expertise or time than evaluators possess.
Alpheva perspective
What we take from the work
- Preference data can outperform model scaling when the base objective does not capture the desired quality.
- Labeling instructions, training, compensation, and calibration directly shape the learned behavior.
- Track unintended preferences such as output length alongside overall judgments.
- Limit reward-model exploitation with constraints and repeated evaluation by people.
- Include affected communities when defining quality for contested or high-impact tasks.