CoChat
Sign Up
PreprintCited by 550Topic ModelingAdvanced Text Analysis TechniquesNatural Language Processing Techniques

Learning to summarize from human feedback

Open access
Stiennon, Nisan, Ouyang, Long, Wu, Jeff, Ziegler, Daniel M., Lowe, Ryan, Voss, Chelsea, Radford, Alec, Amodei, Dario +1 more·2020·arXiv
DOI 10.48550/arxiv.2009.01325arXiv 2009.01325
Open this paper in CoChat — free accountOpen PDF

Abstract

As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles, producing summaries nearly as good as the human reference without any news-specific fine-tuning. We conduct extensive analyses to understand our human feedback dataset and fine-tuned models We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2009.01325

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In both consultancy and RLHF, a single non-expert human interacts with an unreliable expert.

    “In both consultancy and RLHF, a single (non-expert) human interacts with an expert that is unreliable”

    From Debate Helps Supervise Unreliable Experts (2023) · cites this paper as Stiennon et al., 2020

    Appendix C.2 (p. 19) says the RLHF study usually collected one human label per comparison of model-generated summaries. The Discussion (p. 9) also discusses the difficulty of evaluating model outputs. The paper does not describe consultancy or establish that, in either approach, a non-expert human interacts with an unreliable expert. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Large language models are probabilistic generators of token distributions in big public text corpora, with tokens being words, word fragments, or characters.

    “LLMs are generative mathematical models of the statistical distribution of tokens in the vast public corpus of human-generated text, where the tokens in question include words, parts of words, or individual charactersincluding punctuation marks.”

    From Talking about Large Language Models (2024) · cites this paper as [20]

    Section 3.4 (p. 5) says the models predict the next token in a large text corpus, and Appendix B.1 (p. 17) identifies the corpus sources and says the models use byte-pair encoding. The paper does not establish that the entire corpus is public or explicitly describe tokens as words, word fragments, individual characters, and punctuation marks. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Starting with fine-tuning on high-quality human demonstrations results in significant improvements over starting with PLMs.

    “Empirical studies have demonstrated that starting with fine-tuning on high-quality human demonstrations results in significant improvements over starting with PLMs [136,177].”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [177]

    The Introduction (p. 1) says supervised fine-tuning on human demonstrations has markedly improved performance, and §3.4 (p. 5) describes fine-tuned and pretrained zero-shot models. But Appendix D (p. 28) explicitly says the authors did not collect a comparable high-quality-demonstrations baseline. The paper therefore does not establish the claimed significant improvement from starting with high-quality demonstrations rather than a pretrained language model. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Reinforcement learning approaches differ from SELF-REFINE in that the model cannot access feedback on an intermediate generation.

    “These methods differ from SELF-REFINE (and more generally, refinement-based approaches) in that the model cannot access feedback on an intermediate generation.”

    From Self-Refine: Iterative Refinement with Self-Feedback (2023) · cites this paper as Stiennon et al., 2020

    The Introduction (p. 2) says this paper’s RL policy is updated using a reward for the entire generated summary, and a note on p. 6 explicitly says the reward model gives no rewards at intermediate time steps. That supports the distinction for this method, but not the broader claim that reinforcement-learning approaches generally cannot access feedback on intermediate generations. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The bias is a risk for safety and alignment methods such as reward modeling and constitutional AI because the reward model gives higher scores to models similar to itself, leading to weaker oversight and supervision.

    “The bias is also a risk for methods designed for safety and alignment, such as reward modeling (Leike et al., 2018;Stiennon et al., 2020) and constitutional AI (Bai et al., 2022), for similar reasons: the reward model gives higher scores to models similar to itself, leading to weaker oversight and supervision.”

    From LLM Evaluators Recognize and Favor Their Own Generations (2024) · cites this paper as Stiennon et al., 2020

    Section 3.4 (pp. 5–6) describes reward modeling as a method for training a policy to produce outputs judged better by humans. Section 4.3 (p. 7) shows a risk: with further optimization, reward-model predictions diverge from human preferences. It does not say that reward models favor models similar to themselves or that this causes weaker oversight and supervision. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. [1]D. Bahdanau, P. Brakel, K. Xu, A. Goyal, R. Lowe, J. Pineau, A. Courville, and Y. Bengio. An actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086 , 2016. · arXiv
  2. [2]B. T. Bartell, G. W. Cottrell, and R. K. Belew. Automatic combination of multiple ranked retrieval systems. In SIGIR'94 , pages 173-181. Springer, 1994.
  3. [3]F. Böhm, Y. Gao, C. M. Meyer, O. Shapira, I. Dagan, and I. Gurevych. Better rewards yield better summaries: Learning to summarise without references. arXiv preprint arXiv:1909.01214 , 2019. · arXiv
  4. [5]S. Cabi, S. Gómez Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Y. Aytar, D. Budden, M. Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. arXiv , pages arXiv-1909, 2019.
  5. [6]A. T. Chaganty, S. Mussman, and P. Liang. The price of debiasing automatic metrics in natural language evaluation. arXiv preprint arXiv:1807.02202 , 2018. · arXiv
  6. [7]W. S. Cho, P. Zhang, Y. Zhang, X. Li, M. Galley, C. Brockett, M. Wang, and J. Gao. Towards coherent and cohesive long-form text generation. arXiv preprint arXiv:1811.00511 , 2018. · arXiv
  7. [8]S. Chopra, M. Auli, and A. M. Rush. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 93-98, 2016.
  8. [9]P. Christiano, B. Shlegeris, and D. Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575 , 2018. · arXiv
  9. [10]P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems , pages 4299-4307, 2017.
  10. [11]P. Covington, J. Adams, and E. Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems , pages 191-198, 2016.
  11. [12]A. M. Dai and Q. V. Le. Semi-supervised sequence learning. In Advances in neural information processing systems , pages 3079-3087, 2015.
  12. [13]J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305 , 2020. · arXiv
  13. [15]Y. Dong, Y. Shen, E. Crawford, H. van Hoof, and J. C. K. Cheung. Banditsum: Extractive summarization as a contextual bandit. arXiv preprint arXiv:1809.09672 , 2018. · arXiv
  14. [16]B. Dorr, D. Zajic, and R. Schwartz. Hedge trimmer: A parse-and-trim approach to headline generation. In Proceedings of the HLT-NAACL 03 on Text summarization workshop-Volume 5 , pages 1-8. Association for Computational Linguistics, 2003.
  15. [17]S. Fidler et al. Teaching machines to describe images with natural language feedback. In Advances in Neural Information Processing Systems , pages 5068-5078, 2017.
  16. [18]N. Fuhr. Optimum polynomial retrieval functions based on the probability ranking principle. ACM Transactions on Information Systems (TOIS) , 7(3):183-204, 1989.
  17. [19]Y. Gao, C. M. Meyer, M. Mesgar, and I. Gurevych. Reward learning for efficient reinforcement learning in extractive document summarisation. arXiv preprint arXiv:1907.12894 , 2019. · arXiv
  18. [20]X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256, 2010.
  19. [22]K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems , pages 1693-1701, 2015.
  20. [24]B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in neural information processing systems , pages 8011-8023, 2018.
  21. [25]N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456 , 2019. · arXiv
  22. [26]N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning , pages 1645-1654. PMLR, 2017.
  23. [28]H. J. Jeon, S. Milli, and A. D. Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. arXiv preprint arXiv:2002.04833 , 2020. · arXiv
  24. [29]T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining , pages 133-142, 2002.
  25. [33]W. Kryscinski, N. S. Keskar, B. McCann, C. Xiong, and R. Socher. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 540-551, 2019.
  26. [34]C. Lawrence and S. Riezler. Improving a neural semantic parser by counterfactual learning from human bandit feedback. arXiv preprint arXiv:1805.01252 , 2018. · arXiv
  27. [35]J. Leike, D. Krueger, T. Everitt, M. Martic, V . Maini, and S. Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871 , 2018. · arXiv
  28. [37]M. Li, J. Weston, and S. Roller. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087 , 2019. · arXiv
  29. [39]C.-Y. Lin and F. J. Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics , page 605. Association for Computational Linguistics, 2004.
  30. [40]T.-Y. Liu. Learning to rank for information retrieval . Springer Science & Business Media, 2011.
  31. [42]B. McCann, N. S. Keskar, C. Xiong, and R. Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730 , 2018. · arXiv
  32. [44]T. Niu and M. Bansal. Polite dialogue generation without parallel data. Transactions of the Association for Computational Linguistics , 6:373-389, 2018.
  33. [45]R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304 , 2017. · arXiv
  34. [46]E. Perez, S. Karamcheti, R. Fergus, J. Weston, D. Kiela, and K. Cho. Finding generalizable evidence by learning to convince q&a models. arXiv preprint arXiv:1909.05863 , 2019. · arXiv
  35. [48]A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
  36. [50]M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732 , 2015. · arXiv
  37. [52]S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635, 2011.
  38. [53]S. Rothe, S. Narayan, and A. Severyn. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics , 2020.
  39. [54]A. M. Rush, S. Chopra, and J. Weston. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685 , 2015. · arXiv
  40. [56]F. Schmidt. Generalization in generation: A closer look at exposure bias. arXiv preprint arXiv:1910.00292 , 2019. · arXiv
  41. [57]J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR) , 2016.
  42. [59]A. See, P. J. Liu, and C. D. Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368 , 2017. · arXiv
  43. [60]K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450 , 2019. · arXiv
  44. [61]P. Tambwekar, M. Dhuliawala, A. Mehta, L. J. Martin, B. Harrison, and M. O. Riedl. Controllable neural story generation via reinforcement learning. arXiv preprint arXiv:1809.10736 , 2018. · arXiv
  45. [62]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998-6008, 2017.
  46. [63]M. Völske, M. Potthast, S. Syed, and B. Stein. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization , pages 59-63, 2017.
  47. [64]S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319 , 2019. · arXiv
  48. [65]Y. Wu and B. Hu. Learning to extract coherent summary via deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.
  49. [66]Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 , 2016. · arXiv
  50. [67]Y. Yan, W. Qi, Y. Gong, D. Liu, N. Duan, J. Chen, R. Zhang, and M. Zhou. Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training. arXiv preprint arXiv:2001.04063 , 2020. · arXiv
  51. [68]S. Yi, R. Goel, C. Khatri, A. Cervone, T. Chung, B. Hedayatnia, A. Venkatesh, R. Gabriel, and D. Hakkani-Tur. Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. arXiv preprint arXiv:1904.13015 , 2019. · arXiv
  52. [69]H. Zhang, D. Duckworth, D. Ippolito, and A. Neelakantan. Trading off diversity and quality in natural language generation. arXiv preprint arXiv:2004.10450 , 2020. · arXiv
  53. [70]J. Zhang, Y. Zhao, M. Saleh, and P. J. Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777 , 2019. · arXiv
  54. [71]Y. Zhang, D. Li, Y. Wang, Y. Fang, and W. Xiao. Abstract text summarization with a convolutional seq2seq model. Applied Sciences , 9(8):1665, 2019.
  55. [72]W. Zhou and K. Xu. Learning to compare for better training and evaluation of open domain natural language generation models. arXiv preprint arXiv:2002.05058 , 2020. · arXiv

Cited by

Sorted by citations

Authors
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P.
ORCID
Wu, Jeff
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
33
Pages
3008-3021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2009.01325
arXiv
2009.01325
OpenAlex
W3082115681
Cited by
550
References
13
Citations by year
2026: 8 · 2025: 102 · 2024: 172 · 2023: 192 · 2022: 31 · 2021: 37 · 2020: 8
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • TIB Data Manager
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
abstractive summarization, human preference modeling, reward modeling, reinforcement learning from human feedback, summary quality, ROUGE, transfer learning, TL

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review