The DOI matches its DataCite record; 45 of the 45 references it cites were found in a registry or an index.
Fine-Tuning Language Models from Human Preferences
Open accessAbstract
Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, while the combination of RL and human feedback has been extensively studied [30,82], it is only recently that RLHF with LLMs has achieved significant success in sequence-to-sequence tasks such as Summarization [177,194,222], providing reliable answers with citations to queries [53,123], creating Helpful, Harmless and Honest dialogue agents aligned with broad human values [12,136].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [222]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It appears that with recent increases in scale, LLMs have encoded in ever more granular detail nuanced language patterns, or styles (65), that bring about these mimicking abilities, further enhanced via fine-tuning for human-like conversation through reinforcement learning from human feedback (48).”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [48]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This is the motivation for fine-tuning efforts via RLHF or other methods: to align the outputs of a pretrained model with an acceptable imitation of an ideal for an obedient assistant (Ziegler et al. 2019;Bai et al. 2022).”
From Epistemic Injustice in Generative AI (2024) · cites this paper as Ziegler et al. 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unlike traditional finetuning, RL does not require large datasets; instead, models learn from comparisons rather than absolute labels, reducing redundancy and cost (Ziegler et al., 2019).”
From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Ziegler et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Akey limitation of our study is the exclusive use of models that have undergone Reinforcement Learning with Human Feedback (RLHF; Christiano et al., 2017; Ziegler et al., 2019).”
From Simulating Opinion Dynamics with Networks of LLM-based Agents (2024) · cites this paper as Ziegler et al., 2019
In the citing paper
“Yet another approach is to fine-tune using constitutional rules provided by humans 190 .”
From Machine culture (2023) · cites this paper as [190]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Self-citationGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855
Image-Based Recommendations on Styles and Substitutes
McAuley, J., Targett, C., Shi, Q., & van den Hengel, A. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767755 · Cited by 2,348
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06724 · Cited by 2,264
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732 · Cited by 1,426
A Deep Reinforced Model for Abstractive Summarization
Paulus, R., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1705.04304 · Cited by 1,182
Semi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
Bottom-Up Abstractive Summarization
Gehrmann, S., Deng, Y., & Rush, A. M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1443 · Cited by 692
Deep Batch Active Learning by Diverse, Uncertain Gradient Lower BoundsPreprint
Ash, J. T., Zhang, C., Krishnamurthy, A., Langford, J. C., & Agarwal, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.03671 · Cited by 608
Fine-tuned Language Models for Text Classification.Preprint
Howard, J., & Ruder, S. (2018) · arXiv (Cornell University) · Cited by 357
Neural Text Summarization: A Critical Evaluation
Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1051 · Cited by 335
Learning to Generate Reviews and Discovering Sentiment
Radford, A., Jozefowicz, R., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1704.01444 · Cited by 330 · cited 2×
“Since the reward model needs to understand language, we initialize it as a random linear function of the final embedding output of the language model policy ρ following Radford et al. (2018) (see section 4.”
BackgroundMethodSelf-citationA Survey of Reinforcement Learning Informed by Natural Language
Luketina, J., Nardelli, N., Farquhar, G., Foerster, J., Andreas, J., Grefenstette, E., Whiteson, S., & Rocktäschel, T. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/880 · Cited by 279
Discriminative Batch Mode Active Learning
Guo, Y., & Schuurmans, D. (2007) · Cited by 232
Discriminative Active LearningPreprint
Gissin, D., & Shalev‐Shwartz, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.06347 · Cited by 193
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Hancock, B., Bordes, A., Mazare, P.-E., & Weston, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1358 · Cited by 161
Learning to Extract Coherent Summary via Deep Reinforcement Learning
Wu, Y., & Hu, B. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11987 · Cited by 145
TL;DR: Mining Reddit to Learn Automatic Summarization
Völske, M., Potthast, M., Syed, S., & Stein, B. (2017) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/w17-4508 · Cited by 140
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., & Picard, R. (2019) · arXiv · doi:10.48550/arxiv.1907.00456 · Cited by 126
Scalable agent alignment via reward modeling: a research direction
Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., & Legg, S. (2018) · arXiv · doi:10.48550/arxiv.1811.07871 · Cited by 121
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Nguyen, K., Daumé III, H., & Boyd-Graber, J. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1153 · Cited by 110
Get To The Point: Summarization with Pointer-Generator NetworksPreprint
See, A., Liu, P. J., & Manning, C. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.04368 · Cited by 110
Better Rewards Yield Better Summaries: Learning to Summarise Without References
Böhm, F., Gao, Y., Meyer, C. M., Shapira, O., Dagan, I., & Gurevych, I. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1307 · Cited by 105
Sample Efficient Text Summarization Using a Single Pre-Trained TransformerPreprint
Khandelwal, U., Clark, K., Jurafsky, D., & Kaiser, Ł. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08836 · Cited by 99
Dialogue Learning With Human-In-The-Loop
Li, J., Miller, A. H., Chopra, S., Ranzato, M., & Weston, J. (2016) · arXiv · doi:10.48550/arxiv.1611.09823 · Cited by 90
Learning to Understand Goal Specifications by Modelling RewardPreprint
Bahdanau, D., Hill, F., Leike, J., Hughes, E., Hosseini, A., Kohli, P., & Grefenstette, E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.01946 · Cited by 90
Irving, G., Christiano, P., & Amodei, D. (2018) · arXiv · doi:10.48550/arxiv.1805.00899 · Cited by 71
Self-citationActive Learning for Speech Recognition: the Power of GradientsPreprint
Huang, J., Child, R., Rao, V., Liu, H., Satheesh, S., & Coates, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.03226 · Cited by 47
Supervising strong learners by amplifying weak experts
Christiano, P., Shlegeris, B., & Amodei, D. (2018) · arXiv · doi:10.48550/arxiv.1810.08575 · Cited by 39
Self-citationKreutzer, J., Uyheng, J., & Riezler, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1165 · Cited by 38
Finding Generalizable Evidence by Learning to Convince Q&A Models
Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., & Cho, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1244 · Cited by 27
Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisation
Gao, Y., Meyer, C. M., Mesgar, M., & Gurevych, I. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/326 · Cited by 24
Controllable Neural Story Generation via Reinforcement Learning.Preprint
Tambwekar, P., Dhuliawala, M., Mehta, A., Martin, L. J., Harrison, B. E., & Riedl, M. O. (2018) · arXiv (Cornell University) · Cited by 23
Towards Coherent and Cohesive Long-form Text Generation
Cho, W. S., Zhang, P., Zhang, Y., Li, X., Galley, M., Brockett, C., Wang, M., & Gao, J. (2019) · doi:10.18653/v1/w19-2401 · Cited by 23
Preference-based interactive multi-document summarisation
Gao, Y., Meyer, C. M., & Gurevych, I. (2019) · Information Retrieval · doi:10.1007/s10791-019-09367-8 · Cited by 22
Generating Abstractive Summaries with Finetuned Language Models
Gehrmann, S., Ziegler, Z. M., & Rush, A. M. (2019) · International Conference on Natural Language Generation · doi:10.18653/v1/w19-8665 · Cited by 22
Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., & Hakkani-Tür, D. Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.13015 · Cited by 10
Reward Learning for Efficient Reinforcement Learning in Extractive Document SummarisationPreprint
Gao, Y., Meyer, C. M., Mesgar, M., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.12894 · Cited by 2
Research works printed but not linked: 3
- Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems , pages 4302-4310, 2017.
- Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari. In Advances in Neural Information Processing Systems , 2018. URL https://arxiv.org/abs/1811.06521 . · arXiv
- Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1645-1654. JMLR. org, 2017.
Cited by
Sorted by citations
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Self-citationTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Self-citationA Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Feuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00834-7 · Cited by 2,267
Autonomous chemical research with large language models
Boiko, D. A., MacKnight, R., Kline, B., & Gomes, G. (2023) · Nature · doi:10.1038/s41586-023-06792-0 · Cited by 1,050
Metadata
- Authors
- Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1909.08593
- OpenAlex
- W2973379954
- Cited by
- 413
- References
- 45
- Citations by year
- 2026: 19 · 2025: 89 · 2024: 95 · 2023: 109 · 2022: 33 · 2021: 41 · 2020: 25 · 2019: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- reward learning, human preference data, reinforcement learning from human feedback, language model fine-tuning, text summarization, sentiment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
