CoChat
Sign Up
PreprintCited by 465Topic ModelingReinforcement Learning in RoboticsNatural Language Processing Techniques

Fine-Tuning Language Models from Human Preferences

Open access
Ziegler, Daniel M., Stiennon, Nisan, Wu, Jeffrey, Brown, Tom B., Radford, Alec, Amodei, Dario, Christiano, Paul, Irving, Geoffrey·2019·arXiv
DOI 10.48550/arxiv.1909.08593arXiv 1909.08593
Open this paper in CoChat — free accountOpen PDF

Abstract

Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1909.08593

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RLHF with LLMs has recently achieved significant success in sequence-to-sequence tasks such as summarization, providing reliable answers with citations, and creating Helpful, Harmless and Honest dialogue agents aligned with broad human values.

    “However, while the combination of RL and human feedback has been extensively studied [30,82], it is only recently that RLHF with LLMs has achieved significant success in sequence-to-sequence tasks such as Summarization [177,194,222], providing reliable answers with citations to queries [53,123], creating Helpful, Harmless and Honest dialogue agents aligned with broad human values [12,136].”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [222]

    This source is cited alongside others for the summarization example, not for answers with citations or dialogue agents. The Abstract (p. 1) reports very good human-labeler ratings and reasonable ROUGE scores for its RL-fine-tuned summarizers. But the Conclusion (p. 13) calls the results mixed and says the summarizers were only extractive “smart copiers,” so “significant success” overstates this paper’s finding. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • With recent increases in scale, LLMs have encoded in ever more granular detail nuanced language patterns, or styles, that bring about these mimicking abilities, further enhanced via fine-tuning for human-like conversation through reinforcement learning from human feedback.

    “It appears that with recent increases in scale, LLMs have encoded in ever more granular detail nuanced language patterns, or styles (65), that bring about these mimicking abilities, further enhanced via fine-tuning for human-like conversation through reinforcement learning from human feedback (48).”

    From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [48]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Fine-tuning efforts via RLHF or other methods are motivated by aligning the outputs of a pretrained model with an acceptable imitation of an ideal for an obedient assistant.

    “This is the motivation for fine-tuning efforts via RLHF or other methods: to align the outputs of a pretrained model with an acceptable imitation of an ideal for an obedient assistant (Ziegler et al. 2019;Bai et al. 2022).”

    From Epistemic Injustice in Generative AI (2024) · cites this paper as Ziegler et al. 2019

    The Introduction (p. 2) says the authors fine-tune pretrained language models using rewards learned from human preferences, supporting the RLHF part of the claim. Its stated aims are stylistic continuation and summarization, not imitation of an ideal obedient assistant; the paper also does not establish that broader aim for other fine-tuning methods. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • RL does not require large datasets; models learn from comparisons rather than absolute labels, reducing redundancy and cost.

    “Unlike traditional finetuning, RL does not require large datasets; instead, models learn from comparisons rather than absolute labels, reducing redundancy and cost (Ziegler et al., 2019).”

    From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Ziegler et al., 2019

    Section 3.1.2 (p. 7) reports that very little human data was needed for the paper’s stylistic-continuation tasks, and Section 2.4 (p. 4) describes collecting comparison labels. The paper does not establish that RL generally needs less data than traditional fine-tuning or that using comparisons reduces redundancy and cost. Appendix B (p. 17) discusses difficulties with absolute ratings, but does not make that cost claim. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • The study exclusively used models that have undergone RLHF.

    “Akey limitation of our study is the exclusive use of models that have undergone Reinforcement Learning with Human Feedback (RLHF; Christiano et al., 2017; Ziegler et al., 2019).”

    From Simulating Opinion Dynamics with Networks of LLM-based Agents (2024) · cites this paper as Ziegler et al., 2019

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems , pages 4302-4310, 2017.
  2. Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari. In Advances in Neural Information Processing Systems , 2018. URL https://arxiv.org/abs/1811.06521 . · arXiv
  3. Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1645-1654. JMLR. org, 2017.

Cited by

Sorted by citations

Related

Authors
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G.
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1909.08593
arXiv
1909.08593
OpenAlex
W2973379954
Cited by
413
References
45
Citations by year
2026: 19 · 2025: 89 · 2024: 95 · 2023: 109 · 2022: 33 · 2021: 41 · 2020: 25 · 2019: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
reward learning, human preference data, reinforcement learning from human feedback, language model fine-tuning, text summarization, sentiment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review