CoChat

RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs

Open access

The DOI matches its Crossref record; 239 of the 239 references it cites were found in a registry or an index; 149 of the 165 sources its claims cite were checked for support.

Abstract

A significant challenge in training large language models (LLMs) as effective assistants is aligning them with human preferences. Reinforcement learning from human feedback (RLHF) has emerged as a promising solution. However, our understanding of RLHF is often limited to initial design choices. This article analyzes RLHF through reinforcement learning principles, focusing on the reward model. It examines modeling choices and function approximation caveats, highlighting assumptions about reward expressivity and revealing limitations like incorrect generalization, model misspecification, and sparse feedback. A categorical review of current literature provides insights for researchers to understand the challenges of RLHF and build upon existing methods.

Overview

CoChat Summary

TL;DR

This article analyzes RLHF for LLMs through reinforcement learning principles, focusing on the reward model and its assumptions. It argues that reward-model misgeneralization, model misspecification, and sparse feedback are fundamental limitations, and surveys current methods and alternatives.

Key findings

  • RLHF’s reward model is the core component: the paper formalizes an “oracular reward” \(R^\star(c,o)\) and shows that because human feedback covers only a tiny subset of all context-output pairs, the learned RM \(R_\phi\) cannot be guaranteed to generalize out-of-distribution, leading to misgeneralization, reward hacking, and hallucinations.
  • The paper categorizes RM limitations as misgeneralization, marginalization over preferences/model misspecification, and delayed/sparse feedback; it notes that a deterministic scalar reward averages over annotators and cannot represent variability in human preferences.
  • The literature review classifies feedback into rating, ranking/preference, language, and miscellaneous feedback; it reports that ranking \(N\) outputs yields \(N(N-1)/2\) pairwise comparisons versus \(N\) training points from \(N\) ratings, and that labelers took only 3x longer to provide language feedback than preference-based feedback in one cited study.
  • Practical RLHF results cited include Llama-2-7B improving on TruthfulQA from 33% to 57% with RLHF and toxicity generation decreasing from 22% to near 0 on ToxiGen; Ouyang et al. are cited as reporting good performance with an RM 30 times smaller than the policy model.

Methods

  • Design: critical analysis and survey, not a new empirical study; it formulates text generation as a Markov decision process and adopts a Bayesian perspective on RLHF to analyze reward modeling assumptions.
  • Data/sample: no new dataset; it reviews existing RLHF literature, including feedback types, reward modeling, RLHF finetuning, practical applications, and RL-free alternatives.
  • Main analysis: theoretical examination of RM generalization using context coverage \(\kappa\) and output coverage \(\rho\), likelihood functions such as Bradley-Terry and Plackett-Luce, and policy-gradient algorithms including PPO and A2C.

Limitations

  • The paper states that its analysis focuses on deterministic rewards, not stochastic reward distributions, and that RLHF itself suffers from unstable training, hyperparameter sensitivity, high memory usage, reward overoptimization, alignment tax, low output entropy, and persistent bias/toxicity issues.
  • The paper states that alternatives to conventional RL have been evaluated only on specific domains and constrained settings, and that a comprehensive comparison of scaling effects on RLHF versus SFT models is currently lacking.
  • Evident from the text: as a survey/critical analysis, it provides no new empirical validation or benchmark results of its own; many quantitative claims depend on cited works rather than original experiments.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3743127

Cites

239 of 239 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

81 Valid · 27 Not valid · 10 Not checked · 65 Own results and interpretations

74 more claims · 43 with their cited sources checked

Create a free account to see all 183 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 70
  • Probably not a substantial claim. 54

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Research works printed but not linked: 11

  1. Andrea V. Bajcsy, Dylan P. Losey, Marcia Kilchenman O’Malley, and Anca D. Dragan. 2017. Learning robot objectives from physical human interaction. In Proceedings of the Conference on Robot Learning. Retrieved from https://api.semanticscholar.org/CorpusID:28406224
  2. Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. 2016. Unifying count-based exploration and intrinsic motivation. In Proceedings of the NIPS. Retrieved from https://api.semanticscholar.org/CorpusID:8310565
  3. 10.5555/560669 · doi:10.5555/560669
  4. DeepSeek-AI Aixin Liu Bei Feng Bing Xue Bingxuan Wang Bochao Wu Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang et al. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL]. https://arxiv.org/abs/2412.19437. arXiv:2310.09520 [cs.CL]. · arXiv
  5. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the LAW@ACL.
  6. 10.5555/1622737.1622748 · doi:10.5555/1622737.1622748
  7. Andrei Andreevich Markov. 1954. The theory of algorithms. Trudy Matematicheskogo Instituta Imeni VA Steklova 42 (1954), 3–375.
  8. A. Ng, Daishi Harada, and Stuart J. Russell. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the International Conference on Machine Learning. Retrieved from https://api.semanticscholar.org/CorpusID:5730166
  9. Silviu Pitis. 2023. Failure modes of learning reward models for LLMs and other sequence models. In Proceedings of the ICML 2023 Workshop The Many Facets of Preference-Based Learning. Retrieved from https://openreview.net/forum?id=NjOoxFRZA4
  10. Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, L’eonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, and Sylvain Gelly. 2021. What matters for on-policy deep actor-critic methods? A large-scale study. In Proceedings of the International Conference on Learning Representations. Retrieved from https://api.semanticscholar.org/CorpusID:233340556
  11. Deepak Ramachandran and Eyal Amir. 2007. Bayesian inverse reinforcement learning. In Proceedings of the IJCAI, Vol. 7. 2586–2591.

Cited by

See all 92

Metadata

Authors
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B.
Affiliations
  • University of Massachusetts Amherst (US)
  • Carnegie Mellon University (US)
  • Princeton University (US)
  • Independent Researcher
  • University of Massachusetts Amherst
Venue
ACM Computing Surveys
Published
2025-09-10
Volume
58
Issue
2
Pages
1-37
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W4411063470
Cited by
92
References
36
Citations by year
2026: 60 · 2025: 26
FWCI
34.37
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • ACM Computing Surveys (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
reinforcement learning from human feedback, large language models, reward modeling, function approximation, model misspecification, generalization, sparse feedback, human preference alignment, language model alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.