The DOI matches its Crossref record; 238 of the 238 references it cites were found in a registry or an index; 149 of the 165 sources its claims cite were checked for support.
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Open accessAbstract
A significant challenge in training large language models (LLMs) as effective assistants is aligning them with human preferences. Reinforcement learning from human feedback (RLHF) has emerged as a promising solution. However, our understanding of RLHF is often limited to initial design choices. This article analyzes RLHF through reinforcement learning principles, focusing on the reward model. It examines modeling choices and function approximation caveats, highlighting assumptions about reward expressivity and revealing limitations like incorrect generalization, model misspecification, and sparse feedback. A categorical review of current literature provides insights for researchers to understand the challenges of RLHF and build upon existing methods.
Overview
CoChat Summary
TL;DR
This article analyzes RLHF for LLMs through reinforcement learning principles, focusing on the reward model and its assumptions. It argues that reward-model misgeneralization, model misspecification, and sparse feedback are fundamental limitations, and surveys current methods and alternatives.
Key findings
- RLHF’s reward model is the core component: the paper formalizes an “oracular reward” \(R^\star(c,o)\) and shows that because human feedback covers only a tiny subset of all context-output pairs, the learned RM \(R_\phi\) cannot be guaranteed to generalize out-of-distribution, leading to misgeneralization, reward hacking, and hallucinations.
- The paper categorizes RM limitations as misgeneralization, marginalization over preferences/model misspecification, and delayed/sparse feedback; it notes that a deterministic scalar reward averages over annotators and cannot represent variability in human preferences.
- The literature review classifies feedback into rating, ranking/preference, language, and miscellaneous feedback; it reports that ranking \(N\) outputs yields \(N(N-1)/2\) pairwise comparisons versus \(N\) training points from \(N\) ratings, and that labelers took only 3x longer to provide language feedback than preference-based feedback in one cited study.
- Practical RLHF results cited include Llama-2-7B improving on TruthfulQA from 33% to 57% with RLHF and toxicity generation decreasing from 22% to near 0 on ToxiGen; Ouyang et al. are cited as reporting good performance with an RM 30 times smaller than the policy model.
Methods
- Design: critical analysis and survey, not a new empirical study; it formulates text generation as a Markov decision process and adopts a Bayesian perspective on RLHF to analyze reward modeling assumptions.
- Data/sample: no new dataset; it reviews existing RLHF literature, including feedback types, reward modeling, RLHF finetuning, practical applications, and RL-free alternatives.
- Main analysis: theoretical examination of RM generalization using context coverage \(\kappa\) and output coverage \(\rho\), likelihood functions such as Bradley-Terry and Plackett-Luce, and policy-gradient algorithms including PPO and A2C.
Limitations
- The paper states that its analysis focuses on deterministic rewards, not stochastic reward distributions, and that RLHF itself suffers from unstable training, hyperparameter sensitivity, high memory usage, reward overoptimization, alignment tax, low output entropy, and persistent bias/toxicity issues.
- The paper states that alternatives to conventional RL have been evaluated only on specific domains and constrained settings, and that a comprehensive comparison of scaling effects on RLHF versus SFT models is currently lacking.
- Evident from the text: as a survey/critical analysis, it provides no new empirical validation or benchmark results of its own; many quantitative claims depend on cited works rather than original experiments.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
81 Valid · 27 Not valid · 10 Not checked · 65 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Recently, [163] has shown that language feedback is more effective than preference-based feedback in the context of summarization systems.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > Ranking or Preference Feedback.
What the sources say
- Disputedcites [163] 163. J'er'emy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2022. Training language models with language feedback. sourceExperimental — accuracy not yet measured
In §H.2 (page unavailable), the preference-based OPT-RM method achieved a 45.1 ± 1.9% win rate against human summaries, versus 36.0 ± 1.8% for ILF trained on language-feedback refinements. Section 4.3.2 also says OPT-RM outperformed the finetuning methods; the strongest result came from combining language and binary feedback, not from language feedback alone. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“This inaccurate generalization in the RM is one of the primary causes of phenomena like”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 5.1 Limitations of the Reward Model
What the sources say
- Not matchedcites [193] 193. Lilian Weng. 2018. Policy gradient algorithms. Blog post, April 8, 2018. lilianweng.github.io (2018). https://lilianweng. github.io/posts/2018-04-08policy-gradient/
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [73] 73. Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore. 1996. Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4 (1996), 237-285. Retrieved from https://api.semanticscholar.org/CorpusID:1708582Experimental — accuracy not yet measured
The Abstract (p. 1) attributes a lower bound on some hallucinations to statistical properties of calibrated pretrained language models. Searches for reward-model generalization and reward hacking found no passage connecting either to hallucinations, so our copy does not address the attributed causal claim. (LLM, conf=0.75)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Firstly, we get more training data for the RM.In practice, people collect a ranking of 𝑁 outputs and create preference pairs [136].Collecting 𝑁 ratings for 𝑁 outputs provides we get 𝑁 training points.Ranking 𝑁 outputs provided 𝑁 (𝑁 -1)/2 pairwise comparisons.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 4.1 Oracular Reward and the Role of Human Feedback
What the sources say
- Partly supportedcites [136] 136. Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V. Hafner. 2007. Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation 11, 2 (2007), 265-286. https://doi.org/10.1109/TEVC. 2006.890271 sourceExperimental — accuracy not yet measured
Section 3.5 (page unavailable) says ranking K responses produces K(K−1)/2 comparisons used to train the reward model. It does not compare this with collecting ratings, and cautions that comparisons from one ranking are highly correlated, so the broader claim of more training data than ratings is not fully established. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Naturally, the reward function plays a critical role in the process of fine-tuning model outputs, determining practical and fundamental limits [24] of the efficacy of RLHF.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > Markov decision process.
What the sources say
- Supportedcites [24] 24. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv:2307.15217 [cs.AI]. source
Section 2, Step 3 states that RLHF fine-tunes the base model by optimizing against the reward model. The passages on evaluating reward models and on reward hacking describe practical difficulties and fundamental limits arising from that model. The supplied passages have section labels but no page numbers. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“(1) The action space is extremely large.For example, the LLaMa model [184,185] employs a vocabulary of size 32K.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > Markov decision process.
What the sources say
- Could not verifycites [184] 184. Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, and Bo Zheng. 2025. Equilibrate RLHF: Towards balancing helpfulness-safety trade-off in large language models. arXiv:2502.11555 [cs.AI]. Retrieved from https://arxiv.org/abs/2502.11555 sourceExperimental — accuracy not yet measured
The paper’s Introduction (p. 1) discusses LLM text generation, and its alignment-method sections discuss reinforcement learning, but the passages found do not describe the size of a text-generation MDP’s action space. Searches for “action space,” “vocabulary,” and “32K” also found no passage establishing the sentence’s LLaMa example. (LLM, conf=0.74)
- Could not verifycites [185] 185. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. sourceExperimental — accuracy not yet measured
The Tokenizer passage in §2.1 (p. 2) describes LLaMA’s byte-pair encoding but gives no vocabulary size. Searches for “32K” and “32,000” found no passage in our copy establishing the cited 32K example or describing the action space of a text-generation MDP. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
74 more claims · 43 with their cited sources checked
Create a free account to see all 183 claims and to verify the 2 references still open.
- It may state two separate claims as one. 70
- Probably not a substantial claim. 54
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 2 Motivation: Eliminating Objective Mismatch in Pre-Trained Language Models
“The first term (a) is the log-likelihood of the feedback dataset, specifying how a human's internal objective (reward function) governs their feedback.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 2 Motivation: Eliminating Objective Mismatch in Pre-Trained Language Models
“𝜙 MAP = arg max 𝜙 Pr(𝜙 | D HF ) = arg max 𝜙 Pr(D HF | 𝑅 𝜙 ) (a) Pr(𝜙) (b)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > Markov decision process.
“𝜌 : S → Δ(S) is the initial state distribution.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 7 Discussion and Conclusion
“The agent starts in state 𝑠 1 ∼ 𝜌 (•).”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 17,283
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Research works printed but not linked: 12
- Andrea V. Bajcsy, Dylan P. Losey, Marcia Kilchenman O’Malley, and Anca D. Dragan. 2017. Learning robot objectives from physical human interaction. In Proceedings of the Conference on Robot Learning. Retrieved from https://api.semanticscholar.org/CorpusID:28406224
- Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. 2016. Unifying count-based exploration and intrinsic motivation. In Proceedings of the NIPS. Retrieved from https://api.semanticscholar.org/CorpusID:8310565
- 10.5555/560669 · doi:10.5555/560669
- DeepSeek-AI Aixin Liu Bei Feng Bing Xue Bingxuan Wang Bochao Wu Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang et al. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL]. https://arxiv.org/abs/2412.19437. arXiv:2310.09520 [cs.CL]. · arXiv
- Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the LAW@ACL.
- 10.5555/1622737.1622748 · doi:10.5555/1622737.1622748
- Andrei Andreevich Markov. 1954. The theory of algorithms. Trudy Matematicheskogo Instituta Imeni VA Steklova 42 (1954), 3–375.
- A. Ng, Daishi Harada, and Stuart J. Russell. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the International Conference on Machine Learning. Retrieved from https://api.semanticscholar.org/CorpusID:5730166
- Silviu Pitis. 2023. Failure modes of learning reward models for LLMs and other sequence models. In Proceedings of the ICML 2023 Workshop The Many Facets of Preference-Based Learning. Retrieved from https://openreview.net/forum?id=NjOoxFRZA4
- Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, L’eonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, and Sylvain Gelly. 2021. What matters for on-policy deep actor-critic methods? A large-scale study. In Proceedings of the International Conference on Learning Representations. Retrieved from https://api.semanticscholar.org/CorpusID:233340556
- Deepak Ramachandran and Eyal Amir. 2007. Bayesian inverse reinforcement learning. In Proceedings of the IJCAI, Vol. 7. 2586–2591.
- Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. 2008. Maximum entropy inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence.
Cited by
Large language models: an overview of foundational architectures, recent trends, and a new taxonomy
Mienye, I. D., Jere, N., Obaido, G., Ogunruku, O. O., Esenogho, E., & Modisane, C. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07668-w · Cited by 46
“Reinforcement Learning with Human Feedback (RLHF) aligns LLM outputs with human preferences by incorporating human evaluations into the training loop [55, 56].”
Miedema, E., Waschull, S., & Emmanouilidis, C. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104409 · Cited by 46
Edge large language models: a comprehensive survey
Jiang, S., Zhou, X., Zhang, M., Xu, C., Liao, G., Chen, J., & Cao, J. (2026) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-025-00227-7 · Cited by 24
RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders
Yang, Z., Sun, A., Zhao, Y. X., et al. (2025) · International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC) · doi:10.1109/aiotc66747.2025.11198794 · Cited by 10
“Recent Reinforcement Learning with Human Feedback (RLHF) has shown positive results in accounting for the LLM output to human preferences in instruction-following trials[14].”
A Survey on Human Preference Learning for Aligning Large Language Models
Jiang, R., Chen, K., Bai, X., He, Z., Li, J., Yang, M., Zhao, T., Nie, L., & Zhang, M. (2025) · ACM Computing Surveys · doi:10.1145/3773279 · Cited by 85
Metadata
- Authors
- Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B.
- Affiliations
- University of Massachusetts Amherst (US)
- Carnegie Mellon University (US)
- Princeton University (US)
- Independent Researcher
- University of Massachusetts Amherst
- ORCID
- Venue
- ACM Computing Surveys
- Published
- 2025-09-10
- Volume
- 58
- Issue
- 2
- Pages
- 1-37
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4411063470
- Cited by
- 92
- References
- 36
- Citations by year
- 2026: 60 · 2025: 26
- FWCI
- 34.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- reinforcement learning from human feedback, large language models, reward modeling, function approximation, model misspecification, generalization, sparse feedback, human preference alignment, language model alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs. https://app.cochat.ai/papers/10.1145/3743127
- The paper
- Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026). RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs. ACM Computing Surveys. https://doi.org/10.1145/3743127
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.