The DOI matches its Crossref record; 238 of the 238 references it cites were found in a registry or an index; 149 of the 165 sources its claims cite were checked for support.
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Open accessAbstract
A significant challenge in training large language models (LLMs) as effective assistants is aligning them with human preferences. Reinforcement learning from human feedback (RLHF) has emerged as a promising solution. However, our understanding of RLHF is often limited to initial design choices. This article analyzes RLHF through reinforcement learning principles, focusing on the reward model. It examines modeling choices and function approximation caveats, highlighting assumptions about reward expressivity and revealing limitations like incorrect generalization, model misspecification, and sparse feedback. A categorical review of current literature provides insights for researchers to understand the challenges of RLHF and build upon existing methods.
Overview
CoChat Summary
TL;DR
This article analyzes RLHF for LLMs through reinforcement learning principles, focusing on the reward model and its assumptions. It argues that reward-model misgeneralization, model misspecification, and sparse feedback are fundamental limitations, and surveys current methods and alternatives.
Key findings
- RLHF’s reward model is the core component: the paper formalizes an “oracular reward” \(R^\star(c,o)\) and shows that because human feedback covers only a tiny subset of all context-output pairs, the learned RM \(R_\phi\) cannot be guaranteed to generalize out-of-distribution, leading to misgeneralization, reward hacking, and hallucinations.
- The paper categorizes RM limitations as misgeneralization, marginalization over preferences/model misspecification, and delayed/sparse feedback; it notes that a deterministic scalar reward averages over annotators and cannot represent variability in human preferences.
- The literature review classifies feedback into rating, ranking/preference, language, and miscellaneous feedback; it reports that ranking \(N\) outputs yields \(N(N-1)/2\) pairwise comparisons versus \(N\) training points from \(N\) ratings, and that labelers took only 3x longer to provide language feedback than preference-based feedback in one cited study.
- Practical RLHF results cited include Llama-2-7B improving on TruthfulQA from 33% to 57% with RLHF and toxicity generation decreasing from 22% to near 0 on ToxiGen; Ouyang et al. are cited as reporting good performance with an RM 30 times smaller than the policy model.
Methods
- Design: critical analysis and survey, not a new empirical study; it formulates text generation as a Markov decision process and adopts a Bayesian perspective on RLHF to analyze reward modeling assumptions.
- Data/sample: no new dataset; it reviews existing RLHF literature, including feedback types, reward modeling, RLHF finetuning, practical applications, and RL-free alternatives.
- Main analysis: theoretical examination of RM generalization using context coverage \(\kappa\) and output coverage \(\rho\), likelihood functions such as Bradley-Terry and Plackett-Luce, and policy-gradient algorithms including PPO and A2C.
Limitations
- The paper states that its analysis focuses on deterministic rewards, not stochastic reward distributions, and that RLHF itself suffers from unstable training, hyperparameter sensitivity, high memory usage, reward overoptimization, alignment tax, low output entropy, and persistent bias/toxicity issues.
- The paper states that alternatives to conventional RL have been evaluated only on specific domains and constrained settings, and that a comprehensive comparison of scaling effects on RLHF versus SFT models is currently lacking.
- Evident from the text: as a survey/critical analysis, it provides no new empirical validation or benchmark results of its own; many quantitative claims depend on cited works rather than original experiments.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
81 Valid · 27 Not valid · 10 Not checked · 65 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Recently, [163] has shown that language feedback is more effective than preference-based feedback in the context of summarization systems.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > Ranking or Preference Feedback.
What the sources say
- Disputedcites [163] 163. J'er'emy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2022. Training language models with language feedback. sourceExperimental — accuracy not yet measured
In §H.2 (page unavailable), the preference-based OPT-RM method achieved a 45.1 ± 1.9% win rate against human summaries, versus 36.0 ± 1.8% for ILF trained on language-feedback refinements. Section 4.3.2 also says OPT-RM outperformed the finetuning methods; the strongest result came from combining language and binary feedback, not from language feedback alone. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“This inaccurate generalization in the RM is one of the primary causes of phenomena like”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 5.1 Limitations of the Reward Model
What the sources say
- Not matchedcites [193] 193. Lilian Weng. 2018. Policy gradient algorithms. Blog post, April 8, 2018. lilianweng.github.io (2018). https://lilianweng. github.io/posts/2018-04-08policy-gradient/
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [73] 73. Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore. 1996. Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4 (1996), 237-285. Retrieved from https://api.semanticscholar.org/CorpusID:1708582Experimental — accuracy not yet measured
The Abstract (p. 1) attributes a lower bound on some hallucinations to statistical properties of calibrated pretrained language models. Searches for reward-model generalization and reward hacking found no passage connecting either to hallucinations, so our copy does not address the attributed causal claim. (LLM, conf=0.75)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Firstly, we get more training data for the RM.In practice, people collect a ranking of 𝑁 outputs and create preference pairs [136].Collecting 𝑁 ratings for 𝑁 outputs provides we get 𝑁 training points.Ranking 𝑁 outputs provided 𝑁 (𝑁 -1)/2 pairwise comparisons.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 4.1 Oracular Reward and the Role of Human Feedback
What the sources say
- Partly supportedcites [136] 136. Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V. Hafner. 2007. Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation 11, 2 (2007), 265-286. https://doi.org/10.1109/TEVC. 2006.890271 sourceExperimental — accuracy not yet measured
Section 3.5 (page unavailable) says ranking K responses produces K(K−1)/2 comparisons used to train the reward model. It does not compare this with collecting ratings, and cautions that comparisons from one ranking are highly correlated, so the broader claim of more training data than ratings is not fully established. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Naturally, the reward function plays a critical role in the process of fine-tuning model outputs, determining practical and fundamental limits [24] of the efficacy of RLHF.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > Markov decision process.
What the sources say
- Supportedcites [24] 24. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv:2307.15217 [cs.AI]. source
Section 2, Step 3 states that RLHF fine-tunes the base model by optimizing against the reward model. The passages on evaluating reward models and on reward hacking describe practical difficulties and fundamental limits arising from that model. The supplied passages have section labels but no page numbers. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“(1) The action space is extremely large.For example, the LLaMa model [184,185] employs a vocabulary of size 32K.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > Markov decision process.
What the sources say
- Could not verifycites [184] 184. Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, and Bo Zheng. 2025. Equilibrate RLHF: Towards balancing helpfulness-safety trade-off in large language models. arXiv:2502.11555 [cs.AI]. Retrieved from https://arxiv.org/abs/2502.11555 sourceExperimental — accuracy not yet measured
The paper’s Introduction (p. 1) discusses LLM text generation, and its alignment-method sections discuss reinforcement learning, but the passages found do not describe the size of a text-generation MDP’s action space. Searches for “action space,” “vocabulary,” and “32K” also found no passage establishing the sentence’s LLaMa example. (LLM, conf=0.74)
- Could not verifycites [185] 185. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. sourceExperimental — accuracy not yet measured
The Tokenizer passage in §2.1 (p. 2) describes LLaMA’s byte-pair encoding but gives no vocabulary size. Searches for “32K” and “32,000” found no passage in our copy establishing the cited 32K example or describing the action space of a text-generation MDP. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
74 more claims · 43 with their cited sources checked
Create a free account to see all 183 claims and to verify the 2 references still open.
- It may state two separate claims as one. 70
- Probably not a substantial claim. 54
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 2 Motivation: Eliminating Objective Mismatch in Pre-Trained Language Models
“The first term (a) is the log-likelihood of the feedback dataset, specifying how a human's internal objective (reward function) governs their feedback.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 2 Motivation: Eliminating Objective Mismatch in Pre-Trained Language Models
“𝜙 MAP = arg max 𝜙 Pr(𝜙 | D HF ) = arg max 𝜙 Pr(D HF | 𝑅 𝜙 ) (a) Pr(𝜙) (b)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > Markov decision process.
“𝜌 : S → Δ(S) is the initial state distribution.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 7 Discussion and Conclusion
“The agent starts in state 𝑠 1 ∼ 𝜌 (•).”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 17,283
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Research works printed but not linked: 12
- Andrea V. Bajcsy, Dylan P. Losey, Marcia Kilchenman O’Malley, and Anca D. Dragan. 2017. Learning robot objectives from physical human interaction. In Proceedings of the Conference on Robot Learning. Retrieved from https://api.semanticscholar.org/CorpusID:28406224
- Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. 2016. Unifying count-based exploration and intrinsic motivation. In Proceedings of the NIPS. Retrieved from https://api.semanticscholar.org/CorpusID:8310565
- 10.5555/560669 · doi:10.5555/560669
- DeepSeek-AI Aixin Liu Bei Feng Bing Xue Bingxuan Wang Bochao Wu Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang et al. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL]. https://arxiv.org/abs/2412.19437. arXiv:2310.09520 [cs.CL]. · arXiv
- Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the LAW@ACL.
- 10.5555/1622737.1622748 · doi:10.5555/1622737.1622748
- Andrei Andreevich Markov. 1954. The theory of algorithms. Trudy Matematicheskogo Instituta Imeni VA Steklova 42 (1954), 3–375.
- A. Ng, Daishi Harada, and Stuart J. Russell. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the International Conference on Machine Learning. Retrieved from https://api.semanticscholar.org/CorpusID:5730166
- Silviu Pitis. 2023. Failure modes of learning reward models for LLMs and other sequence models. In Proceedings of the ICML 2023 Workshop The Many Facets of Preference-Based Learning. Retrieved from https://openreview.net/forum?id=NjOoxFRZA4
- Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, L’eonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, and Sylvain Gelly. 2021. What matters for on-policy deep actor-critic methods? A large-scale study. In Proceedings of the International Conference on Learning Representations. Retrieved from https://api.semanticscholar.org/CorpusID:233340556
- Deepak Ramachandran and Eyal Amir. 2007. Bayesian inverse reinforcement learning. In Proceedings of the IJCAI, Vol. 7. 2586–2591.
- Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. 2008. Maximum entropy inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence.
Cited by
Large language models: an overview of foundational architectures, recent trends, and a new taxonomy
Mienye, I. D., Jere, N., Obaido, G., Ogunruku, O. O., Esenogho, E., & Modisane, C. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07668-w · Cited by 46
“Reinforcement Learning with Human Feedback (RLHF) aligns LLM outputs with human preferences by incorporating human evaluations into the training loop [55, 56].”
Miedema, E., Waschull, S., & Emmanouilidis, C. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104409 · Cited by 46
Edge large language models: a comprehensive survey
Jiang, S., Zhou, X., Zhang, M., Xu, C., Liao, G., Chen, J., & Cao, J. (2026) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-025-00227-7 · Cited by 24
RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders
Yang, Z., Sun, A., Zhao, Y. X., et al. (2025) · International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC) · doi:10.1109/aiotc66747.2025.11198794 · Cited by 10
“Recent Reinforcement Learning with Human Feedback (RLHF) has shown positive results in accounting for the LLM output to human preferences in instruction-following trials[14].”
A Survey on Human Preference Learning for Aligning Large Language Models
Jiang, R., Chen, K., Bai, X., He, Z., Li, J., Yang, M., Zhao, T., Nie, L., & Zhang, M. (2025) · ACM Computing Surveys · doi:10.1145/3773279 · Cited by 85
Stabilizing Cloud Elastic Scaling with Risk-Constrained Reinforcement Learning Under Workload Drift
Huang, W., Wei, R., Kou, J., et al. (2026) · 2026 IEEE 8th International Conference on Communications, Information System and Computer Engineering (CISCE) · doi:10.1109/cisce69494.2026.11504526 · Cited by 9
Linguistic features of AI mis/disinformation and the detection limits of LLMs
Ma, Y., Zhang, X., Ren, J., Wang, R., Wang, M., & Chen, Y. (2025) · Nature Communications · doi:10.1038/s41467-025-67145-1 · Cited by 16
“RLHF is used to reward models to output more favored and polite responses to humans, and existing research shows that it may lead to more positive emotional responses 33,34 .”
AI-driven design: powered by large language model and intelligent computation
Sa, G., Li, Z., Liu, Z., & Tan, J. (2025) · International Journal of Extreme Manufacturing · doi:10.1088/2631-7990/adea23 · Cited by 8
“…address these issues through various approaches: suppressing large model hallucinations through Reinforcement Learning from Human Feedback (RLHF) [35,36] , improving geometric generation controllability through conditional diffusion models and hierarchical generation architectures [37] , and…”
Trustworthy AI in digital health: a comprehensive review of robustness and explainability
Mamun, A., Soumma, S. B., & Ghasemzadeh, H. G. (2026) · Progress in Biomedical Engineering · doi:10.1088/2516-1091/ae4e74 · Cited by 11
“Researchers are focusing on methods for detecting and mitigating biases, ensuring that LLMs treat all users equitably [40, 54, 191].”
A survey on large language models unlearning: taxonomy, evaluations, and future directions
Le-Khac, U. N., & Truong, V. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11376-7 · Cited by 16
“…binary rating system (Li et al. 2016; Scheurer et al. 2023), or the ranking system where human experts pick out the most appropriate and preferable response among a set of instructions (Ziegler et al. 2019), and can be manually created or automatically generated using LLMs (Chaudhari et al. 2024).”
Towards neuroadaptive chatbots: a feasibility study
Gherman, D. E., & Zander, T. O. (2025) · Frontiers in Neuroergonomics · doi:10.3389/fnrgo.2025.1589734 · Cited by 5
Artificial Creativity: from predictive AI to Generative System 3
Chávez-Autor, J. C. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1654716 · Cited by 3
Deep Reinforcement Learning in the Era of Foundation Models: A Survey
Mienye, I. D., Esenogho, E., & Modisane, C. (2026) · Computers · doi:10.3390/computers15010040 · Cited by 13
Indirect prompt injection in large language models
Milani, A., Franzoni, V., & Florindi, E. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12266-x · Cited by 9
“This suggests that models with extensive safety alignment based on Reinforcement Learning from Human Feed-back (RLHF) [32, 33] are more adept at prioritizing system-level instructions over document-embedded content.”
How secure is forgetting? Linking machine unlearning to machine learning attacks
P., M. S. K., Nicolazzo, S., Nocera, A., & Vinod, P. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131971 · Cited by 11
Yaacoub, J.-P. A., Noura, H., Salman, O., & Pujolle, G. (2025) · Annals of Telecommunications · doi:10.1007/s12243-025-01134-9 · Cited by 4
“…of human-in-the-loop refinement [125], where human evaluators score model outputs on quality, assistance, accuracy, or ethical considerations [126], and then use the scores as a reward signal to iteratively adjust the model’s parameters to match human-approved outcomes, resulting in more…”
Lodoen, S., & Orchard, A. (2025) · IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS) · doi:10.1109/ethics65148.2025.11098367 · Cited by 6
“For example, in the case of translation tasks, bilingual annotators with proficiency in both languages are preferred [15], [23].”
InfluentialLevesque, J. (2025) · Journal of Human Trafficking · doi:10.1080/23322705.2025.2572911 · Cited by 3
Coverage-Based Harmfulness Testing for LLM Code Transformation
Tan, H., Wang, H., Pressato, D., Xu, Y., & Tan, S. H. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00086 · Cited by 5
Miao, Q., Zhou, R. H. F. L., Meng, J., & Wei, Y. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356825 · Cited by 1
“The biomedical community has increasingly adopted alignment paradigms combining Supervised Fine-Tuning (SFT) with Reinforcement Learning from Human Feedback (RLHF), typically via Proximal Policy Optimization (PPO) [11], [12], to enforce structured outputs.”
Metaphor Components Identification with Feedback-enhanced Feature-driven In-context Learning
Liu, H., He, C., Zhu, S., Niu, C., & Jia, Y. (2026) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3801739 · Cited by 1
Njuguna, G. N., & Min, Q. (2026) · AI & Society · doi:10.1007/s00146-026-02973-3 · Cited by 2
“This enables an explicit signal-reward model for RL, updating the LLM policy parameter via proximal policy optimization, steering the model to maximize predicted human preferences while enforcing safety boundaries (Chaudhari et al. 2026).”
Rădulescu, C. Z., Rădulescu, M., & Mihăilescu, A. I. (2026) · Future Internet · doi:10.3390/fi18040191 · Cited by 1
Mengistu, T., Faisal, K., Choi, W., & Kim, T. (2026) · Computer Science Review · doi:10.1016/j.cosrev.2026.101014 · Cited by 1
From the Field to the Algorithm: Understanding Indian Ethnographers' Perspectives on Responsible AI
Ghoshal, A., & Ghoshal, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791872
“[71, 82] RLHF is the process where human evaluators rate and compare AI outputs, training systems to align with preferred behaviors and values [23].”
Yu, Y., Yao, L., Xie, Y., Tan, Q., Feng, J., Li, Y., & Wu, L. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.981 · Cited by 4
Detecção automatizada de contrabando legislativo com Grandes modelos de linguagem
da Silva, F. T. D. D. O., & García, A. C. B. (2025) · doi:10.5753/sbsc_estendido.2025.6983
Exploring AI Applications and Innovations in Supply Chain Management
Ghosh, S., & Pramanik, P. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-6534-1_9 · Cited by 1
Tan, X., Wang, H., Li, S., Xu, Y., & Qiu, X. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11247288
“By providing asynchronous feed-back, H E S T I A enhances the learning process with high-level semantic understanding and world knowledge, while the core control algorithm continues to operate the on-policy agent at the required update frequency for dynamic locomotion learning [20].”
Zhu, L. (2025) · International Conference on Intelligent Computing and Data Analysis · doi:10.1145/3772726.3772846
“Effective shaping strategies have been explored in robotics, navigation, and multi-agent systems [2, 3].”
Liang, W., Guo, Z., Li, J., Li, X., Yang, J., Zeng, Y., & Zhou, X. (2025) · IEEE International Symposium on Product Compliance Engineering - Asia (ISPCE-ASIA) · doi:10.1109/ispce-asia69076.2025.11312923
“As alternatives or supplements to RLHF, an improved RLHF match-ing by fuzzy semantic matching of securely aligned content [16].”
Long, Y. (2025) · International Conference on Electronic Information Engineering and Computer Technology (EIECT) · doi:10.1109/eiect68017.2025.11331767
“For example, some methods rely on specific network structure assumptions, lacking generalizability and cross-scenario transferability [11].”
Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
Song, H., Choi, J., & Kim, M. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3777939 · Cited by 5
DSVA: An Iterative Refinement Multi-Agent Framework for NL-to-MTL Translation
Lyu, J., Wu, Y., & Cai, Y. H. (2026) · IEEE Access · doi:10.1109/access.2026.3665806
Yan, Z., Wang, Y., Yue, Q., & Wang, X. (2026) · Sensors · doi:10.3390/s26051433
“Representative approaches include reinforcement learning from human feedback (RLHF) frameworks based on Proximal Policy Optimization (PPO) [23–26], as well as subsequent variants that incorporate reference policies and KL regularization to improve training stability [27].”
Soft-Prompted Semantic Normalization for Unsupervised Analysis of the Scientific Literature
Malashin, I. P., Martysyuk, D., Тынченко, В. С., Gantimurov, A., Nelyub, V., & Borodulin, A. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8030063 · Cited by 1
TADEL: Task-Aware Dynamic Ensemble of Lightweight LLMs for Improved Inference Accuracy in Edge AI
Li, H., Sahni, Y., Cao, J., & Xiao, F. (2025) · IEEE Annual Congress on Artificial Intelligence of Things (AIoT) · doi:10.1109/aiot66900.2025.00022
“Among them, reinforcement learning based on Human Feedback (RLHF) and reinforcement learning based on Model Feedback (RLAIF) are currently two widely used RL techniques based on reward models, mainly used for output alignment of a single model [30].”
NL2SQL Generation Technology for Multi-Source Heterogeneous Data Based on RLHF and Knowledge Graph
Wang, D., & Le, Y. (2025) · doi:10.1109/aibdf67964.2025.11440762
“Compared to traditional knowledge representation methods, they have significant advantages in data-driven approaches, management, and type representation [5-6].”
Improving AI Accuracy: Identifying and Fixing Hallucinations in Large Language Models
Dadoriya, N., & Chandra, S. K. (2025) · International Conference on Data Science and Intelligent Network Computing (ICDSINC) · doi:10.1109/icdsinc66221.2025.11448207
“Human feedback is received from the AI learning process through RLHF, which im-proves AI output and causes misleading accuracy [14].”
Xuan, T. N., Gia, B. N., Hoai, N. V., & Cam, V. T. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-18159-6_32
Cheng, H., Wang, K., Wang, Q., Liu, T., & Sheng, K. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114670 · Cited by 4
Exploring attribution bias in LLMs: Social influences and prompt-based mitigation
Jiang, L., Cao, J., Zhu, G., & Wang, Y. (2026) · Technology in Society · doi:10.1016/j.techsoc.2026.103361 · Cited by 1
Padidela, S., Kalari, S., Ashok, V., et al. (2026) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa67686.2026.11468212
“Existing strategies for domain adaptation typically rely on fine-tuning pre-trained models using specialized datasets from a specific domain, such as medical datasets in healthcare domain [4]–[6].”
Small Language Model Jailbreak Defender Framework
Castro-Puello, E., Li, W., Wang, L., Kumar, A., & Mulissa, Y. G. (2026) · IEEE SoutheastCon · doi:10.1109/southeastcon63549.2026.11476263
“A contributing factor is that SLMs are frequently trained without RLHF (often due to resource constraints) and/or exposed to less adversarial data during fine tuning [5].”
Multi-agent Reinforcement Learning in Collaborative Optimization of Integrated Energy Systems
Zhang, X. (2025) · International Conference Communication, Networks & Computing · doi:10.1109/cnc68716.2025.11484756
“This mechanism allows the agent to distinguish useful signals from useless information in the process of distinguishing the energy flow coupling, enhancing the stability of the training process and the correctness of decision making [15].”
Gamification for Inclusive Large Language Models
Samimi-Dehkordi, L., Kolahdouz-Rahimi, S., & Fatima, S. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-6063-8.ch003
Chen, X., Shi, S., & Shi, X. (2026) · Internet Technology Letters · doi:10.1002/itl2.70307
“Broader analyses of RLHF further stress that preference feedback can be inconsistent and artifact-prone, motivating principled robustness rather than heuristic smoothing [14].”
Yun, L., Hao, Y., Gong, H., & Li, B. (2026) · iScience · doi:10.1016/j.isci.2026.116056 · Cited by 1
Shea, G. K., & Wang, H. (2026) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.26.00241
Zhang, Y. F. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.04.191
Poltorakov, О. Y., & Rakhimov, V. (2026) · Politology bulletin · doi:10.17721/2415-881x.2026.100.2.281-296
Quan, X. (2026) · Scientific Reports · doi:10.1038/s41598-026-52424-8 · Cited by 1
“Through this structured approach which can be improved by RLHF, the interaction can be transformed into a dynamic and factual dialogue 54,55 .”
Security, Privacy, and Ethical Challenges in Large Language Models: A Review
Yogi, B., Ghosh, N., & Sahid, S. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10680-x
Human-in-the-Loop Reinforcement Learning for Intelligent File Cleanup in Personal Computing Systems
Suryateja, T., Srimannath, S., Reddy, A. H., & Kumaran, K. (2026) · International Conference on Inventive Research in Computing Applications (ICIRCA) · doi:10.1109/icirca69024.2026.11570627
“Deep RL handles high-dimensional optimization tasks [7][8][9], while RLHF improves policy alignment using human feedback [14][15].”
Online Communities as Collaborative Testbeds for Collective Alignment
Zent, M., Khadar, M., Yarosh, S., Chancellor, S., & Kaur, H. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812370
Alexan, W., & Temperini, M. (2026) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon67543.2026.11574057
“Bayesian Knowledge Tracing (BKT) [17] and Item Response Theory (IRT) [18] provide transparent estimates of concept mastery and item characteristics, while reinforcement learning (RL) is explored to optimize task sequencing and feedback policies based on observed learning signals [19].”
Adaptive reinforcement learning based projected gradient descent attack
Zhu, Z. D. J. D., Zhang, Y., Ye, A., Wang, X. Y., Wang, C., & Zhu, T. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08687-z · Cited by 1
“…still face three bottlenecks: (i) RL-based studies often adjust only the objective while lacking a coordinated framework for complex defenses [9]; (ii) discrete methods such as GCG require expensive token-level trial-and-error over large candidate sets [3]; and (iii) gradient-based methods…”
Human-in-the-Loop Framework for Hallucination Mitigation: A Controlled Comparison of GPT and LLaMA
Kai, J. N. J., Syahidi, A. A., & Kiong, K. B. (2026) · International Conference on Business and Industrial Research (ICBIR) · doi:10.1109/icbir69932.2026.11584794
“Dominant alignment paradigms, such as Reinforcement Learning from Human Feedback (RLHF) and preference optimization methods, rely on rich and continuous supervision signals collected at scale, including demonstrations, ranked preferences, and detailed edits [16], [17], [18], [19].”
Composer: Learning to Orchestrate Generative Primitives for Controllable Speech Synthesis
Cao, J., Sheng, L., Fu, B., et al. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582218
“Following DPO [5]and RLHF [15], we generate candidate scripts S + and S − , render audio, and collect pairwise preferences via crowdworkers or audio-language models.”
Wu, C., Wu, Q., Liu, K., Jin, S., & Ma, D. (2026) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2026.105873
Repetition is not Redundancy: AI and Relational Listening in Communicating about Extreme Weather
Antonellos, M., Aquino, A., Gumbula, I. M., Cheung, J., & Bidwell, N. J. (2026) · ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies (COMPASS) · doi:10.1145/3811242.3827554
Query-MARFT: Query-guided multi-agent reinforcement fine-tuning for end-to-end multi-object tracking
Wen, J., Su, Y., Liu, H. F., & Li, J. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134603
GuidaPA: Privacy-Preserving Chatbot for Public Administration via Federated Learning
Jimenez-Gutierrez, D. M., Cirillo, A., Nicolussi, R., Beltrame, A., & Vitaletti, A. (2026) · International Conference on Federated Learning and Intelligent Computing Systems (FLICS) · doi:10.1109/flics70075.2026.11621908
“Second, evaluation is based on offline text-generation metrics (ROUGE, BLEU-4, METEOR) on held-out splits and does not include human assessment [43], end-to-end task success, or live deployment measurements.”
A Security-Oriented Lifecycle Model for Large Language Model Systems
Batzolis, E., Drosatos, G., Katsouros, V., & Rantos, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37218-5_24
“As a worked example, the separation of Supervised Fine-Tuning (SFT, Stage 2.3) from Alignment (Stage 2.4), here understood as preference-based fine-tuning that shapes model behaviour toward human preferences and safety constraints [4], follows directly from this principle.”
InfluentialA Review of the Applications of Vision-Language Models in the Low-Altitude Economy
Yang, Z., Meng, Z., Ji, J., Liu, S., & Sun, Z. (2026) · Springer aerospace technology · doi:10.1007/978-981-95-6988-5_44
Large artificial intelligence models for coal mining: Development, applications and challenges
Wang, G., Luo, Y., Yang, L., Wang, D., & Cheng, J. (2026) · International Journal of Minerals Metallurgy and Materials · doi:10.1007/s12613-026-3492-8
Liu, Y., Ji, Q., Lu, S., & Chen, J. (2026) · Mathematics · doi:10.3390/math14173168
A side-channel attack on reinforcement learning models
Zhou, H., Wang, Z., Yin, D., Feng, Y., Wang, Y., Pan, H., Guo, W., Wang, R., & Chen, Y. (2026) · Integration · doi:10.1016/j.vlsi.2026.102869
Walters, D. M., & Wang, B. (2026) · IEEE National Aerospace and Electronics Conference (NAECON) · doi:10.1109/naecon70028.2026.11674895
Gwinnutt, J. M., de Oliveira, R., Haviland, M. J., Heidt, J., Eldridge, E. H., Pereira, L. M., Bratton, E., Nanavati, J., & Mack, C. D. (2026) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.70452
Xiaoxu, L., Yi, H., 王肇炽, et al. (2026) · Xibei Gongye Daxue Xuebao/Journal of Northwestern Polytechnical University · doi:10.1051/jnwpu/20264440828
UbiHearo: Bringing Scenario-Aware Sound Guidance to DHH Users with Mobile Agents
Wu, F. 中., Liu, S., Zhou, Z., Xu, C., Zhao, W., Sun, P., & Yu, Z. (2026) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3832010 · Cited by 1
Park, W., Mansour, S., Kim, L., & Eid, M. (2026) · Frontiers in Neuroscience · doi:10.3389/fnins.2026.1934150
How philosophical reasoning shapes the alignment problem: evolutionary debunking as a case study
Dale, M. T., & Saad, B. (2026) · Synthese · doi:10.1007/s11229-026-05796-3
“Feedback on the validity of inferences from 74 See Christiano et al. (2017) and Chaudhari et al. (2025).”
A survey of LLM reasoning in healthcare and medicine: Individual modeling and collaborative agents
Lin, Q., Xiao, L., Pu, B., Shi, K., Du, Y., Xu, J., He, K., Zhao, S., Zhang, J., Cambria, E., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104827 · Cited by 6
Yu, Y., Yao, L., Xie, Y., Tan, Q., Feng, J., Li, Y., & Wu, L. (2026) · arXiv · doi:10.48550/arxiv.2601.01885
“Reinforcement learning for LLMs. Reinforcement learning has become an effective paradigm for improving the decision-making and reasoning capabilities of LLM-based agents (Yao et al., 2023; Jin et al., 2025; Qian et al., 2025; Chaudhari et al., 2025).”
Metadata
- Authors
- Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B.
- Affiliations
- University of Massachusetts Amherst (US)
- Carnegie Mellon University (US)
- Princeton University (US)
- Independent Researcher
- University of Massachusetts Amherst
- ORCID
- Venue
- ACM Computing Surveys
- Published
- 2025-09-10
- Volume
- 58
- Issue
- 2
- Pages
- 1-37
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4411063470
- Cited by
- 92
- References
- 36
- Citations by year
- 2026: 60 · 2025: 26
- FWCI
- 34.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- reinforcement learning from human feedback, large language models, reward modeling, function approximation, model misspecification, generalization, sparse feedback, human preference alignment, language model alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.