RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Open accessThe DOI matches its Crossref record; 239 of the 239 references it cites were found in a registry or an index; 149 of the 165 sources its claims cite were checked for support.
Abstract
A significant challenge in training large language models (LLMs) as effective assistants is aligning them with human preferences. Reinforcement learning from human feedback (RLHF) has emerged as a promising solution. However, our understanding of RLHF is often limited to initial design choices. This article analyzes RLHF through reinforcement learning principles, focusing on the reward model. It examines modeling choices and function approximation caveats, highlighting assumptions about reward expressivity and revealing limitations like incorrect generalization, model misspecification, and sparse feedback. A categorical review of current literature provides insights for researchers to understand the challenges of RLHF and build upon existing methods.
Overview
CoChat Summary
TL;DR
This article analyzes RLHF for LLMs through reinforcement learning principles, focusing on the reward model and its assumptions. It argues that reward-model misgeneralization, model misspecification, and sparse feedback are fundamental limitations, and surveys current methods and alternatives.
Key findings
- RLHF’s reward model is the core component: the paper formalizes an “oracular reward” \(R^\star(c,o)\) and shows that because human feedback covers only a tiny subset of all context-output pairs, the learned RM \(R_\phi\) cannot be guaranteed to generalize out-of-distribution, leading to misgeneralization, reward hacking, and hallucinations.
- The paper categorizes RM limitations as misgeneralization, marginalization over preferences/model misspecification, and delayed/sparse feedback; it notes that a deterministic scalar reward averages over annotators and cannot represent variability in human preferences.
- The literature review classifies feedback into rating, ranking/preference, language, and miscellaneous feedback; it reports that ranking \(N\) outputs yields \(N(N-1)/2\) pairwise comparisons versus \(N\) training points from \(N\) ratings, and that labelers took only 3x longer to provide language feedback than preference-based feedback in one cited study.
- Practical RLHF results cited include Llama-2-7B improving on TruthfulQA from 33% to 57% with RLHF and toxicity generation decreasing from 22% to near 0 on ToxiGen; Ouyang et al. are cited as reporting good performance with an RM 30 times smaller than the policy model.
Methods
- Design: critical analysis and survey, not a new empirical study; it formulates text generation as a Markov decision process and adopts a Bayesian perspective on RLHF to analyze reward modeling assumptions.
- Data/sample: no new dataset; it reviews existing RLHF literature, including feedback types, reward modeling, RLHF finetuning, practical applications, and RL-free alternatives.
- Main analysis: theoretical examination of RM generalization using context coverage \(\kappa\) and output coverage \(\rho\), likelihood functions such as Bradley-Terry and Plackett-Luce, and policy-gradient algorithms including PPO and A2C.
Limitations
- The paper states that its analysis focuses on deterministic rewards, not stochastic reward distributions, and that RLHF itself suffers from unstable training, hyperparameter sensitivity, high memory usage, reward overoptimization, alignment tax, low output entropy, and persistent bias/toxicity issues.
- The paper states that alternatives to conventional RL have been evaluated only on specific domains and constrained settings, and that a comprehensive comparison of scaling effects on RLHF versus SFT models is currently lacking.
- Evident from the text: as a survey/critical analysis, it provides no new empirical validation or benchmark results of its own; many quantitative claims depend on cited works rather than original experiments.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
81 Valid · 27 Not valid · 10 Not checked · 65 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Recently, [163] has shown that language feedback is more effective than preference-based feedback in the context of summarization systems.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > Ranking or Preference Feedback.
What the sources say
- Disputedcites [163] 163. J'er'emy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2022. Training language models with language feedback. sourceExperimental — accuracy not yet measured
In §H.2 (page unavailable), the preference-based OPT-RM method achieved a 45.1 ± 1.9% win rate against human summaries, versus 36.0 ± 1.8% for ILF trained on language-feedback refinements. Section 4.3.2 also says OPT-RM outperformed the finetuning methods; the strongest result came from combining language and binary feedback, not from language feedback alone. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“This inaccurate generalization in the RM is one of the primary causes of phenomena like”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 5.1 Limitations of the Reward Model
What the sources say
- Not matchedcites [193] 193. Lilian Weng. 2018. Policy gradient algorithms. Blog post, April 8, 2018. lilianweng.github.io (2018). https://lilianweng. github.io/posts/2018-04-08policy-gradient/
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Could not verifycites [73] 73. Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore. 1996. Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4 (1996), 237-285. Retrieved from https://api.semanticscholar.org/CorpusID:1708582Experimental — accuracy not yet measured
The Abstract (p. 1) attributes a lower bound on some hallucinations to statistical properties of calibrated pretrained language models. Searches for reward-model generalization and reward hacking found no passage connecting either to hallucinations, so our copy does not address the attributed causal claim. (LLM, conf=0.75)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Firstly, we get more training data for the RM.In practice, people collect a ranking of 𝑁 outputs and create preference pairs [136].Collecting 𝑁 ratings for 𝑁 outputs provides we get 𝑁 training points.Ranking 𝑁 outputs provided 𝑁 (𝑁 -1)/2 pairwise comparisons.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 4.1 Oracular Reward and the Role of Human Feedback
What the sources say
- Partly supportedcites [136] 136. Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V. Hafner. 2007. Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation 11, 2 (2007), 265-286. https://doi.org/10.1109/TEVC. 2006.890271 sourceExperimental — accuracy not yet measured
Section 3.5 (page unavailable) says ranking K responses produces K(K−1)/2 comparisons used to train the reward model. It does not compare this with collecting ratings, and cautions that comparisons from one ranking are highly correlated, so the broader claim of more training data than ratings is not fully established. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Naturally, the reward function plays a critical role in the process of fine-tuning model outputs, determining practical and fundamental limits [24] of the efficacy of RLHF.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > Markov decision process.
What the sources say
- Supportedcites [24] 24. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv:2307.15217 [cs.AI]. source
Section 2, Step 3 states that RLHF fine-tunes the base model by optimizing against the reward model. The passages on evaluating reward models and on reward hacking describe practical difficulties and fundamental limits arising from that model. The supplied passages have section labels but no page numbers. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“(1) The action space is extremely large.For example, the LLaMa model [184,185] employs a vocabulary of size 32K.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > Markov decision process.
What the sources say
- Could not verifycites [184] 184. Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, and Bo Zheng. 2025. Equilibrate RLHF: Towards balancing helpfulness-safety trade-off in large language models. arXiv:2502.11555 [cs.AI]. Retrieved from https://arxiv.org/abs/2502.11555 sourceExperimental — accuracy not yet measured
The paper’s Introduction (p. 1) discusses LLM text generation, and its alignment-method sections discuss reinforcement learning, but the passages found do not describe the size of a text-generation MDP’s action space. Searches for “action space,” “vocabulary,” and “32K” also found no passage establishing the sentence’s LLaMa example. (LLM, conf=0.74)
- Could not verifycites [185] 185. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. sourceExperimental — accuracy not yet measured
The Tokenizer passage in §2.1 (p. 2) describes LLaMA’s byte-pair encoding but gives no vocabulary size. Searches for “32K” and “32,000” found no passage in our copy establishing the cited 32K example or describing the action space of a text-generation MDP. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
74 more claims · 43 with their cited sources checked
Create a free account to see all 183 claims and to verify the 2 references still open.
- It may state two separate claims as one. 70
- Probably not a substantial claim. 54
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 2 Motivation: Eliminating Objective Mismatch in Pre-Trained Language Models
“The first term (a) is the log-likelihood of the feedback dataset, specifying how a human's internal objective (reward function) governs their feedback.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 2 Motivation: Eliminating Objective Mismatch in Pre-Trained Language Models
“𝜙 MAP = arg max 𝜙 Pr(𝜙 | D HF ) = arg max 𝜙 Pr(D HF | 𝑅 𝜙 ) (a) Pr(𝜙) (b)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > Markov decision process.
“𝜌 : S → Δ(S) is the initial state distribution.”
§ RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs > 7 Discussion and Conclusion
“The agent starts in state 𝑠 1 ∼ 𝜌 (•).”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 17,283
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824 · cited 1×
“The regression problem is however underdetermined [19], and consequently multiple 𝑅 𝜙 functions can perfectly fit the training data D rew .”
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Even after filtering, some methods ensure a hands-on relationship with labellers [177] and have also created Slack groups for discussing any bugs, issues, or edge cases [12].”
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972 · cited 1×
“Foundational surveys on RL [7, 72] provide broad overviews of classical and deep RL, respectively, but do not focus on human feedback or LLMs. Recent literature surveys have also focussed on integrating (human) feedback specifically for LMs [47, 188], but are limited to feedback without providing a…”
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733 · cited 1×
“We next discuss a few commonly used RL algorithms for the process.”
A Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“The remarkable utility of these models holds the potential to transform established workflows in critical sectors such as technology, healthcare, finance, and education [173, 195, 203].”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons
Bradley, R. A., & Terry, M. E. (1952) · Biometrika · doi:10.2307/2334029 · Cited by 2,412
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,869
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815
Learning to Rank for Information Retrieval
Liu, T.-Y. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000016 · Cited by 1,568
Practical and ethical challenges of large language models in education: A systematic scoping review
Yan, L., Sha, L., Zhao, L., Li, Y., Martinez‐Maldonado, R., Chen, G., Li, X., Jin, Y., & Gašević, D. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13370 · Cited by 1,435
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,435
RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS: THE METHOD OF PAIRED COMPARISONS
BRADLEY, R. A., & TERRY, M. E. (1952) · Biometrika · doi:10.1093/biomet/39.3-4.324 · Cited by 1,434 · cited 3×
“The challenges encountered in learning these reward functions highlight both the practical and fundamental limitations of RLHF, as thoroughly analyzed by Casper et al. [24].”
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732 · Cited by 1,426 · cited 1×
“The prompting dataset may or may not be identical to the one used for collecting human demonstrations in the SFT phase [136].”
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., & Zaremba, W. (2017) · arXiv · doi:10.48550/arxiv.1707.01495 · Cited by 1,355
ALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185 · cited 1×
“More recently RL has been extensively used for improving code and math performance in verifiable domains [34, 64, 89, 134, 204]”
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185 · cited 1×
“More recently RL has been extensively used for improving code and math performance in verifiable domains [34, 64, 89, 134, 204]”
Individual choice behavior: A theoretical analysis.
Luce, R. D. (2005) · doi:10.1037/14396-000 · Cited by 1,157
Intrinsic Motivation Systems for Autonomous Mental Development
Oudeyer, P.-Y., Kaplan, F., & Hafner, V. V. (2007) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2006.890271 · Cited by 1,126
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 2×
“In contrast, our work explicitly analyzes the theoretical underpinnings of RLHF, particularly focusing on RM assumptions, generalization issues, and feedback sparsity, providing deeper and broader insights.”
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Self-citationDeep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · cited 6×
“This form of reward modeling has been found to provide smoother rewards and is less noisy [30].”
InfluentialRecent Advances in Hierarchical Reinforcement Learning
Barto, A. G., & Mahadevan, S. (2003) · Discrete Event Dynamic Systems · doi:10.1023/a:1022140919877 · Cited by 1,024
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005 · cited 1×
“Automated metrics like BLEU and ROUGE [99] often do not correlate with human judgment [103, 164, 166, 177], making them unreliable for evaluation and training.”
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. P. (2020) · arXiv · doi:10.48550/arxiv.2004.04696 · Cited by 1,005 · cited 1×
“Automated metrics like BLEU and ROUGE [99] often do not correlate with human judgment [103, 164, 166, 177], making them unreliable for evaluation and training.”
(2011) · Encyclopedia of Machine Learning · doi:10.1007/978-0-387-30164-8_588 · Cited by 950
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2021) · arXiv · doi:10.48550/arxiv.2109.07958 · Cited by 899
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
Dong, H., Xiong, W. H., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., & Zhang, T. (2023) · Transactions on machine learning research · doi:10.48550/arxiv.2304.06767 · Cited by 856
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834
Training Diffusion Models with Reinforcement LearningPreprint
Black, K., Janner, M., Du, Y., Kostrikov, I., & Levine, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13301 · Cited by 813
Econometric Models for Probabilistic Choice Among Products
McFadden, D. (1980) · The Journal of Business · doi:10.1086/296093 · Cited by 766
Lin, C.-Y., & Och, F. J. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1219032 · Cited by 737 · cited 1×
“Automated metrics like BLEU and ROUGE [99] often do not correlate with human judgment [103, 164, 166, 177], making them unreliable for evaluation and training.”
Plackett, R. L. (1975) · Applied Statistics · doi:10.2307/2346567 · Cited by 731 · cited 1×
“A similar method can then be used for ranking between k trajectories (k > 2), where the reward is modeled as a Plackett–Luce ( PL ) model [109, 144].”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2338 · Cited by 686
Liu, C., Lowe, R., Serban, I. V., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08023 · Cited by 603
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575
Deep double descent: where bigger models and more data hurt*
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2021) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ac3a74 · Cited by 565 · cited 1×
“…of all possible inputs and outputs D can be measured by: Well-understood results in supervised learning suggest that the ratios 𝜌 and 𝜅 along with the generalization capabilities of the function approximator [13, 124, 161] determine the generalization performance of the RM for ( 𝑐,𝑜 ) ∈ C , O .”
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
A Survey of Multi-Objective Sequential Decision-Making
Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3987 · Cited by 461
Silver, D., Singh, S., Precup, D., & Sutton, R. S. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103535 · Cited by 445 · cited 1×
“The oracular reward may not be accessible or learnable, but under the reward hypothesis [172, 180], the mere existence of such a reward may be assumed— though this may be challenged [81].”
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442
An Actor-Critic Algorithm for Sequence Prediction
Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., Courville, A., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1607.07086 · Cited by 423
Fine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413
Secrets of RLHF in Large Language Models Part I: PPOPreprint
Zheng, R., Dou, S., Gao, S., Yuan, H. Q., Shen, W., Wang, B., Liu, Y., Jin, S., Liu, Q., Zhou, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.04964 · Cited by 391
The False Promise of Imitating Proprietary LLMs
Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., & Song, D. (2023) · arXiv · doi:10.48550/arxiv.2305.15717 · Cited by 377
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · cited 3×
“This can lead to harmful and unintended behavior, making it easier to prompt such models to elicit them [50, 142].”
InfluentialBloombergGPT: A Large Language Model for Finance
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., & Mann, G. (2023) · arXiv · Cited by 357
Ramé, A., Couairon, G., Dancette, C., Gaya, J.-B., Shukor, M., Soulier, L., & Cord, M. (2023) · neural information processing systems · doi:10.52202/075280-3114 · Cited by 355
Deal or No Deal? End-to-End Learning of Negotiation Dialogues
Lewis, M., Yarats, D., Dauphin, Y., Parikh, D., & Batra, D. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1259 · Cited by 347 · cited 1×
“Future research should explore the integration of these techniques to address the unique challenges in training LLMs with RLHF, potentially improving generalization and robustness.”
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326
Recent Advances in Hierarchical Reinforcement Learning
Barto, A. G., & Mahadevan, S. (2003) · Discrete Event Dynamic Systems · doi:10.1023/a:1025696116075 · Cited by 317
Deduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.577 · Cited by 304
Sources of Hallucination by Large Language Models on Inference TasksPreprint
McKenna, N., Li, T., Cheng, L., Hosseini, M., Johnson, M., & Steedman, M. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2305.14552 · Cited by 297
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293
Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models
Ferrara, E. (2023) · arXiv · doi:10.48550/arxiv.2304.03738 · Cited by 282
Jailbroken: How Does LLM Safety Training Fail?
Wei, A., Haghtalab, N., & Steinhardt, J. (2023) · arXiv · doi:10.48550/arxiv.2307.02483 · Cited by 280
Statistical Rejection Sampling Improves Preference Optimization
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., & Liu, J. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.06657 · Cited by 280
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.234 · Cited by 279
Structured Pruning Learns Compact and Accurate Models
Xia, M., Zhong, Z., & Chen, D. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2204.00408 · Cited by 277
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022) · arXiv · doi:10.48550/arxiv.2203.09509 · Cited by 276
LIMA: Less Is More for Alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.11206 · Cited by 275
LIMA: Less Is More for Alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2400 · Cited by 275
WebGPT: Browser-assisted question-answering with human feedback
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.09332 · Cited by 272 · cited 5×
“…is only recently that RLHF with LLMs has achieved significant success in sequence-to-sequence tasks such as Summarization [177, 194, 222], providing reliable answers with citations to queries [53, 123], creating Helpful, Harmless and Honest dialogue agents aligned with broad human values [12, 136].”
InfluentialBailly, A., Blanc, C., Francis, É., Guillotin, T., Jamal, F., Wakim, B., & Roy, P. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106504 · Cited by 252
A Joint Model of Language and Perception for Grounded Attribute Learning
Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., & Fox, D. (2012) · Maryland Shared Open Access Repository · doi:10.13016/m2ehu6-1m90 · Cited by 248
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · cited 2×
“In addition to scale, the quality of training data, ensured through deduplication and filtering of low-quality content, is a key determinant of model performance [42, 65, 91, 148].”
Deep Reinforcement Learning for Sequence-to-Sequence Models
Keneshloo, Y., Shi, T., Ramakrishnan, N., & Reddy, C. K. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2929141 · Cited by 230 · cited 2×
“These are gradient-based methods, called policy-gradient algorithms, that directly update the parameters of the policy using the evaluative reward feedback.”
Nash Learning from Human FeedbackPreprint
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Rowland, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Michi, A., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.00886 · Cited by 211
Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise ComparisonsPreprint
Zhu, B., Jiao, J., & Jordan, M. I. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.11270 · Cited by 188
Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMsPreprint
Gupta, S., Shrivastava, V., Deshpande, A., Kalyan, A., Clark, P. A., Sabharwal, A., & Khot, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.04892 · Cited by 185
Self-citationToxicity in chatgpt: Analyzing persona-assigned language models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.88 · Cited by 178
Self-citationQuality-Aware Decoding for Neural Machine Translation
Fernandes, P., Farinhas, A., Rei, R., de Souza, J. G. C., Ogayo, P., Neubig, G., & Martins, A. F. T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.100 · Cited by 178
Toxicity in ChatGPT: Analyzing Persona-assigned Language Models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2304.05335 · Cited by 178
Self-citationLearning from Dialogue after Deployment: Feed Yourself, Chatbot!
Hancock, B., Bordes, A., Mazare, P.-E., & Weston, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1358 · Cited by 161 · cited 1×
“Future research should explore the integration of these techniques to address the unique challenges in training LLMs with RLHF, potentially improving generalization and robustness.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Fırat, O., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.06905 · Cited by 157
RL with KL penalties is better viewed as Bayesian inferencePreprint
Korbak, T., Perez, E., & Buckley, C. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2205.11275 · Cited by 154
The Curse of Recursion: Training on Generated Data Makes Models Forget
Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2023) · arXiv · doi:10.48550/arxiv.2305.17493 · Cited by 151
Liu, B., Tür, G., Hakkani-Tür, D., Shah, P., & Heck, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1187 · Cited by 148 · cited 1×
“The reward is typically modeled as a Bradley-Terry-Luce ( BTL ) model [21], where the probability of preferring one trajectory over another is a function of the difference in their reward scores.”
A General Language Assistant as a Laboratory for Alignment
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00861 · Cited by 146
Sources of Hallucination by Large Language Models on Inference Tasks
McKenna, N., Li, T., Cheng, L., Hosseini, M., Johnson, M., & Steedman, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.182 · Cited by 142
Structured Pruning Learns Compact and Accurate Models
Xia, M., Zhong, Z., & Chen, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.107 · Cited by 136
Improving alignment of dialogue agents via targeted human judgements
Glaese, A., McAleese, N., Trębacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.14375 · Cited by 134 · cited 7×
“[53] even provides a tutorial to the selected few labelers.”
InfluentialShould ChatGPT Be Biased? Challenges and Risks of Bias in Large Language Models
Ferrara, E. (2023) · doi:10.2139/ssrn.4627814 · Cited by 133
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.07858 · Cited by 130 · cited 1×
“In contrast, our work explicitly analyzes the theoretical underpinnings of RLHF, particularly focusing on RM assumptions, generalization issues, and feedback sparsity, providing deeper and broader insights.”
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.15115 · Cited by 128
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., & Picard, R. (2019) · arXiv · doi:10.48550/arxiv.1907.00456 · Cited by 126
Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S., & Dragan, A. (2017) · arXiv · doi:10.48550/arxiv.1711.02827 · Cited by 125
Block Pruning For Faster Transformers
Lagunas, F., Charlaix, E., Sanh, V., & Rush, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.829 · Cited by 123 · cited 1×
“While major progress has been made toward efficient training and inference of LLMs [66, 88, 121, 122, 178, 199, 200, 206, 209], there is still a lot of scope for improvement in the sample efficiency and stability of training algorithms for RLHF. Recent work has addressed these challenges with…”
Reinforcement Learning with Sparse Rewards using Guidance from Offline DemonstrationPreprint
Rengarajan, D., Vaidya, G., Sarvesh, A., Kalathil, D. M., & Shakkottai, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.04628 · Cited by 118
TAMER: Training an Agent Manually via Evaluative Reinforcement
Bradley Knox, W., & Stone, P. (2008) · 2008 7th IEEE International Conference on Development and Learning · doi:10.1109/devlrn.2008.4640845 · Cited by 117 · cited 1×
“The model outputs, along with the inputs, are passed to the RM that generates a scalar output indicating the reward for this input-output pair.”
Reward learning from human preferences and demonstrations in Atari
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., & Amodei, D. (2018) · arXiv · doi:10.48550/arxiv.1811.06521 · Cited by 117
Whose Opinions Do Language Models Reflect?
Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2303.17548 · Cited by 114
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Nguyen, K., Daumé III, H., & Boyd-Graber, J. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1153 · Cited by 110 · cited 4×
“The model outputs, along with the inputs, are passed to the RM that generates a scalar output indicating the reward for this input-output pair.”
InfluentialThe limits of automatic summarisation according to ROUGE
Schluter, N. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2007 · Cited by 108 · cited 1×
“Automated metrics like BLEU and ROUGE [99] often do not correlate with human judgment [103, 164, 166, 177], making them unreliable for evaluation and training.”
Reward Shaping in Episodic Reinforcement Learning
Grześ, M. (2017) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/czhk5726 · Cited by 104
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.03300 · Cited by 101
Aligning Large Language Models through Synthetic FeedbackPreprint
Kim, S., Bae, S., Shin, J., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2305.13735 · Cited by 95
Don’t Blame the Annotator: Bias Already Starts in the Annotation Instructions
Parmar, M., Mishra, S. R., Geva, M., & Baral, C. R. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.130 · Cited by 95
Toward Diverse Text Generation with Inverse Reinforcement Learning
Shi, Z., Chen, X., Qiu, X., & Huang, X. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/606 · Cited by 94
Re3: Generating Longer Stories With Recursive Reprompting and Revision
Yang, K., Tian, Y., Peng, N., & Klein, D. (2022) · arXiv · doi:10.48550/arxiv.2210.06774 · Cited by 91 · cited 1×
“Future research should explore the integration of these techniques to address the unique challenges in training LLMs with RLHF, potentially improving generalization and robustness.”
Re3: Generating Longer Stories With Recursive Reprompting and Revision
Yang, K., Tian, Y., Peng, N., & Klein, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.296 · Cited by 90
Ranking vs. Preference: A Comparative Study of Self-reporting
Yannakakis, G. N., & Hallam, J. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24600-5_47 · Cited by 90 · cited 1×
“These approaches include reward shaping, where the original reward signal is augmented with additional feedback [56, 125]; intrinsic motivation, which encourages exploration and learning through internal rewards based on novelty, curiosity, or learning progress [17, 135, 140]; and multi-objective…”
Dialogue Learning With Human-In-The-Loop
Li, J., Miller, A. H., Chopra, S., Ranzato, M., & Weston, J. (2016) · arXiv · doi:10.48550/arxiv.1611.09823 · Cited by 90
Calibrated Language Models Must Hallucinate
Kalai, A. T., & Vempala, S. S. (2023) · arXiv · Cited by 90
Learning preferences for manipulation tasks from online coactive feedback
Jain, A., Sharma, S., Joachims, T., & Saxena, A. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364915581193 · Cited by 87 · cited 1×
“Moreover, richer forms of feedback, such as learning from corrections [14, 70], demonstrations [156], and language feedback [49, 112], have proven beneficial.”
Improving Code Generation by Training with Natural Language FeedbackPreprint
Chen, A., Scheurer, J., Korbak, T., Campos, J. A., Chan, J. S., Bowman, S. R., Cho, K., & Perez, E. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.16749 · Cited by 85
Self-consuming generative models go MAD
Casco-Rodriguez, J., Alemohammad, S., Luzi, L., Imtiaz, A., Babaei, H., LeJeune, D., Siahkoohi, A., & Baraniuk, R. (2023) · LatinX in AI at Neural Information Processing Systems Conference 2023 · doi:10.52591/lxai202312101 · Cited by 83 · cited 1×
“However, these methods still use little to no human feedback and may have several unexplored limitations such as instability and lack of robustness [3, 57, 171] and are not the focus of this survey.”
Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D., Yang, Y., & Gan, C. (2023) · arXiv · doi:10.48550/arxiv.2305.03047 · Cited by 81
(2023) · doi:10.52202/075280-0115 · Cited by 81
Preference Ranking Optimization for Human Alignment
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2306.17492 · Cited by 77 · cited 1×
“Further, various works [41, 176, 210] use the trained RM to rank multiple responses and use the signal from the ranked responses to train the policy model, without using an elaborate RL algorithm.”
Recursively Summarizing Books with Human Feedback
Wu, J., Ouyang, L., Ziegler, D. M., Stiennon, N., Lowe, R., Leike, J., & Christiano, P. (2021) · arXiv · doi:10.48550/arxiv.2109.10862 · Cited by 76
Investigations of Performance and Bias in Human-AI Teamwork in Hiring
Peng, A., Nushi, B., Kıcıman, E., Inkpen, K. M., & Kamar, E. S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21468 · Cited by 71 · cited 1×
“Recent works, such as [104, 213], provide both positive and negative demonstrations/feedback and maximize the likelihood of generating positive/preferred output.”
Schaeffer, R., Khona, M., Robertson, Z., Boopathy, A., Pistunova, K., Rocks, J. W., Fiete, I., & Koyejo, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.14151 · Cited by 71
Aligning Large Language Models with Human: A Survey
Wang, Y., Zhong, W., Li, L., Mi, F., Zeng, X., Huang, W., Shang, L., Jiang, X., & Liu, Q. (2023) · arXiv · doi:10.48550/arxiv.2307.12966 · Cited by 69 · cited 1×
“…and deep RL, respectively, but do not focus on human feedback or LLMs. Recent literature surveys have also focussed on integrating (human) feedback specifically for LMs [47, 188], but are limited to feedback without providing a comprehensive survey of complete RLHF pipelines and alternatives.”
The alignment problem from a deep learning perspective
Ngo, R., Chan, L., & Mindermann, S. (2022) · arXiv · doi:10.48550/arxiv.2209.00626 · Cited by 68
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., & Cobbe, K. (2023) · arXiv · doi:10.48550/arxiv.2305.20050 · Cited by 65 · cited 2×
“Thus, research in this field is critical in shaping the future of large-scale language modeling and its societal impact.”
Unified Pragmatic Models for Generating and Following Instructions
Fried, D., Andreas, J., & Klein, D. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1177 · Cited by 64
Director: Generator-Classifiers For Supervised Language Modeling
Arora, K., Shuster, K., Sukhbaatar, S., & Weston, J. (2022) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2022.aacl-main.39 · Cited by 62
Scaling Laws for Reward Model Overoptimization
Gao, L., Schulman, J., & Hilton, J. (2022) · arXiv · doi:10.48550/arxiv.2210.10760 · Cited by 62
Sample Efficient Text Summarization Using a Single Pre-Trained Transformer
Khandelwal, U., Clark, K., Jurafsky, D., & Kaiser, Ł. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08836 · Cited by 59
Anthropomorphization of AI: Opportunities and Risks
Deshpande, A., Rajpurohit, T., Narasimhan, K., & Kalyan, A. (2023) · Proceedings of the Natural Legal Language Processing Workshop 2023 · doi:10.18653/v1/2023.nllp-1.1 · Cited by 57
Self-citationRamamurthy, R., Ammanabrolu, P., Brantley, K., Hessel, J., Sifa, R., Bauckhage, C., Hajishirzi, H., & Choi, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.01241 · Cited by 56
Director: Generator-Classifiers For Supervised Language Modeling
Arora, K., Shuster, K., Sukhbaatar, S., & Weston, J. (2022) · AACL · doi:10.48550/arxiv.2206.07694 · Cited by 55
Teaching language models to support answers with verified quotes
Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.11147 · Cited by 53 · cited 2×
“This follows the intuition, that the policy model can be implicitly used as an RM for training itself based on the collected feedback.”
Models of human behaviour and confidence in judgement: A review
O’Connor, M. J. (1989) · International Journal of Forecasting · doi:10.1016/0169-2070(89)90083-6 · Cited by 50
OpenAI, :, Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., et al. (2024) · arXiv · Cited by 50
Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation
Kiegeland, S., & Kreutzer, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.133 · Cited by 48
Towards Measuring the Representation of Subjective Global Opinions in Language Models
Durmus, E., Nguyen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds, Z., Hernandez, D., Joseph, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.16388 · Cited by 48
Self-consuming generative models go MAD
Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., & Baraniuk, R. G. (2023) · arXiv · doi:10.48550/arxiv.2307.01850 · Cited by 44
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.00955 · Cited by 44
Minimum Risk Training for Neural Machine TranslationPreprint
Shen, S., Cheng, Y., He, Z., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02433 · Cited by 43
A Survey of Reinforcement Learning from Human Feedback
Kaufmann, T., Weng, P., Bengs, V., & Hüllermeier, E. (2023) · arXiv · doi:10.48550/arxiv.2312.14925 · Cited by 43
Calibrating Sequence likelihood Improves Conditional Language Generation
Zhao, Y., Khalman, M., Joshi, R., Narayan, S., Saleh, M., & Liu, P. J. (2022) · arXiv · doi:10.48550/arxiv.2210.00045 · Cited by 42
RRHF: Rank Responses to Align Language Models with Human Feedback without tears
Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., & Huang, F. (2023) · arXiv · doi:10.48550/arxiv.2304.05302 · Cited by 41
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N. A., Ostendorf, M., & Hajishirzi, H. (2023) · arXiv · doi:10.48550/arxiv.2306.01693 · Cited by 40
Kreutzer, J., Uyheng, J., & Riezler, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1165 · Cited by 38 · cited 6×
“These approaches include reward shaping, where the original reward signal is augmented with additional feedback [56, 125]; intrinsic motivation, which encourages exploration and learning through internal rewards based on novelty, curiosity, or learning progress [17, 135, 140]; and multi-objective…”
InfluentialImproving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Lawrence, C., & Riezler, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1169 · Cited by 38 · cited 1×
“These are gradient-based methods, called policy-gradient algorithms, that directly update the parameters of the policy using the evaluative reward feedback.”
Language Model Alignment with Elastic Reset
Noukhovitch, M., Lavoie, S., Strub, F., & Courville, A. C. (2023) · neural information processing systems · doi:10.52202/075280-0152 · Cited by 35
Deep Bayesian Reward Learning from PreferencesPreprint
Brown, D. S., & Niekum, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.04472 · Cited by 34
Investigations of Performance and Bias in Human-AI Teamwork in Hiring
Peng, A., Nushi, B., Kiciman, E., Inkpen, K., & Kamar, E. (2022) · arXiv · doi:10.48550/arxiv.2202.11812 · Cited by 34
Safe RLHF: Safe Reinforcement Learning from Human Feedback
Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., & Yang, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.12773 · Cited by 32
Interactive Learning from Activity DescriptionPreprint
Nguyen, K., Misra, D. K., Schapire, R. E., Dudík, M., & Shafto, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07024 · Cited by 32
Multi-objective decision-theoretic planning
Roijers, D. M. (2016) · AI Matters · doi:10.1145/3008665.3008670 · Cited by 31 · cited 1×
“…reward signal is augmented with additional feedback [56, 125]; intrinsic motivation, which encourages exploration and learning through internal rewards based on novelty, curiosity, or learning progress [17, 135, 140]; and multi-objective optimization with multiple reward signals [157, 158].”
Zhou, W., & Xu, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6521 · Cited by 30 · cited 1×
“These are gradient-based methods, called policy-gradient algorithms, that directly update the parameters of the policy using the evaluative reward feedback.”
Chain of Hindsight Aligns Language Models with Feedback
Liu, H., Sferrazza, C., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2302.02676 · Cited by 30
KTO: Model Alignment as Prospect Theoretic Optimization
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., & Kiela, D. (2024) · arXiv · doi:10.48550/arxiv.2402.01306 · Cited by 29
On the Weaknesses of Reinforcement Learning for Neural Machine Translation
Choshen, L., Fox, L., Aizenbud, Z., & Abend, O. (2019) · arXiv · doi:10.48550/arxiv.1907.01752 · Cited by 28
(2023) · doi:10.52202/075280-3059 · Cited by 27
Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., & Hakkani-Tur, D. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8608 · Cited by 26 · cited 3×
“Sparse rewards can lead to sample-inefficient learning due to extensive exploration requirements [17].”
Solving math word problems with process- and outcome-based feedback
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., & Higgins, I. (2022) · arXiv · doi:10.48550/arxiv.2211.14275 · Cited by 25
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
Pan, A., Bhatia, K., & Steinhardt, J. (2022) · arXiv · doi:10.48550/arxiv.2201.03544 · Cited by 25
Scaling Laws and Interpretability of Learning from Repeated Data
Hernandez, D., Brown, T., Conerly, T., DasSarma, N., Drain, D., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Henighan, T., Hume, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10487 · Cited by 24
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
Yin, Y., Chen, C., Shang, L., Jiang, X., Chen, X., & Liu, Q. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.400 · Cited by 24
Pretraining Language Models with Human Preferences
Korbak, T., Shi, K., Chen, A., Bhalerao, R., Buckley, C. L., Phang, J., Bowman, S. R., & Perez, E. (2023) · arXiv · doi:10.48550/arxiv.2302.08582 · Cited by 23 · cited 2×
“Future research should explore the integration of these techniques to address the unique challenges in training LLMs with RLHF, potentially improving generalization and robustness.”
Paraphrase Generation with Deep Reinforcement LearningPreprint
Li, Z., Jiang, X., Shang, L., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00279 · Cited by 21
A General Theoretical Paradigm to Understand Learning from Human Preferences
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., & Munos, R. (2023) · arXiv · doi:10.48550/arxiv.2310.12036 · Cited by 21
2008 7th IEEE International Conference on Development and Learning
(2008) · doi:10.1109/icdl14527.2008 · Cited by 20
Ramé, A., Couairon, G., Shukor, M., Dancette, C., Gaya, J.-B., Soulier, L., & Cord, M. (2023) · arXiv · doi:10.48550/arxiv.2306.04488 · Cited by 20
Continually Improving Extractive QA via Human Feedback
Gao, G., Chen, H., Artzi, Y., & Choi, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.27 · Cited by 18
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17
DataMUX: Data Multiplexing for Neural Networks
Murahari, V., Jimenez, C., Yang, R., & Narasimhan, K. (2022) · neural information processing systems · doi:10.52202/068431-1273 · Cited by 16
Self-citationMake The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback
Nguyen, D.-H., Nghiem, N.-V.-D., Nguyen, B.-S., et al. (2022) · NAACL-HLT · doi:10.48550/arxiv.2204.05512 · Cited by 15
TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models
Yang, Z., Cui, Y., & Chen, Z. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.4 · Cited by 14
Training Language Models with Language Feedback at Scale
Scheurer, J., Campos, J. A., Korbak, T., Chan, J. S., Chen, A., Cho, K., & Perez, E. (2023) · arXiv · doi:10.48550/arxiv.2303.16755 · Cited by 14 · cited 18×
“Challenges with Sparse Rewards in Traditional RL. RL has conventionally employed delayed and sparse rewards, where agents receive scalar feedback at the end of a trajectory or episode [181].”
InfluentialBandit Structured Prediction for Learning from Partial Feedback in Statistical Machine Translation
Sokolov, A., Riezler, S., & Urvoy, T. (2016) · arXiv · doi:10.48550/arxiv.1601.04468 · Cited by 14
Anthropomorphization of AI: Opportunities and RisksPreprint
Deshpande, A., Rajpurohit, T., Narasimhan, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.14784 · Cited by 14
Self-citationThe Wisdom of Hindsight Makes Language Models Better Instruction Followers
Zhang, T., Liu, F., Wong, J., Abbeel, P., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2302.05206 · Cited by 13
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15124 · Cited by 12 · cited 2×
“At the same time Tulu-3 [89] used PPO additionally used rule-based rewards in verifiable domains (such as math, instruction following) instead of RMs. Similarly DeepSeek [35] and Qwen [146] series of LLMs have shown remarkable performance improvements using RLHF for a variety of capabilities,…”
Second Thoughts are Best: Learning to Re-Align With Human Values from Text EditsPreprint
Liu, R., Jia, C., Zhang, G., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.00355 · Cited by 12
SimPO: Simple Preference Optimization with a Reference-Free Reward
Meng, Y., Xia, M., & Chen, D. (2024) · arXiv · doi:10.48550/arxiv.2405.14734 · Cited by 12
SLiC-HF: Sequence Likelihood Calibration with Human Feedback
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., & Liu, P. J. (2023) · arXiv · doi:10.48550/arxiv.2305.10425 · Cited by 11
Training Language Models with Language Feedback
Scheurer, J., Campos, J. A., Chan, J. S., Chen, A., Cho, K., & Perez, E. (2022) · arXiv · doi:10.48550/arxiv.2204.14146 · Cited by 11
Mitigating the Alignment Tax of RLHF
Lin, Y., Lin, H., Xiong, W., Diao, S., Liu, J., Zhang, J., Pan, R., Wang, H., Hu, W., Zhang, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.06256 · Cited by 10
A framework for mitigating malicious RLHF feedback in LLM training using consensus based reward
Haider, Z., Rahman, M. H., Devabhaktuni, V., Moeykens, S., & Chakraborty, P. (2025) · Scientific Reports · doi:10.1038/s41598-025-92889-7 · Cited by 9 · cited 1×
“Various approaches have been proposed, such as SAFE RLHF with dual objectives for helpfulness and harmlessness [33], Equilibrate RLHF using fine-grained data-centric alignment [183], and the COBRA framework for mitigating biases by aggregating RMs [60].”
InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling
Miao, Y., Zhang, S., Ding, L., Bao, R., Zhang, L., & Tao, D. (2024) · arXiv · doi:10.48550/arxiv.2402.09345 · Cited by 9
RL with KL penalties is better viewed as Bayesian inference
Korbak, T., Perez, E., & Buckley, C. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.77 · Cited by 8
Research works printed but not linked: 11
- Andrea V. Bajcsy, Dylan P. Losey, Marcia Kilchenman O’Malley, and Anca D. Dragan. 2017. Learning robot objectives from physical human interaction. In Proceedings of the Conference on Robot Learning. Retrieved from https://api.semanticscholar.org/CorpusID:28406224
- Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. 2016. Unifying count-based exploration and intrinsic motivation. In Proceedings of the NIPS. Retrieved from https://api.semanticscholar.org/CorpusID:8310565
- 10.5555/560669 · doi:10.5555/560669
- DeepSeek-AI Aixin Liu Bei Feng Bing Xue Bingxuan Wang Bochao Wu Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang et al. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL]. https://arxiv.org/abs/2412.19437. arXiv:2310.09520 [cs.CL]. · arXiv
- Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the LAW@ACL.
- 10.5555/1622737.1622748 · doi:10.5555/1622737.1622748
- Andrei Andreevich Markov. 1954. The theory of algorithms. Trudy Matematicheskogo Instituta Imeni VA Steklova 42 (1954), 3–375.
- A. Ng, Daishi Harada, and Stuart J. Russell. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the International Conference on Machine Learning. Retrieved from https://api.semanticscholar.org/CorpusID:5730166
- Silviu Pitis. 2023. Failure modes of learning reward models for LLMs and other sequence models. In Proceedings of the ICML 2023 Workshop The Many Facets of Preference-Based Learning. Retrieved from https://openreview.net/forum?id=NjOoxFRZA4
- Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, L’eonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, and Sylvain Gelly. 2021. What matters for on-policy deep actor-critic methods? A large-scale study. In Proceedings of the International Conference on Learning Representations. Retrieved from https://api.semanticscholar.org/CorpusID:233340556
- Deepak Ramachandran and Eyal Amir. 2007. Bayesian inverse reinforcement learning. In Proceedings of the IJCAI, Vol. 7. 2586–2591.
Cited by
Large language models: an overview of foundational architectures, recent trends, and a new taxonomy
Mienye, I. D., Jere, N., Obaido, G., Ogunruku, O. O., Esenogho, E., & Modisane, C. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07668-w · Cited by 46
“Reinforcement Learning with Human Feedback (RLHF) aligns LLM outputs with human preferences by incorporating human evaluations into the training loop [55, 56].”
Miedema, E., Waschull, S., & Emmanouilidis, C. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104409 · Cited by 48
Edge large language models: a comprehensive survey
Jiang, S., Zhou, X., Zhang, M., Xu, C., Liao, G., Chen, J., & Cao, J. (2026) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-025-00227-7 · Cited by 24
RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders
Yang, Z., Sun, A., Zhao, Y. X., Yang, Y., Li, D., & Zhou, C. (2025) · International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC) · doi:10.1109/aiotc66747.2025.11198794 · Cited by 15
“Recent Reinforcement Learning with Human Feedback (RLHF) has shown positive results in accounting for the LLM output to human preferences in instruction-following trials[14].”
A Survey on Human Preference Learning for Aligning Large Language Models
Jiang, R., Chen, K., Bai, X., He, Z., Li, J., Yang, M., Zhao, T., Nie, L., & Zhang, M. (2025) · ACM Computing Surveys · doi:10.1145/3773279 · Cited by 85
Metadata
- Authors
- Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B.
- Affiliations
- University of Massachusetts Amherst (US)
- Carnegie Mellon University (US)
- Princeton University (US)
- Independent Researcher
- University of Massachusetts Amherst
- ORCID
- Venue
- ACM Computing Surveys
- Published
- 2025-09-10
- Volume
- 58
- Issue
- 2
- Pages
- 1-37
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4411063470
- Cited by
- 92
- References
- 36
- Citations by year
- 2026: 60 · 2025: 26
- FWCI
- 34.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- ACM Computing Surveys (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- reinforcement learning from human feedback, large language models, reward modeling, function approximation, model misspecification, generalization, sparse feedback, human preference alignment, language model alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.