Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Open accessAbstract
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RLHF-PPO has shown promising results but suffers from sensitivity to hyperparameters, the need for training additional models, and potential misalignment of the RM.
“While RLHF-PPO has shown promising results, it suffers from sensitivity to hyperparameters, the need for training additional models, and potential misalignment of the RM [138,149,174,197,220].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The Introduction (page unavailable) says RLHF produces impressive results but requires training multiple models, including a reward model; the Abstract calls the procedure complex and often unstable. The Abstract and Discussion contrast DPO’s limited hyperparameter tuning with RLHF, but the cited passages do not specifically establish PPO’s sensitivity to hyperparameters or potential misalignment of its reward model. (LLM, conf=0.85)
Experimental — accuracy not yet measuredTraining RLHF models in practice is very difficult for practitioners owing to unstable training, hyperparameter sensitivity, and high memory usage from loading multiple models.
“Further, training RLHF models in practice is very difficult for practitioners owing to unstable training [28], hyperparameter sensitivity [149,210], loading multiple models leading to high memory usage [159].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The Introduction says the RLHF pipeline involves training multiple language models and incurs significant computational costs. It does not specifically say that loading multiple models causes high memory usage. The other difficulties in the citing sentence are attached to different citations. (LLM, conf=0.82)
Experimental — accuracy not yet measuredIn natural language and image generation, algorithmic methods have been developed that improve output quality beyond sampling from the pretrained model alone.
“Promisingly, in these domains, algorithmic techniques have emerged to improve the quality of generations compared with sampling from the pretrained model alone (83–85).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
The Related Work section says fine-tuning language models with human preferences improves performance on several text-generation tasks, and Section 6.2 reports better summarization results for DPO than for other methods. This supports the claim for natural-language generation, but the paper does not establish it for image generation. (LLM, conf=0.87)
Experimental — accuracy not yet measured'Finetune GPT-2 for QA' can be approached with fairly standard DPO techniques.
“Similarly, 'Finetune GPT-2 for QA' can be approached with fairly standard DPO techniques [58].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [58]
The “DPO outline” section describes a general procedure for fine-tuning a language model using preferred and rejected responses, and Appendix B says DPO is relatively straightforward to implement. Section 6.2 demonstrates it on summarization and single-turn dialogue, but our copy does not establish its use for GPT-2 question answering specifically. The proposed application is plausible, rather than one the cited paper demonstrates. (LLM, conf=0.84)
Experimental — accuracy not yet measuredGenerative models have demonstrated unprecedented capabilities to process, understand, and generate human-like text.
“These generative models have demonstrated unprecedented capabilities to process, understand, and generate human-like text.”
DPO has significantly lower computational cost while maintaining similar performance.
“Interestingly, both Llama-3 and Tulu-2 [69] utilized some variant of DPO [149] citing significantly lower computational cost while maintaining similar performance.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
Both Llama-3 and Tulu-2 utilized some variant of DPO.
“Interestingly, both Llama-3 and Tulu-2 [69] utilized some variant of DPO [149] citing significantly lower computational cost while maintaining similar performance.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
RLHF is highly sensitive to hyperparameters, necessitating a significant amount of hyperparameter tuning.
“Moreover, it is highly sensitive to hyperparameters, necessitating a significant amount of hyperparameter tuning [149,210].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
Azar et al. and Rafailov et al. reformulate the objective encoded in the RLHF PPO algorithm and train the model directly on the new objective, without the need for a separate RM.
“Azar et al. [9], Rafailov et al. [149] further reformulate the objective encoded in the RLHF PPO algorithm, and train the model directly on the new objective, without the need for a separate RM.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The limitations resemble those of natural language models, which have improved through scaling, labeled data, prompt engineering, and human-preference alignment.
“These limitations mirror the constraints of natural language models, which have been improved over time with increased scale, labeled data, prompt engineering, and alignment with human preferences (39, 83–85, 87).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
Sampling directly from a large natural language model usually yields text that is grammatical but locally favors simpler sentences and is globally incoherent, particularly over long spans.
“For example, directly sampling from a large natural language model typically produces sequences that are grammatically correct yet locally biased toward simpler sentence constructions and that are globally incoherent, especially at long lengths.”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
The Introduction discusses why language-model outputs may need steering, but does not describe the claimed grammaticality, preference for simpler sentences, or long-span incoherence. The Additional information section discusses meaningless responses from an unlikelihood-training baseline, not the typical results of directly sampling a large language model. Searches for the claim’s distinctive wording and for long-text incoherence found no supporting passage in our copy; page numbers were not available. (LLM, conf=0.76)
Experimental — accuracy not yet measuredNatural language models frequently have difficulty producing coherent and diverse text over long sequences.
“Natural language models often struggle to maintain coherent and diverse generation over long sequences”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
The Additional information section says that an unlikelihood-training baseline produced meaningless responses on summarization and dialogue tasks, but it does not discuss whether language models generally struggle with coherence and diversity over long sequences. Searches for coherent and diverse long-sequence generation found no passage addressing that claim; the copy provides no page number for the cited section. (LLM, conf=0.72)
Experimental — accuracy not yet measuredRecent advancements in artificial intelligence, particularly generative large language models, are rapidly transforming key aspects of health and healthcare, including clinical research informatics.
“Recent advancements of artificial intelligence (AI), particularly the rise of generative large language models (LLMs), are rapidly transforming key aspects of health and healthcare, including clinical research informatics (CRI).”
From Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently (2025) · cites this paper as [2]
The Introduction and Related Work discuss language-model capabilities and improvements in tasks such as translation and summarization, but do not address transformation of health, healthcare, or clinical research informatics. Searches for those healthcare terms and for medical or clinical applications found no supporting passage in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 32
- H. Bong and A. Rinaldo. Generalized results for the existence and consistency of the MLE in the Bradley-Terry-Luce model. International Conference on Machine Learning, 2022. arXiv:2110.11487. · arXiv
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877- 1901.
- Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_ files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023. arXiv preprint arXiv:2303.12712. · arXiv
- Y. Chen, R. Wang, H. Jiang, S. Shi, and R.-L. Xu. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. ArXiv, abs/2304.00723, 2023. · arXiv
- A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. · arXiv
- P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 30.
- Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/ paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf.
- M. Dudík, K. Hofmann, R. E. Schapire, A. Slivkins, and M. Zoghi. Contextual dueling bandits. In P. Grünwald, E. Hazan, and S. Kale, editors, Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pages 563-587, Paris, France, 03-06 Jul 2015. PMLR. URL https://proceedings.mlr.press/v40/Dudik15.html.
- D. Go, T. Korbak, G. Kruszewski, J. Rozen, N. Ryu, and M. Dymetman. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
- A. Jain, B. Wojcik, T. Joachims, and A. Saxena. Learning trajectory preferences for manip- ulators via iterative improvement. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26.
- Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/ 2013/file/c058f544c737782deacefa532d9add4c-Paper.pdf.
- N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pages 1645-1654. PMLR, 2017.
- N. Jaques, J. H. Shen, A. Ghandeharioun, C. Ferguson, A. Lapedriza, N. Jones, S. S. Gu, and R. Picard. Human-centric dialog training via offline reinforcement learning. arXiv preprint arXiv:2010.05848, 2020. · arXiv
- T. Korbak, H. Elsahar, G. Kruszewski, and M. Dymetman. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 16203-16220.
- Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 67496dfa96afddab795530cc7c69b57a-Paper-Conference.pdf.
- S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018.
- A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/ anthology/P11-1015.
- L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730-27744.
- Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
- R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018. URL https://openreview. net/forum?id=HkAClQgA-.
- X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. · arXiv
- J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
- R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview. net/forum?id=8aHzds2uUyB.
- M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. CoRR, abs/1511.06732, 2015. · arXiv
- A. Saha, A. Pacchiano, and J. Lee. Dueling rl: Reinforcement learning with trajectory preferences. In F. Ruiz, J. Dy, and J.-W. van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceed- ings of Machine Learning Research, pages 6263-6289. PMLR, 25-27 Apr 2023. URL https://proceedings.mlr.press/v206/saha23a.html.
- V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chh- ablani, N. Nayak, D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. A. Fries, R. Teehan, T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush. Multi- task prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4.
- H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. · arXiv
- S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319, 2019. · arXiv
- Y. Wu and B. Hu. Learning to extract coherent summary via deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'18/IAAI'18/EAAI'18. AAAI Press, 2018. ISBN 978-1-57735-800-8.
- X. Yan, C. Luo, C. L. A. Clarke, N. Craswell, E. M. Voorhees, and P. Castells. Human preferences as dueling bandits. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 567-577, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450387323. doi: 10.1145/3477495.3531991. URL https://doi.org/10.1145/3477495.3531991. · DOI
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580 · see citing works
Self-citationEmpowering biomedical discovery with AI agents
Gao, S., Fang, A., Huang, Y., Giunchiglia, V., Noori, A., Schwarz, J. R., Ektefaie, Y., Kondic, J., & Zitnik, M. (2024) · Cell · doi:10.1016/j.cell.2024.09.022 · Cited by 315 · see citing works
A review of large language models and autonomous agents in chemistry
Ramos, M. C., Collison, C. J., & White, A. D. (2025) · Chemical Science · doi:10.1039/d4sc03921a · Cited by 274 · see citing works
Cascella, M., Semeraro, F., Montomoli, J., Bellini, V., Piazza, O., & Bignami, E. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02045-3 · Cited by 219 · see citing works
(2024) · doi:10.1145/3704435 · Cited by 184 · see citing works
Tool Learning with Foundation Models
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.08354 · Cited by 184 · see citing works
A generalist medical language model for disease diagnosis assistance
Liu, X., Liu, H., Yang, G., Jiang, Z., Cui, S., Zhang, Z., Wang, H., Tao, L., Sun, Y., Song, Z., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03416-6 · Cited by 183 · see citing works
Diffusion Model Alignment Using Direct Preference Optimization
Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., & Naik, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00786 · Cited by 185 · see citing works
Self-citationHagos, D. H., Battle, R., & Rawat, D. B. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3444742 · Cited by 192 · see citing works
Detecting and Preventing Hallucinations in Large Vision Language Models
Gunjal, A., Yin, J., & Bas, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29771 · Cited by 161 · see citing works
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04088 · Cited by 143 · see citing works
Large language models for disease diagnosis: a scoping review
Zhou, S., Xu, Z., Zhang, M., Xu, C., Guo, Y., Zhan, Z., Fang, Y., Ding, S., Wang, J., Xu, K., et al. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00011-z · Cited by 143 · see citing works
Cui, C., Ma, Y., Cao, X., Ye, W., & Wang, Z. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00101 · Cited by 132 · see citing works
Multimodal generative AI for medical image interpretation
Rao, V. M., Hla, M., Moor, M., Adithan, S., Kwak, S., Topol, E. J., & Rajpurkar, P. (2025) · Nature · doi:10.1038/s41586-025-08675-y · Cited by 128 · see citing works
The life cycle of large language models in education: A framework for understanding sources of bias
Lee, J., Hicke, Y., Yu, R., Brooks, C., & Kizilcec, R. F. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13505 · Cited by 119 · see citing works
Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., & Sun, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01310 · Cited by 156 · see citing works
Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles
Cui, C., Ma, Y., Cao, X., Ye, W., & Wang, Z. (2024) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2024.3381793 · Cited by 121 · see citing works
A comprehensive survey of large language models and multimodal large language models in medicine
Xiao, H., Zhou, F., Liu, X., Liu, T., Ли, Ж., Liu, X., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102888 · Cited by 130 · see citing works
Lu, W., Luu, R. K., & Buehler, M. J. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01564-y · Cited by 126 · see citing works
Rethinking machine unlearning for large language models
Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Yao, Y., Liu, C. Y., Xu, X., Li, H., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-00985-0 · Cited by 173 · see citing works
A survey on interpretable reinforcement learning
Glanois, C., Weng, P., Zimmer, M., Li, D., Yang, T., Hao, J., & Liu, W. (2024) · Machine Learning · doi:10.1007/s10994-024-06543-w · Cited by 138 · see citing works
Recent Advances in Large Language Models for Healthcare
Nassiri, K., & Akhloufi, M. A. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4020062 · Cited by 105 · see citing works
ChatGPT: perspectives from human–computer interaction and psychology
Liu, J. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1418869 · Cited by 99 · see citing works
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
LLM Fine-Tuning: Concepts, Opportunities, and Challenges
Wu, X.-K., Chen, M., Li, W., Wang, R., Lu, L., Liu, J., Hwang, K., Hao, Y., Pan, Y., Meng, Q., et al. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9040087 · Cited by 94 · see citing works
A tutorial on open-source large language models for behavioral science
Hussain, Z., Binz, M., Mata, R., & Wulff, D. U. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02455-8 · Cited by 91 · see citing works
Preference Ranking Optimization for Human Alignment
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., & Wang, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29865 · Cited by 77 · see citing works
Preference Ranking Optimization for Human Alignment
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2306.17492 · Cited by 77 · see citing works
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
Multi-Modal Hallucination Control by Visual Information Grounding
Favero, A., Zancato, L., Trager, M., Choudhary, S., Perera, P., Achille, A., Swaminathan, A., & Soatto, S. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.01356 · Cited by 299 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69 · see citing works
Han, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 72 · see citing works
Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model
Yang, K., Tao, J., Lyu, J., Ge, C., Chen, J., Shen, W., Zhu, X., & Li, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00854 · Cited by 57 · see citing works
Dahlgren Lindström, A., Methnani, L., Krause, L., Ericson, P., de Rituerto de Troya, Í. M., Coelho Mollo, D., & Dobbe, R. (2025) · Ethics and Information Technology · doi:10.1007/s10676-025-09837-2 · Cited by 54 · see citing works
“DPO, like PPO-based RLHF, still relies on human-annotated preference data.”
InfluentialFranceschelli, G., & Musolesi, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15278 · Cited by 52 · see citing works
Efficient GPT-4V level multimodal large language model for deployment on edge devices
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Chen, C., Li, H., Zhao, W., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-61040-5 · Cited by 50 · see citing works
Nie, T., Sun, J., & Ma, W. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100003 · Cited by 56 · see citing works
Qwen2.5-Coder Technical Report
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.12186 · Cited by 65 · see citing works
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 57 · see citing works
Scaling Unlocks Broader Generation and Deeper Functional Understanding of Proteins
Bhatnagar, A., Jain, S., Beazer, J., Curran, S. C., Hoffnagle, A. M., Ching, K., Martyn, M., Nayfach, S., Ruffolo, J. A., & Madani, A. (2025) · neural information processing systems · doi:10.52202/085713-1536 · Cited by 46 · see citing works
Systematic Biases in LLM Simulations of Debates
Taubenfeld, A., Dover, Y., Reichart, R., & Goldstein, A. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.16 · Cited by 47 · see citing works
Are large language models qualified reviewers in originality evaluation?
Huang, S., Huang, Y., Liu, Y., Luo, Z., & Lu, W. P. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103973 · Cited by 41 · see citing works
TurkishBERTweet: Fast and reliable large language model for social media analysis
Najafi, A., & Varol, O. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124737 · Cited by 48 · see citing works
Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld
Yang, Y., Zhou, T., Li, K., Tao, D., Li, L., Shen, L., He, X., Jiang, J., & Shi, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02482 · Cited by 43 · see citing works
Shashikumar, S. P., Mohammadi, S., Krishnamoorthy, R., Patel, A., Wardi, G., Ahn, J. C., Singh, K., Aronoff-Spencer, E. S., & Nemati, S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01689-w · Cited by 39 · see citing works
Yu, L., Huang, Z., Yuan, H., Cheng, S., Yang, L., Zhang, F., Shen, C., Ma, J., Zhang, J., Lu, J., et al. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728878 · Cited by 40 · see citing works
Large models in medical imaging: Advances and prospects
Fang, M., Wang, Z., Pan, S., Feng, X., Zhao, Y., Hou, D., Wu, L., Xie, X., Zhang, X.-Y., Tian, J., et al. (2025) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003699 · Cited by 39 · see citing works
Ding, W., Abdel‐Basset, M., Ali, A. M., & Moustafa, N. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112663 · Cited by 41 · see citing works
Beyond Preferences in AI Alignment
Zhi-Xuan, T., Carroll, M., Franklin, M., & Ashton, H. (2024) · Philosophical Studies · doi:10.1007/s11098-024-02249-w · Cited by 85 · see citing works
Deliberative Alignment: Reasoning Enables Safer Language Models
Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., Dias, R., Vallone, A., Ren, H., Wei, J., et al. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15159 · Cited by 106 · see citing works
Aligning protein generative models with experimental fitness via Direct Preference OptimizationPreprint
Widatalla, T., Rafailov, R., & Hie, B. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.05.20.595026 · Cited by 35 · see citing works
Self-citationRisks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation
Grabb, D., Lamparth, M., & Vasan, N. (2024) · doi:10.1101/2024.04.07.24305462 · Cited by 40 · see citing works
Kozlowski, A. C., & Evans, J. A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251337316 · Cited by 34 · see citing works
Xu, L., Zhang, J., Li, B., Wang, J., Chen, S., Zhao, W. X., & Wen, J.-R. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3726871 · Cited by 60 · see citing works
A call for built-in biosecurity safeguards for generative AI tools
Wang, M., Zhang, Z., Bedi, A. S., Velasquez, A., Guerra, S., Lin‐Gibson, S., Cong, L., Qu, Y., Chakraborty, S., Blewett, M. M., et al. (2025) · Nature Biotechnology · doi:10.1038/s41587-025-02650-8 · Cited by 35 · see citing works
Heinzinger, M., & Rost, B. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.102986 · Cited by 25 · see citing works
Reasoning beyond limits: Advances and open problems for LLMs
Ferrag, M. A., Tihanyi, N., & Debbah, M. A. (2025) · ICT Express · doi:10.1016/j.icte.2025.09.003 · Cited by 33 · see citing works
Protein Language Model Fitness Is a Matter of PreferencePreprint
Gordon, C., Lu, A. X., & Abbeel, P. (2024) · bioRxiv · doi:10.1101/2024.10.03.616542 · Cited by 36 · see citing works
Towards a mixed human–machine creativity
Farina, M., Pedrycz, W., & Lavazza, A. (2024) · Journal of Cultural Cognitive Science · doi:10.1007/s41809-024-00146-6 · Cited by 27 · see citing works
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2306.04930 · Cited by 22 · see citing works
Negative chemical data boosts language models in reaction outcome prediction
Toniato, A., Vaucher, A. C., Laino, T., & Graziani, M. (2025) · Science Advances · doi:10.1126/sciadv.adt5578 · Cited by 23 · see citing works
Large Language Models in peri-implant disease: How well do they perform?
Koidou, V. P., Chatzopoulos, G. S., Tsalikis, L., & Kaklamanos, E. G. (2025) · Journal of Prosthetic Dentistry · doi:10.1016/j.prosdent.2025.02.008 · Cited by 22 · see citing works
A reliable knowledge processing framework for combustion science using foundation models
Sharma, V., & Raman, V. (2024) · Energy and AI · doi:10.1016/j.egyai.2024.100365 · Cited by 22 · see citing works
Development of AI-assisted microscopy frameworks through realistic simulation with pySTED
Bilodeau, A., Michaud-Gagnon, A., Chabbert, J., Turcotte, B., Heine, J., Durand, A., & Lavoie‐Cardinal, F. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00903-w · Cited by 25 · see citing works
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28878 · Cited by 23 · see citing works
Hung, T. K. W., Kuperman, G. J., Sherman, E. J., Ho, A. L., Weng, C., Pfister, D. G., & Mao, J. J. J. (2024) · Journal of Medical Internet Research · doi:10.2196/60695 · Cited by 24 · see citing works
A survey of communicating robot learning during human-robot interaction
Habibian, S., Valdivia, A. A., Blumenschein, L. H., & Losey, D. P. (2024) · The International Journal of Robotics Research · doi:10.1177/02783649241281369 · Cited by 24 · see citing works
Large language models for bioinformatics
Ruan, W., Lyu, Y., Zhang, J., Cai, J., Shu, P., Yang, G., Lu, Y., Gao, S., Wang, Y., Wang, P., et al. (2025) · Quantitative Biology · doi:10.1002/qub2.70014 · Cited by 35 · see citing works
How to Protect Copyright Data in Optimization of Large Language Models?
Chu, T., Song, Z., & Yang, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29741 · Cited by 21 · see citing works
Natural Language Processing with Commonsense Knowledge: A SurveyPreprint
Xie, Y., Liu, Z., Ma, Z., Meng, F., Xiao, Y., Miao, F., & Pu, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04674 · Cited by 23 · see citing works
In Situ Graph Reasoning and Knowledge Expansion Using Graph‐PRefLexOR
Buehler, M. J. (2025) · Advanced Intelligent Discovery · doi:10.1002/aidi.202500006 · Cited by 20 · see citing works
Artificial intelligence security and privacy: a survey
He, X., Xu, G., Han, X., Wang, Q., Zhao, L., Shen, C., Lin, C., Zhao, Z., Li, Q., Yang, L., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-025-4388-5 · Cited by 23 · see citing works
Aggarwal, T., Salatino, A., Osborne, F., & Motta, E. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2025.104262 · Cited by 30 · see citing works
Abbasi, K., Razzaghi, P., Gharizadeh, A., Ghareyazi, A., Dehnad, A., Rabiee, H. R., & Mofrad, M. R. K. (2025) · Pharmacological Reviews · doi:10.1016/j.pharmr.2025.100095 · Cited by 23 · see citing works
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 23 · see citing works
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P. (2026) · IEEE Access · doi:10.1109/access.2026.3675554 · Cited by 28 · see citing works
“Similarly, in [180], the authors introduce SecAlign , which leverages alignment training via direct preference optimization (DPO) [181] to align agents toward preferring benign instructions over adversarial ones.”
Xiong, Y., Zhan, Z., Zhong, C., Zeng, W., Guo, J. X., Tang, W., & Liu, C. (2025) · European Journal Of Dental Education · doi:10.1111/eje.13073 · Cited by 19 · see citing works
Multimodal AI and Large Language Models for Orthopantomography Radiology Report Generation and Q&A
Dasanayaka, C., Dandeniya, K., Dissanayake, M. B., Gunasena, C., & Jayasinghe, R. D. (2025) · Applied System Innovation · doi:10.3390/asi8020039 · Cited by 21 · see citing works
Ma, Y., Ye, W., Cui, C., Zhang, H., Xing, S., Ke, F., Wang, J., Miao, C., Chen, J., Rezatofighi, H., et al. (2025) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw65960.2025.00114 · Cited by 19 · see citing works
Tucano: Advancing neural text generation for Portuguese
Corrêa, N. K., Sen, A., Falk, S., & Fatimah, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101325 · Cited by 25 · see citing works
A Survey on the Integration and Optimization of Large Language Models in Edge Computing Environments
Bhardwaj, S., Singh, P., & Pandit, M. K. (2024) · 2024 16th International Conference on Computer and Automation Engineering (ICCAE) · doi:10.1109/iccae59995.2024.10569285 · Cited by 18 · see citing works
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
Vashurin, R., Fadeeva, E., Vazhentsev, A., Rvanova, L., Vasilev, D., Tsvigun, A., Petrakov, S., Xing, R., Sadallah, A., Grishchenkov, K., et al. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00737 · Cited by 21 · see citing works
Caution Ahead: Numerical Reasoning and Look‐Ahead Bias in AI Models
LEVY, B. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70058 · Cited by 20 · see citing works
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · doi:10.31219/osf.io/ac4sp · Cited by 51 · see citing works
Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings
Wolfe, R., Slaughter, I., Han, B., Wen, B., Yang, Y., Rosenblatt, L., Herman, B., Brown, E., Qu, Z., Weber, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658966 · Cited by 17 · see citing works
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17 · see citing works
Beyond Text: Frozen Large Language Models in Visual Signal Comprehension
Zhu, L., Wei, F., & Lu, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02554 · Cited by 17 · see citing works
Li, C., & Qi, Y. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108687 · Cited by 16 · see citing works
Medical large language model for diagnostic reasoning across specialties
(2025) · Nature Medicine · doi:10.1038/s41591-025-03520-1 · Cited by 14 · see citing works
Rethinking chemical research in the age of large language models
MacKnight, R., Boiko, D. A., Regio, J. E., Gallegos, L. C., Neukomm, T. A., & Gomes, G. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00811-y · Cited by 17 · see citing works
Teaching Large Language Models to Translate with Comparison
Zeng, J., Meng, F., Yin, Y., & Zhou, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29920 · Cited by 32 · see citing works
PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models
Deng, F., Wang, Q., Wei, W., Hou, T., & Grundmann, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00709 · Cited by 15 · see citing works
Comparing the dental knowledge of large language models
Tussie, C., & Starosta, A. (2024) · BDJ · doi:10.1038/s41415-024-8015-2 · Cited by 14 · see citing works
Nagakalyani, G., Chaudhary, S., Apte, V., Ramakrishnan, G., & Tamilselvam, S. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701913 · Cited by 14 · see citing works
Language models for protein design
Lee, J. S., Abdin, O., & Kim, P. M. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.103027 · Cited by 15 · see citing works
Zhu, C., Hu, X., Wu, H., Qin, C., Zhu, H., & Xiong, H. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104261 · Cited by 16 · see citing works
Kirk, H. R., Bean, A., Vidgen, B., Röttger, P., & Hale, S. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.148 · Cited by 26 · see citing works
Are Large Language Models Consistent over Value-laden Questions?
Moore, J., Deshpande, T., & Yang, D. (2024) · arXiv · doi:10.48550/arxiv.2407.02996 · Cited by 12 · see citing works
- Authors
- Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C.
- Affiliations
- Chan Zuckerberg Biohub San Francisco (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.18290
- OpenAlex
- W4378771755
- Cited by
- 293
- References
- 0
- Citations by year
- 2026: 36 · 2025: 149 · 2024: 97 · 2023: 10 · 2021: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Canadian Institute for Advanced Research
- Office of Naval Research (N00014)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- active inference, reward modeling, language models, trauma, affect labeling, amygdala reactivity, Landauer limit, information thermodynamics, deception, allostasis, free energy principle, distributed cognition, extended mind, interpersonal neural synchronization, criticality, integrated information, mode collapse, reinforcement learning from human feedback, AI alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
