The DOI matches its DataCite record; the references it cites are not checked yet.
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Open accessAbstract
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While RLHF-PPO has shown promising results, it suffers from sensitivity to hyperparameters, the need for training additional models, and potential misalignment of the RM [138,149,174,197,220].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Further, training RLHF models in practice is very difficult for practitioners owing to unstable training [28], hyperparameter sensitivity [149,210], loading multiple models leading to high memory usage [159].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Promisingly, in these domains, algorithmic techniques have emerged to improve the quality of generations compared with sampling from the pretrained model alone (83–85).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similarly, 'Finetune GPT-2 for QA' can be approached with fairly standard DPO techniques [58].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [58]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These generative models have demonstrated unprecedented capabilities to process, understand, and generate human-like text.”
The cited source says what the claim attributes to it.
In the citing paper
“Interestingly, both Llama-3 and Tulu-2 [69] utilized some variant of DPO [149] citing significantly lower computational cost while maintaining similar performance.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The cited source says what the claim attributes to it.
In the citing paper
“Interestingly, both Llama-3 and Tulu-2 [69] utilized some variant of DPO [149] citing significantly lower computational cost while maintaining similar performance.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The cited source says what the claim attributes to it.
In the citing paper
“Moreover, it is highly sensitive to hyperparameters, necessitating a significant amount of hyperparameter tuning [149,210].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The cited source says what the claim attributes to it.
In the citing paper
“Azar et al. [9], Rafailov et al. [149] further reformulate the objective encoded in the RLHF PPO algorithm, and train the model directly on the new objective, without the need for a separate RM.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The cited source says what the claim attributes to it.
In the citing paper
“These limitations mirror the constraints of natural language models, which have been improved over time with increased scale, labeled data, prompt engineering, and alignment with human preferences (39, 83–85, 87).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, directly sampling from a large natural language model typically produces sequences that are grammatically correct yet locally biased toward simpler sentence constructions and that are globally incoherent, especially at long lengths.”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Natural language models often struggle to maintain coherent and diverse generation over long sequences”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Recent advancements of artificial intelligence (AI), particularly the rise of generative large language models (LLMs), are rapidly transforming key aspects of health and healthcare, including clinical research informatics (CRI).”
From Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently (2025) · cites this paper as [2]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 32
- H. Bong and A. Rinaldo. Generalized results for the existence and consistency of the MLE in the Bradley-Terry-Luce model. International Conference on Machine Learning, 2022. arXiv:2110.11487. · arXiv
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877- 1901.
- Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_ files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023. arXiv preprint arXiv:2303.12712. · arXiv
- Y. Chen, R. Wang, H. Jiang, S. Shi, and R.-L. Xu. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. ArXiv, abs/2304.00723, 2023. · arXiv
- A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. · arXiv
- P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 30.
- Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/ paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf.
- M. Dudík, K. Hofmann, R. E. Schapire, A. Slivkins, and M. Zoghi. Contextual dueling bandits. In P. Grünwald, E. Hazan, and S. Kale, editors, Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pages 563-587, Paris, France, 03-06 Jul 2015. PMLR. URL https://proceedings.mlr.press/v40/Dudik15.html.
- D. Go, T. Korbak, G. Kruszewski, J. Rozen, N. Ryu, and M. Dymetman. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
- A. Jain, B. Wojcik, T. Joachims, and A. Saxena. Learning trajectory preferences for manip- ulators via iterative improvement. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26.
- Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/ 2013/file/c058f544c737782deacefa532d9add4c-Paper.pdf.
- N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pages 1645-1654. PMLR, 2017.
- N. Jaques, J. H. Shen, A. Ghandeharioun, C. Ferguson, A. Lapedriza, N. Jones, S. S. Gu, and R. Picard. Human-centric dialog training via offline reinforcement learning. arXiv preprint arXiv:2010.05848, 2020. · arXiv
- T. Korbak, H. Elsahar, G. Kruszewski, and M. Dymetman. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 16203-16220.
- Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 67496dfa96afddab795530cc7c69b57a-Paper-Conference.pdf.
- S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018.
- A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/ anthology/P11-1015.
- L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730-27744.
- Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
- R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018. URL https://openreview. net/forum?id=HkAClQgA-.
- X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. · arXiv
- J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
- R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview. net/forum?id=8aHzds2uUyB.
- M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. CoRR, abs/1511.06732, 2015. · arXiv
- A. Saha, A. Pacchiano, and J. Lee. Dueling rl: Reinforcement learning with trajectory preferences. In F. Ruiz, J. Dy, and J.-W. van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceed- ings of Machine Learning Research, pages 6263-6289. PMLR, 25-27 Apr 2023. URL https://proceedings.mlr.press/v206/saha23a.html.
- V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chh- ablani, N. Nayak, D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. A. Fries, R. Teehan, T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush. Multi- task prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4.
- H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. · arXiv
- S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319, 2019. · arXiv
- Y. Wu and B. Hu. Learning to extract coherent summary via deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'18/IAAI'18/EAAI'18. AAAI Press, 2018. ISBN 978-1-57735-800-8.
- X. Yan, C. Luo, C. L. A. Clarke, N. Craswell, E. M. Voorhees, and P. Castells. Human preferences as dueling bandits. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 567-577, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450387323. doi: 10.1145/3477495.3531991. URL https://doi.org/10.1145/3477495.3531991. · DOI
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580
Self-citationEmpowering biomedical discovery with AI agents
Gao, S., Fang, A., Huang, Y., Giunchiglia, V., Noori, A., Schwarz, J. R., Ektefaie, Y., Kondic, J., & Zitnik, M. (2024) · Cell · doi:10.1016/j.cell.2024.09.022 · Cited by 315
A review of large language models and autonomous agents in chemistry
Ramos, M. C., Collison, C. J., & White, A. D. (2025) · Chemical Science · doi:10.1039/d4sc03921a · Cited by 274
Cascella, M., Semeraro, F., Montomoli, J., Bellini, V., Piazza, O., & Bignami, E. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02045-3 · Cited by 219
(2024) · doi:10.1145/3704435 · Cited by 184
Tool Learning with Foundation Models
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.08354 · Cited by 184
A generalist medical language model for disease diagnosis assistance
Liu, X., Liu, H., Yang, G., Jiang, Z., Cui, S., Zhang, Z., Wang, H., Tao, L., Sun, Y., Song, Z., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03416-6 · Cited by 183
Diffusion Model Alignment Using Direct Preference Optimization
Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., & Naik, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00786 · Cited by 359
Self-citationHagos, D. H., Battle, R., & Rawat, D. B. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3444742 · Cited by 227
Detecting and Preventing Hallucinations in Large Vision Language Models
Gunjal, A., Yin, J., & Bas, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29771 · Cited by 161
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04088 · Cited by 143
Large language models for disease diagnosis: a scoping review
Zhou, S., Xu, Z., Zhang, M., Xu, C., Guo, Y., Zhan, Z., Fang, Y., Ding, S., Wang, J., Xu, K., et al. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00011-z · Cited by 188
Cui, C., Ma, Y., Cao, X., Ye, W., & Wang, Z. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00101 · Cited by 156
Multimodal generative AI for medical image interpretation
Rao, V. M., Hla, M., Moor, M., Adithan, S., Kwak, S., Topol, E. J., & Rajpurkar, P. (2025) · Nature · doi:10.1038/s41586-025-08675-y · Cited by 134
The life cycle of large language models in education: A framework for understanding sources of bias
Lee, J., Hicke, Y., Yu, R., Brooks, C., & Kizilcec, R. F. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13505 · Cited by 119
Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., & Sun, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01310 · Cited by 267
Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles
Cui, C., Ma, Y., Cao, X., Ye, W., & Wang, Z. (2024) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2024.3381793 · Cited by 137
A comprehensive survey of large language models and multimodal large language models in medicine
Xiao, H., Zhou, F., Liu, X., Liu, T., Ли, Ж., Liu, X., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102888 · Cited by 154
Lu, W., Luu, R. K., & Buehler, M. J. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01564-y · Cited by 126
Rethinking machine unlearning for large language models
Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Yao, Y., Liu, C. Y., Xu, X., Li, H., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-00985-0 · Cited by 374
A survey on interpretable reinforcement learning
Glanois, C., Weng, P., Zimmer, M., Li, D., Yang, T., Hao, J., & Liu, W. (2024) · Machine Learning · doi:10.1007/s10994-024-06543-w · Cited by 176
Recent Advances in Large Language Models for Healthcare
Nassiri, K., & Akhloufi, M. A. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4020062 · Cited by 108
ChatGPT: perspectives from human–computer interaction and psychology
Liu, J. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1418869 · Cited by 99
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101
LLM Fine-Tuning: Concepts, Opportunities, and Challenges
Wu, X.-K., Chen, M., Li, W., Wang, R., Lu, L., Liu, J., Hwang, K., Hao, Y., Pan, Y., Meng, Q., et al. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9040087 · Cited by 111
A tutorial on open-source large language models for behavioral science
Hussain, Z., Binz, M., Mata, R., & Wulff, D. U. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02455-8 · Cited by 111
Preference Ranking Optimization for Human Alignment
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., & Wang, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29865 · Cited by 77
Preference Ranking Optimization for Human Alignment
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2306.17492 · Cited by 77
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92
Multi-Modal Hallucination Control by Visual Information Grounding
Favero, A., Zancato, L., Trager, M., Choudhary, S., Perera, P., Achille, A., Swaminathan, A. D., & Soatto, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01356 · Cited by 132
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
Han, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 81
Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model
Yang, K., Tao, J., Lyu, J., Ge, C., Chen, J., Shen, W., Zhu, X., & Li, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00854 · Cited by 109
Dahlgren Lindström, A., Methnani, L., Krause, L., Ericson, P., de Rituerto de Troya, Í. M., Coelho Mollo, D., & Dobbe, R. (2025) · Ethics and Information Technology · doi:10.1007/s10676-025-09837-2 · Cited by 54
“DPO, like PPO-based RLHF, still relies on human-annotated preference data.”
InfluentialFranceschelli, G., & Musolesi, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15278 · Cited by 54
Coelho Mollo, D., & Millière, R. (2026) · Philosophy and the Mind Sciences · doi:10.33735/phimisci.2026.12307 · Cited by 48
Efficient GPT-4V level multimodal large language model for deployment on edge devices
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Chen, C., Li, H., Zhao, W., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-61040-5 · Cited by 60
Nie, T., Sun, J., & Ma, W. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100003 · Cited by 70
Qwen2.5-Coder Technical Report
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.12186 · Cited by 65
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 94
Scaling Unlocks Broader Generation and Deeper Functional Understanding of Proteins
Bhatnagar, A., Jain, S., Beazer, J., Curran, S. C., Hoffnagle, A. M., Ching, K., Martyn, M., Nayfach, S., Ruffolo, J. A., & Madani, A. (2025) · neural information processing systems · doi:10.52202/085713-1536 · Cited by 55
Systematic Biases in LLM Simulations of Debates
Taubenfeld, A., Dover, Y., Reichart, R., & Goldstein, A. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.16 · Cited by 47
“This optimization is based on the cutting-edge DPO method (Rafailov et al., 2023), which can be divided into two phases: first, a next-word-prediction phase that acclimates the model to the intended data distribution, followed by a Contrastive Learning phase aimed at teaching the model to…”
Are large language models qualified reviewers in originality evaluation?
Huang, S., Huang, Y., Liu, Y., Luo, Z., & Lu, W. P. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103973 · Cited by 45
TurkishBERTweet: Fast and reliable large language model for social media analysis
Najafi, A., & Varol, O. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124737 · Cited by 54
Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld
Yang, Y., Zhou, T., Li, K., Tao, D., Li, L., Shen, L., He, X., Jiang, J., & Shi, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02482 · Cited by 51
Shashikumar, S. P., Mohammadi, S., Krishnamoorthy, R., Patel, A., Wardi, G., Ahn, J. C., Singh, K., Aronoff-Spencer, E. S., & Nemati, S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01689-w · Cited by 46
Yu, L., Huang, Z., Yuan, H., Cheng, S., Yang, L., Zhang, F., Shen, C., Ma, J., Zhang, J., Lu, J., et al. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728878 · Cited by 40
Large models in medical imaging: Advances and prospects
Fang, M., Wang, Z., Pan, S., Feng, X., Zhao, Y., Hou, D., Wu, L., Xie, X., Zhang, X.-Y., Tian, J., et al. (2025) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003699 · Cited by 43
Ding, W., Abdel‐Basset, M., Ali, A. M., & Moustafa, N. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112663 · Cited by 46
Beyond Preferences in AI Alignment
Zhi‐Xuan, T., Carroll, M., Franklin, M., & Ashton, H. (2024) · Philosophical Studies · doi:10.1007/s11098-024-02249-w · Cited by 82
Deliberative Alignment: Reasoning Enables Safer Language Models
Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., Dias, R., Vallone, A., Ren, H., Wei, J., et al. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15159 · Cited by 390
Aligning protein generative models with experimental fitness via Direct Preference OptimizationPreprint
Widatalla, T., Rafailov, R., & Hie, B. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.05.20.595026 · Cited by 35
Self-citationRisks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation
Grabb, D., Lamparth, M., & Vasan, N. (2024) · doi:10.1101/2024.04.07.24305462 · Cited by 40
Kozlowski, A. C., & Evans, J. A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251337316 · Cited by 49
Xu, L., Zhang, J., Li, B., Wang, J., Chen, S., Zhao, W. X., & Wen, J.-R. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3726871 · Cited by 90
A call for built-in biosecurity safeguards for generative AI tools
Wang, M., Zhang, Z., Bedi, A. S., Velasquez, A., Guerra, S., Lin‐Gibson, S., Cong, L., Qu, Y., Chakraborty, S., Blewett, M. M., et al. (2025) · Nature Biotechnology · doi:10.1038/s41587-025-02650-8 · Cited by 35
Heinzinger, M., & Rost, B. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.102986 · Cited by 27
Reasoning beyond limits: Advances and open problems for LLMs
Ferrag, M. A., Tihanyi, N., & Debbah, M. A. (2025) · ICT Express · doi:10.1016/j.icte.2025.09.003 · Cited by 56
Protein Language Model Fitness Is a Matter of PreferencePreprint
Gordon, C., Lu, A. X., & Abbeel, P. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.03.616542 · Cited by 28
Towards a mixed human–machine creativity
Farina, M., Witold Pedrycz, C., & Lavazza, A. (2024) · Journal of Cultural Cognitive Science · doi:10.1007/s41809-024-00146-6 · Cited by 27
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2306.04930 · Cited by 22
Negative chemical data boosts language models in reaction outcome prediction
Toniato, A., Vaucher, A. C., Laino, T., & Graziani, M. (2025) · Science Advances · doi:10.1126/sciadv.adt5578 · Cited by 32
Large Language Models in peri-implant disease: How well do they perform?
Koidou, V. P., Chatzopoulos, G. S., Tsalikis, L., & Kaklamanos, E. G. (2025) · Journal of Prosthetic Dentistry · doi:10.1016/j.prosdent.2025.02.008 · Cited by 22
A reliable knowledge processing framework for combustion science using foundation models
Sharma, V., & Raman, V. (2024) · Energy and AI · doi:10.1016/j.egyai.2024.100365 · Cited by 24
Development of AI-assisted microscopy frameworks through realistic simulation with pySTED
Bilodeau, A., Michaud-Gagnon, A., Chabbert, J., Turcotte, B., Heine, J., Durand, A., & Lavoie‐Cardinal, F. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00903-w · Cited by 27
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28878 · Cited by 23
Hung, T. K. W., Kuperman, G. J., Sherman, E. J., Ho, A. L., Weng, C., Pfister, D. G., & Mao, J. J. J. (2024) · Journal of Medical Internet Research · doi:10.2196/60695 · Cited by 24
A survey of communicating robot learning during human-robot interaction
Habibian, S., Valdivia, A. A., Blumenschein, L. H., & Losey, D. P. (2024) · The International Journal of Robotics Research · doi:10.1177/02783649241281369 · Cited by 24
Large language models for bioinformatics
Ruan, W., Lyu, Y., Zhang, J., Cai, J., Shu, P., Yang, G., Lu, Y., Gao, S., Wang, Y., Wang, P., et al. (2025) · Quantitative Biology · doi:10.1002/qub2.70014 · Cited by 37
How to Protect Copyright Data in Optimization of Large Language Models?
Chu, T., Song, Z., & Yang, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29741 · Cited by 41
Natural Language Processing with Commonsense Knowledge: A SurveyPreprint
Xie, Y., Liu, Z., Ma, Z., Meng, F., Xiao, Y., Miao, F., & Pu, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04674 · Cited by 25
In Situ Graph Reasoning and Knowledge Expansion Using Graph‐PRefLexOR
Buehler, M. J. (2025) · Advanced Intelligent Discovery · doi:10.1002/aidi.202500006 · Cited by 31
Artificial intelligence security and privacy: a survey
He, X., Xu, G., Han, X., Wang, Q., Zhao, L., Shen, C., Lin, C., Zhao, Z., Li, Q., Yang, L., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-025-4388-5 · Cited by 24
Aggarwal, T., Salatino, A. A., Osborne, F., & Motta, E. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104262 · Cited by 34
Abbasi, K., Razzaghi, P., Gharizadeh, A., Ghareyazi, A., Dehnad, A., Rabiee, H. R., & Mofrad, M. R. K. (2025) · Pharmacological Reviews · doi:10.1016/j.pharmr.2025.100095 · Cited by 24
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 27
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P. (2026) · IEEE Access · doi:10.1109/access.2026.3675554 · Cited by 28
“Similarly, in [180], the authors introduce SecAlign , which leverages alignment training via direct preference optimization (DPO) [181] to align agents toward preferring benign instructions over adversarial ones.”
Xiong, Y., Zhan, Z., Zhong, C., Zeng, W., Guo, J. X., Tang, W., & Liu, C. (2025) · European Journal Of Dental Education · doi:10.1111/eje.13073 · Cited by 20
Multimodal AI and Large Language Models for Orthopantomography Radiology Report Generation and Q&A
Dasanayaka, C., Dandeniya, K., Dissanayake, M. B., Gunasena, C., & Jayasinghe, R. D. (2025) · Applied System Innovation · doi:10.3390/asi8020039 · Cited by 24
Ma, Y., Ye, W., Cui, C., Zhang, H., Xing, S., Ke, F., Wang, J., Miao, C., Chen, J., Rezatofighi, H., et al. (2025) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw65960.2025.00114 · Cited by 23
Tucano: Advancing neural text generation for Portuguese
Corrêa, N. K., Sen, A., Falk, S., & Fatimah, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101325 · Cited by 36
A Survey on the Integration and Optimization of Large Language Models in Edge Computing Environments
Bhardwaj, S., Singh, P., & Pandit, M. K. (2024) · 2024 16th International Conference on Computer and Automation Engineering (ICCAE) · doi:10.1109/iccae59995.2024.10569285 · Cited by 18
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
Vashurin, R., Fadeeva, E., Vazhentsev, A., Rvanova, L., Vasilev, D., Tsvigun, A., Petrakov, S., Xing, R., Sadallah, A., Grishchenkov, K., et al. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00737 · Cited by 21
Caution Ahead: Numerical Reasoning and Look‐Ahead Bias in AI Models
LEVY, B. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70058 · Cited by 20
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings
Wolfe, R., Slaughter, I., Han, B., Wen, B., Yang, Y., Rosenblatt, L., Herman, B., Brown, E., Qu, Z., Weber, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658966 · Cited by 24
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17
Beyond Text: Frozen Large Language Models in Visual Signal Comprehension
Zhu, L., Wei, F., & Lu, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02554 · Cited by 24
Li, C., & Qi, Y. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108687 · Cited by 26
Medical large language model for diagnostic reasoning across specialties
(2025) · Nature Medicine · doi:10.1038/s41591-025-03520-1 · Cited by 14
Rethinking chemical research in the age of large language models
MacKnight, R., Boiko, D. A., Regio, J. E., Gallegos, L. C., Neukomm, T. A., & Gomes, G. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00811-y · Cited by 26
Teaching Large Language Models to Translate with Comparison
Zeng, J., Meng, F., Yin, Y., & Zhou, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29920 · Cited by 52
PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models
Deng, F., Wang, Q., Wei, W., Hou, T., & Grundmann, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00709 · Cited by 25
Comparing the dental knowledge of large language models
Tussie, C., & Starosta, A. (2024) · BDJ · doi:10.1038/s41415-024-8015-2 · Cited by 14
Nagakalyani, G., Chaudhary, S., Apte, V., Ramakrishnan, G., & Tamilselvam, S. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701913 · Cited by 14
Language models for protein design
Lee, J. S., Abdin, O., & Kim, P. M. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.103027 · Cited by 15
Zhu, C., Hu, X., Wu, H., Qin, C., Zhu, H., & Xiong, H. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104261 · Cited by 18
Kirk, H. R., Bean, A., Vidgen, B., Röttger, P., & Hale, S. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.148 · Cited by 48
Metadata
- Authors
- Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C.
- Affiliations
- Chan Zuckerberg Biohub San Francisco (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.18290
- OpenAlex
- W4378771755
- Cited by
- 293
- References
- 0
- Citations by year
- 2026: 36 · 2025: 149 · 2024: 97 · 2023: 10
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Canadian Institute for Advanced Research
- Office of Naval Research (N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- active inference, reward modeling, language models, trauma, affect labeling, amygdala reactivity, Landauer limit, information thermodynamics, deception, allostasis, free energy principle, distributed cognition, extended mind, interpersonal neural synchronization, criticality, integrated information, mode collapse, reinforcement learning from human feedback, AI alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
