The DOI matches its DataCite record; the references it cites are not checked yet.
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Open accessAbstract
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While RLHF-PPO has shown promising results, it suffers from sensitivity to hyperparameters, the need for training additional models, and potential misalignment of the RM [138,149,174,197,220].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Further, training RLHF models in practice is very difficult for practitioners owing to unstable training [28], hyperparameter sensitivity [149,210], loading multiple models leading to high memory usage [159].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Promisingly, in these domains, algorithmic techniques have emerged to improve the quality of generations compared with sampling from the pretrained model alone (83–85).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similarly, 'Finetune GPT-2 for QA' can be approached with fairly standard DPO techniques [58].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [58]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These generative models have demonstrated unprecedented capabilities to process, understand, and generate human-like text.”
The cited source says what the claim attributes to it.
In the citing paper
“Interestingly, both Llama-3 and Tulu-2 [69] utilized some variant of DPO [149] citing significantly lower computational cost while maintaining similar performance.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The cited source says what the claim attributes to it.
In the citing paper
“Interestingly, both Llama-3 and Tulu-2 [69] utilized some variant of DPO [149] citing significantly lower computational cost while maintaining similar performance.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The cited source says what the claim attributes to it.
In the citing paper
“Moreover, it is highly sensitive to hyperparameters, necessitating a significant amount of hyperparameter tuning [149,210].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The cited source says what the claim attributes to it.
In the citing paper
“Azar et al. [9], Rafailov et al. [149] further reformulate the objective encoded in the RLHF PPO algorithm, and train the model directly on the new objective, without the need for a separate RM.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The cited source says what the claim attributes to it.
In the citing paper
“These limitations mirror the constraints of natural language models, which have been improved over time with increased scale, labeled data, prompt engineering, and alignment with human preferences (39, 83–85, 87).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, directly sampling from a large natural language model typically produces sequences that are grammatically correct yet locally biased toward simpler sentence constructions and that are globally incoherent, especially at long lengths.”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Natural language models often struggle to maintain coherent and diverse generation over long sequences”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Recent advancements of artificial intelligence (AI), particularly the rise of generative large language models (LLMs), are rapidly transforming key aspects of health and healthcare, including clinical research informatics (CRI).”
From Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently (2025) · cites this paper as [2]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 32
- H. Bong and A. Rinaldo. Generalized results for the existence and consistency of the MLE in the Bradley-Terry-Luce model. International Conference on Machine Learning, 2022. arXiv:2110.11487. · arXiv
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877- 1901.
- Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_ files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023. arXiv preprint arXiv:2303.12712. · arXiv
- Y. Chen, R. Wang, H. Jiang, S. Shi, and R.-L. Xu. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. ArXiv, abs/2304.00723, 2023. · arXiv
- A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. · arXiv
- P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 30.
- Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/ paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf.
- M. Dudík, K. Hofmann, R. E. Schapire, A. Slivkins, and M. Zoghi. Contextual dueling bandits. In P. Grünwald, E. Hazan, and S. Kale, editors, Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pages 563-587, Paris, France, 03-06 Jul 2015. PMLR. URL https://proceedings.mlr.press/v40/Dudik15.html.
- D. Go, T. Korbak, G. Kruszewski, J. Rozen, N. Ryu, and M. Dymetman. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
- A. Jain, B. Wojcik, T. Joachims, and A. Saxena. Learning trajectory preferences for manip- ulators via iterative improvement. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26.
- Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/ 2013/file/c058f544c737782deacefa532d9add4c-Paper.pdf.
- N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pages 1645-1654. PMLR, 2017.
- N. Jaques, J. H. Shen, A. Ghandeharioun, C. Ferguson, A. Lapedriza, N. Jones, S. S. Gu, and R. Picard. Human-centric dialog training via offline reinforcement learning. arXiv preprint arXiv:2010.05848, 2020. · arXiv
- T. Korbak, H. Elsahar, G. Kruszewski, and M. Dymetman. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 16203-16220.
- Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 67496dfa96afddab795530cc7c69b57a-Paper-Conference.pdf.
- S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018.
- A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/ anthology/P11-1015.
- L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730-27744.
- Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
- R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018. URL https://openreview. net/forum?id=HkAClQgA-.
- X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. · arXiv
- J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
- R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview. net/forum?id=8aHzds2uUyB.
- M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. CoRR, abs/1511.06732, 2015. · arXiv
- A. Saha, A. Pacchiano, and J. Lee. Dueling rl: Reinforcement learning with trajectory preferences. In F. Ruiz, J. Dy, and J.-W. van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceed- ings of Machine Learning Research, pages 6263-6289. PMLR, 25-27 Apr 2023. URL https://proceedings.mlr.press/v206/saha23a.html.
- V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chh- ablani, N. Nayak, D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. A. Fries, R. Teehan, T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush. Multi- task prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4.
- H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. · arXiv
- S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319, 2019. · arXiv
- Y. Wu and B. Hu. Learning to extract coherent summary via deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'18/IAAI'18/EAAI'18. AAAI Press, 2018. ISBN 978-1-57735-800-8.
- X. Yan, C. Luo, C. L. A. Clarke, N. Craswell, E. M. Voorhees, and P. Castells. Human preferences as dueling bandits. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 567-577, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450387323. doi: 10.1145/3477495.3531991. URL https://doi.org/10.1145/3477495.3531991. · DOI
Cited by
Sorted by citations
Items 101–200 of 318
Are Large Language Models Consistent over Value-laden Questions?
Moore, J., Deshpande, T., & Yang, D. (2024) · arXiv · doi:10.48550/arxiv.2407.02996 · Cited by 12
Hu, Y., Dong, T., & Li, S. (2025) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2025.105051 · Cited by 15
Web Archives Metadata Generation with GPT-4o
Nair, A., Goh, Z. R., Liu, T., & Huang, A. (2025) · Information Technology and Libraries · doi:10.5860/ital.v44i2.17305 · Cited by 14
Decoding the Molecular Language of Proteins with EvollaPreprint
Zhou, X., Han, C., Zhang, Y., Du, H., Tian, J., Su, J., Liu, R., Zhuang, K., Jiang, S., Gitter, A., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.01.05.630192 · Cited by 16
SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF
Dong, Y., Wang, Z., Sreedhar, M., Wu, X., & Kuchaiev, O. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.754 · Cited by 15
Generalizable clinical note section identification with large language models
Zhou, W., & Miller, T. A. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae075 · Cited by 11
Large Language Model in Materials Science: Roles, Challenges, and Strategic Outlook
Zhang, J., Chen, X., Ye, X., Yang, Y., & Ai, B. (2025) · Advanced Intelligent Discovery · doi:10.1002/aidi.202500085 · Cited by 17
Functional alignment of protein language models via reinforcement learning
Blalock, N., Seshadri, S., Nakamura, K., Babbar, A., Fahlberg, S. A., Kulkarni, A., & Romero, P. A. (2026) · Nature Communications · doi:10.1038/s41467-026-77557-2 · Cited by 19
Mental Healthcare Chatbot Based on Custom Diagnosis Documents Using a Quantized Large Language Model
Kumar, A., Sharma, S., Gupta, S., & Kumar, D. (2024) · International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) · doi:10.1109/icrito61523.2024.10522371 · Cited by 10
Large Language Models meet moral values: A comprehensive assessment of moral abilities
Bulla, L., Giorgis, S. D., Mongiovì, M., & Aldo Gangemi, A. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100609 · Cited by 20
Yuan, L., Wang, J., Sun, H., Zhang, Y., & Lin, Y. (2025) · Qeios · doi:10.32388/x26ilu · Cited by 15
Integration of LLMs and the Physical World: Research and Application
Luo, X., Liu, D., Dang, F., & Luo, H. (2024) · ACM Turing Celebration Conference - China (ACM TURC) · doi:10.1145/3674399.3674402 · Cited by 10
Open-Ethical AI: Advancements in Open-Source Human-Centric Neural Language Models
Sicari, S., Moreno, J. F. C., Rizzardi, A., & Coen‐Porisini, A. (2024) · ACM Computing Surveys · doi:10.1145/3703454 · Cited by 13
LettinGo: Explore User Profile Generation for Recommendation System
Wang, L., Zhang, D., Yang, F., Zhao, P., Liu, J., Zhan, Y., Sun, H., Lin, Q., Deng, W., Zhang, D., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737024 · Cited by 10
Personalized Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation
Chen, H., Saha, A., Hoi, S. C. H., & Joty, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.417 · Cited by 18
Typhoon: Thai Large Language Models
Pipatanakul, K., Jirabovonvisut, P., Manakul, P., Sripaisarnmongkol, S., Patomwong, R., Chokchainant, P., & Tharnpipitchai, K. (2023) · arXiv · doi:10.48550/arxiv.2312.13951 · Cited by 9
Low-cost language models: Survey and performance evaluation on Python code generation
Espejel, J. L., Alassan, M. S. Y., Bouhandi, M., Dahhane, W., & Ettifouri, E. H. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109490 · Cited by 15
FlowPacker: protein side-chain packing with torsional flow matching
Lee, J. S., & Kim, P. M. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf010 · Cited by 8
LLM-Sentry: A Model-Agnostic Human-in-the-Loop Framework for Securing Large Language Models
Irtiza, S., Akbar, K. A., Yasmeen, A., Khan, L. R., Daescu, O., & Thuraisingham, B. M. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00036 · Cited by 10
Oğul, Í. Ü., Soygazi, F., & Bostanoğlu, B. E. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2662 · Cited by 9
“Desired behaviors”: alignment and the emergence of a machine learning ethics
Schwerzmann, K., & Campolo, A. (2025) · AI & Society · doi:10.1007/s00146-025-02272-3 · Cited by 13
Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
Winata, G. I., Zhao, H., Das, A., Tang, W., Yao, D. D. W., Zhang, S.-X., & Sahu, S. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.17541 · Cited by 30
Training LLMs for Generating IEC 61131-3 Structured Text with Online Feedback
HAAG, A. L., Fuchs, B., Kacan, A., & Lohse, O. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00013 · Cited by 12
RALLRec+: Retrieval augmented large language model recommendation with reasoning
Luo, S., Xu, J., Zhang, X., Wang, L., Liu, S., Hou, H., & Song, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129508 · Cited by 13
Parameter-efficient action planning with large language models for vision-and-language navigation
Mohammadi, B., Abbasnejad, E., Qi, Y., Wu, Q. M., van den Hengel, A., & Shi, Q. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112462 · Cited by 11
Transforming artificial intelligence into artificial wisdom
Jeste, D. V., Paulus, M. P., Alexopoulos, G. S., Barnard, M., Otte, W. M., Satake, Y., Na, P. J., Torous, J. B., Prodan, A., & Occhipinti, J. (2026) · Nature Mental Health · doi:10.1038/s44220-026-00640-6 · Cited by 9
Giuffrè, M., You, K., Pang, Z., Kresevic, S., Chung, S., Chen, R., Ko, Y., Chan, C., Saarinen, T., Ajčević, M., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01589-z · Cited by 10
InstructPLM: Aligning Protein Language Models to Follow Protein Structure InstructionsPreprint
Qiu, J., Xu, J., Hu, J., Cao, H., Hou, L., Gao, Z.-J., Zhou, X., Li, A., Li, X., Cui, B., et al. (2024) · Research Square · doi:10.1101/2024.04.17.589642 · Cited by 10
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
Moon, S., Madotto, A., Lin, Z., Nagarajan, T., Smith, M., Jain, S., Yeh, C.-F., Murugesan, P., Heidari, P., Liu, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16058 · Cited by 7
Aligning large language models with human preferences using historical text edits
Majkutewicz, J., & Szymański, J. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113566 · Cited by 9
Improving Summarization with Human Edits
Yao, Z., Schloss, B., & Selvaraj, S. P. P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.158 · Cited by 15
Gitzel, R., Hoffmann, M. W., Heiden, P. Z., Skolik, A., Kaltenpoth, S., Müller, O., Kanak, C., Kandiah, K., Stroh, M.-F., Boos, W., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3437195 · Cited by 12
Hirano, M., & Imajo, K. (2024) · IIAI International Congress on Advanced Applied Informatics (IIAI-AAI) · doi:10.1109/iiai-aai63651.2024.00059 · Cited by 8
Emptying the Ocean with a Spoon: Should We Edit Models?
Pinter, Y., & Elhadad, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1012 · Cited by 17
Learning to generate and evaluate fact-checking explanations with transformers
Feher, D., Khered, A., Zhang, H., Batista-Navarro, R., & Schlegel, V. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109492 · Cited by 11
Hu, J., Wu, P., Li, Y., Li, Q., Wang, S., Liu, Y., Qian, K., & Yang, G. (2024) · Pharmaceuticals · doi:10.3390/ph17101300 · Cited by 6
Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
Bernardelle, P., & Demartini, G. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698411 · Cited by 7
Real-Time Scheduling with Independent Evaluators: Explainable Multi-Agent Approach
Isakov, A. O., Peregorodiev, D., Tomilov, I., Ye, C., Gusarova, N., Vatian, A. S., & Boukhanovsky, A. V. (2024) · Technologies · doi:10.3390/technologies12120259 · Cited by 7
D’Oosterlinck, K., Xu, W., Develder, C., Demeester, T., Singh, A., Potts, C., Kiela, D., & Mehri, S. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00748 · Cited by 31
Reinforcement Learning and Control
Imran, A., & Gopalakrishnan, K. (2025) · Apress eBooks · doi:10.1007/979-8-8688-0989-7_7 · Cited by 6
Causal Alignment: Augmenting Language Models with A/B TestsPreprint
Angelopoulos, P., Lee, K., & Misra, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4781850 · Cited by 7
Likelihood-based Fine-tuning of Protein Language Models for Few-shot Fitness Prediction and DesignPreprint
Hawkins‐Hooker, A., Surana, S., Simons, J., Kmec, J., Bent, O., & Duckworth, P. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.05.28.596156 · Cited by 10
Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs
Yang, Z., Zhang, Y., Wang, Y., Xu, Z., Lin, J., & Sui, Z. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.203 · Cited by 22
Mitigating Sycophancy in Large Language Models via Direct Preference Optimization
Khan, A. A., Alam, S., Wang, X., Khan, A. F., Neog, D. R., & Anwar, A. (2024) · 2024 IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825538 · Cited by 6
Bronlet, X. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1488102 · Cited by 5
What is AI safety? What do we want it to be?
Harding, J., & Kirk‐Giannini, C. D. (2025) · Philosophical Studies · doi:10.1007/s11098-025-02367-z · Cited by 21
Act-LLM: A whole-process chain for character-centric role-playing with large language models
Han, X., Zhao, W., Guan, Z., & Peng, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129024 · Cited by 5
Njuguna, G. N., & Qingfei, M. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.103080 · Cited by 7
Asymptotics of Language Model Alignment
Yang, J. Q., Salamatian, S., Sun, Z., Suresh, A. T., & Beirami, A. (2024) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit57864.2024.10619456 · Cited by 28
On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlook
Fan, M., Wang, C., Chen, C., Liu, Y., & Huang, J. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02375-w · Cited by 40
WebGLM: Towards an Efficient and Reliable Web-Enhanced Question-Answering System
Lai, H., Liu, X., Yu, H., Xu, Y., Iong, I. L., Yao, S., Zeng, A., Du, Z., Dong, Y., & Tang, J. (2025) · ACM Transactions on Information Systems · doi:10.1145/3729421 · Cited by 7
MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre-TrainingPreprint
Chen, B., Bei, Z., Cheng, X., Li, P., Tang, J., & Song, L. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.10.598380 · Cited by 11
A Dutch Financial Large Language Model
Noels, S., Blaere, J. D., & Bie, T. D. (2024) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3677052.3698628 · Cited by 7
BegoniaGPT: Cultivating the large language model to be an exceptional K-12 English teacher
Rong, L., Zhang, Y., Tiwari, P., & Mei, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107488 · Cited by 4
LR-SQL: A Supervised Fine-Tuning Method for Text2SQL Tasks Under Low-Resource Scenarios
Wen, W., Zhang, Y., Pan, S., Sun, Y., Lu, P., & Ding, C. (2025) · Electronics · doi:10.3390/electronics14173489 · Cited by 8
Exploring Text-to-Motion Generation with Human Preference
Sheng, J., Lin, M. G., Zhao, A., Pruvost, K., Wen, Y., Li, Y., Huang, G. M., & Liu, Y. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00195 · Cited by 8
Jailbreaking large language models: models, origins, and evolution of attacks and defenses
Li, X. H., Wu, J., Zheng, Q., Wang, H., Fan, M. Y., Hu, S., Guo, J., & Ting, L. I. U. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0196 · Cited by 4
Custom training data: Aligning model distribution via Prefix-Guided Preference Data Synthesis
Chen, Z., Qiao, F., Chen, H., Zhang, W., Ren, P., Lin, S., Sun, J., & Li, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104337 · Cited by 4
Detection of Fake News in Romanian: LLM-Based Approaches to COVID-19 Misinformation
Dima, A. M., Ilis, E., Florea, D., & Dascălu, M. (2025) · Information · doi:10.3390/info16090796 · Cited by 5
Sun, Z., Zhou, Y., Hao, J., Fan, X., Lu, Y., Ma, C., Shen, W., & Guo, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.41 · Cited by 4
Refining Large Language Models for Tabular Data Analysis in Business Domain by Laymen Text
Nirusanan, T., Prasanth, S., Kuhaneswaran, B., & Kumara, S. (2024) · International Research Conference on Smart Computing and Systems Engineering (SCSE) · doi:10.1109/scse61872.2024.10550809 · Cited by 4
ClaimBrush: A Novel Framework for Automated Patent Claim Refinement Based on Large Language Models
Kawano, S., Nonaka, H., & Yoshino, K. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825172 · Cited by 4
ExSPIN: Explicit Feedback-Based Self-Play Fine-Tuning for Text-to-SQL Parsing
Yan, L., Su, J., Liu, C., Duan, S., Zhang, Y., Li, J., Han, P., & Liu, Y. (2025) · Entropy · doi:10.3390/e27030235 · Cited by 4
Safeguarding Decentralized Social Media: LLM Agents for Automating Community Rule Compliance
Cava, L. L., & Tagarelli, A. (2025) · Online Social Networks and Media · doi:10.1016/j.osnem.2025.100319 · Cited by 13
A chit-chat between Llama 2 and ChatGPT for the automated creation of exploits
Caturano, F., Ciotola, J., Romano, S. P., & Varlese, M. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111501 · Cited by 4
Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational CompliancePreprint
Imperial, J. M., Jones, M. D., & Madabushi, H. T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5146161 · Cited by 4
EmulatRx: Empowering Clinical Trial Design with Agentic Intelligence and Real World DataPreprint
Li, H., Pan, W., Rajendran, S., Zang, C., & Wang, F. (2025) · medRxiv · doi:10.1101/2025.04.17.25326033 · Cited by 4
Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval
Mekonnen, K. A., Tang, Y., & de Rijke, M. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3730023 · Cited by 3
“Finally, direct learning-to-rank (L2R) optimization is applied, using relevance feedback to refine the model’s ranking quality and align its outputs with document-level relevance (Section 4.3).”
InfluentialNear to Mid-term Risks and Opportunities of Open-Source Generative AI
Eiras, F., Petrov, A., Vidgen, B., de Witt, C. S., Pizzati, F., Elkins, K., Mukhopadhyay, S., Bibi, A., Csaba, B., Steibel, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.17047 · Cited by 2
DeepEnhancerPPO: An Interpretable Deep Learning Approach for Enhancer Classification
Mu, X., Chen, Q., Shi, B., Xu, L., Huang, Z., Xu, Y., & Zhang, K. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms252312942 · Cited by 3
Harnessing LLMs to Build an Autonomous Marketing Agent
Deshmukh, N., Venkatesh, A. S., Mathew, A., Madugula, M., Merchant, P. M., Lanham, M. A., & Shirodkar, S. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-85856-7_21 · Cited by 4
Exploring Large Language Models for Analyzing Open Source License Conflicts: How Far Are We?
Cui, X., Wu, J., Ling, X. G., Luo, T., Yang, M., & Ou, W. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00083 · Cited by 6
Wang, W., Chen, X., Miao, D., Zhang, H., Qin, X., Gu, X., & Lu, P. (2025) · Engineering Analysis with Boundary Elements · doi:10.1016/j.enganabound.2025.106150 · Cited by 5
Ghimire, B., Mahmood, A., & Elleithy, K. M. (2025) · AI · doi:10.3390/ai6040066 · Cited by 3
Putting People in LLMs’ Shoes: Generating Better Answers via Question Rewriter
Chen, J., Wang, B., Jiang, Z., & Nakashima, Y. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i22.34527 · Cited by 3
Diverse AI Feedback For Large Language Model Alignment
Yu, T., Lin, T.-E., Wu, Y., Yang, M., Huang, F., & Li, Y. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00746 · Cited by 12
EMTeC: A corpus of eye movements on machine-generated texts
Bolliger, L. S., Haller, P., Cretton, I. C. R., Reich, D. R., Kew, T., & Jäger, L. A. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02677-4 · Cited by 8
Rynkiewicz, A. A., Palma, R., & Formanowicz, P. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112185 · Cited by 5
Impacts of Artificial Intelligence Development on Humanity and Social Values
鍾, 俊., Tan, Y. K., & Zhou, X. (2025) · Information · doi:10.3390/info16090810 · Cited by 3
GELEX: Generative AI-Hybrid System for Example-Based Learning
Yazici, A., Meija-Domenzain, P., Frej, J., & Käser, T. (2024) · CHI Extended Abstracts · doi:10.1145/3613905.3650900 · Cited by 5
Kuzi, S., & Malmasi, S. (2024) · ACM SIGIR Forum · doi:10.1145/3687273.3687293 · Cited by 4
Xu, T., Piao, Z. H., Mukai, T., Murayama, Y., & Izumi, K. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200496 · Cited by 4
FinRL Contests: Data‐Driven Financial Reinforcement Learning Agents for Stock and Crypto Trading
Wang, K., Holzer, N., Xia, Z., Cao, Y., Gao, J., Walid, A., Xiao, K., & Yanglet, X.-Y. L. (2025) · Artificial Intelligence for Engineering · doi:10.1049/aie2.12004 · Cited by 10
Foundation Models for Low-Resource Language Education (Vision Paper)Preprint
Ding, Z., Liu, Z., Jiang, H., Gao, Y., Zhaı, X., Liu, T., & Liu, N. (2025) · Qeios · doi:10.32388/iqu339 · Cited by 9
PerTurboAgent: A Self-Planning Agent for Boosting Sequential Perturb-seq ExperimentsPreprint
Hao, M., Lee, Y., Wang, H., Scalia, G., & Regev, A. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.05.25.656020 · Cited by 3
RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts
Wijk, H., Lin, T., Becker, J., Jawhar, S., Parikh, N., Broadley, T., Chan, L., Chen, M., Clymer, J., Dhyani, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15114 · Cited by 187
AutoRed: Automated Attack Scenario Generation Framework for Red Teaming of LLMs
Wang, Z., & Tayebi, M. A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825267 · Cited by 3
Shen, J., Su, N.-J., & Lin, Y. (2025) · Systems · doi:10.3390/systems13070523 · Cited by 2
Safe and effective post-fine-tuning alignment in large language models
Jiang, M., Yang, Y., Xie, X., Ke, P., & Liu, G. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114523 · Cited by 3
Ahamed, S. A., Chowdhury, M. H., Zaman, M., Agarwala, A., Mashud, M. A. A., Tariqul, Muntasir, T. A., Rahman, R., Shahriyar, R., Sarker, F., et al. (2025) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2025.101720 · Cited by 3
Evaluation of Medium-Sized Language Models in German and English Language
Peinl, R., & Wirth, J. (2024) · International Journal on Natural Language Computing · doi:10.5121/ijnlc.2024.13101 · Cited by 2
Research on Alignment and Evaluation Methods for Large Language Models
Li, Y., Yan, Q., Ma, N., & Wan, F. (2024) · Guangdong-Hong Kong-Macao Greater Bay Area International Conference on Digital Economy and Artificial Intelligence (DEAI) · doi:10.1145/3675417.3675536 · Cited by 3
Bayesian Constraint Inference from User Demonstrations Based on Margin-Respecting Preference Models
Papadimitriou, D., & Brown, D. S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611095 · Cited by 4
Ho, A. K. N., Anh-Khoi, N.-H., & Khuong-Duy, V. (2024) · International Conference on Multimedia Analysis and Pattern Recognition (MAPR) · doi:10.1109/mapr63514.2024.10660907 · Cited by 2
Park, S.-G., Kim, H.-B., Lee, Y. J., Ahn, W. J., & Lim, M. T. (2025) · International Journal of Control Automation and Systems · doi:10.1007/s12555-024-0528-6 · Cited by 2
VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning
Lee, J. S., Kim, J., Na, J., Park, J., & Kim, H. J. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i4.32474 · Cited by 6
Harris, S., Liu, J., Hadi, H. J., Ahmad, N., & Alshara, M. A. (2025) · Scientific Reports · doi:10.1038/s41598-025-98271-x · Cited by 3
Fostering collective intelligence in CPSS: an LLM-driven multi-agent cooperative tuning framework
Chen, R., & He, C. (2025) · Frontiers in Physics · doi:10.3389/fphy.2025.1613499 · Cited by 4
Yi, X., Li, Y., Shi, D., Wang, L., Wang, X., & He, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129101 · Cited by 8
Li, Y., & Sun, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104270 · Cited by 3
Metadata
- Authors
- Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C.
- Affiliations
- Chan Zuckerberg Biohub San Francisco (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.18290
- OpenAlex
- W4378771755
- Cited by
- 293
- References
- 0
- Citations by year
- 2026: 36 · 2025: 149 · 2024: 97 · 2023: 10
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Canadian Institute for Advanced Research
- Office of Naval Research (N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- active inference, reward modeling, language models, trauma, affect labeling, amygdala reactivity, Landauer limit, information thermodynamics, deception, allostasis, free energy principle, distributed cognition, extended mind, interpersonal neural synchronization, criticality, integrated information, mode collapse, reinforcement learning from human feedback, AI alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
