The DOI matches its DataCite record; the references it cites are not checked yet.
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Open accessAbstract
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While RLHF-PPO has shown promising results, it suffers from sensitivity to hyperparameters, the need for training additional models, and potential misalignment of the RM [138,149,174,197,220].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Further, training RLHF models in practice is very difficult for practitioners owing to unstable training [28], hyperparameter sensitivity [149,210], loading multiple models leading to high memory usage [159].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Promisingly, in these domains, algorithmic techniques have emerged to improve the quality of generations compared with sampling from the pretrained model alone (83–85).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similarly, 'Finetune GPT-2 for QA' can be approached with fairly standard DPO techniques [58].”
From RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (2024) · cites this paper as [58]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These generative models have demonstrated unprecedented capabilities to process, understand, and generate human-like text.”
The cited source says what the claim attributes to it.
In the citing paper
“Interestingly, both Llama-3 and Tulu-2 [69] utilized some variant of DPO [149] citing significantly lower computational cost while maintaining similar performance.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The cited source says what the claim attributes to it.
In the citing paper
“Interestingly, both Llama-3 and Tulu-2 [69] utilized some variant of DPO [149] citing significantly lower computational cost while maintaining similar performance.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The cited source says what the claim attributes to it.
In the citing paper
“Moreover, it is highly sensitive to hyperparameters, necessitating a significant amount of hyperparameter tuning [149,210].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The cited source says what the claim attributes to it.
In the citing paper
“Azar et al. [9], Rafailov et al. [149] further reformulate the objective encoded in the RLHF PPO algorithm, and train the model directly on the new objective, without the need for a separate RM.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [149]
The cited source says what the claim attributes to it.
In the citing paper
“These limitations mirror the constraints of natural language models, which have been improved over time with increased scale, labeled data, prompt engineering, and alignment with human preferences (39, 83–85, 87).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, directly sampling from a large natural language model typically produces sequences that are grammatically correct yet locally biased toward simpler sentence constructions and that are globally incoherent, especially at long lengths.”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Natural language models often struggle to maintain coherent and diverse generation over long sequences”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [85]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Recent advancements of artificial intelligence (AI), particularly the rise of generative large language models (LLMs), are rapidly transforming key aspects of health and healthcare, including clinical research informatics (CRI).”
From Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently (2025) · cites this paper as [2]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 32
- H. Bong and A. Rinaldo. Generalized results for the existence and consistency of the MLE in the Bradley-Terry-Luce model. International Conference on Machine Learning, 2022. arXiv:2110.11487. · arXiv
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877- 1901.
- Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_ files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023. arXiv preprint arXiv:2303.12712. · arXiv
- Y. Chen, R. Wang, H. Jiang, S. Shi, and R.-L. Xu. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. ArXiv, abs/2304.00723, 2023. · arXiv
- A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. · arXiv
- P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 30.
- Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/ paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf.
- M. Dudík, K. Hofmann, R. E. Schapire, A. Slivkins, and M. Zoghi. Contextual dueling bandits. In P. Grünwald, E. Hazan, and S. Kale, editors, Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pages 563-587, Paris, France, 03-06 Jul 2015. PMLR. URL https://proceedings.mlr.press/v40/Dudik15.html.
- D. Go, T. Korbak, G. Kruszewski, J. Rozen, N. Ryu, and M. Dymetman. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
- A. Jain, B. Wojcik, T. Joachims, and A. Saxena. Learning trajectory preferences for manip- ulators via iterative improvement. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26.
- Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/ 2013/file/c058f544c737782deacefa532d9add4c-Paper.pdf.
- N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pages 1645-1654. PMLR, 2017.
- N. Jaques, J. H. Shen, A. Ghandeharioun, C. Ferguson, A. Lapedriza, N. Jones, S. S. Gu, and R. Picard. Human-centric dialog training via offline reinforcement learning. arXiv preprint arXiv:2010.05848, 2020. · arXiv
- T. Korbak, H. Elsahar, G. Kruszewski, and M. Dymetman. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 16203-16220.
- Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 67496dfa96afddab795530cc7c69b57a-Paper-Conference.pdf.
- S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018.
- A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/ anthology/P11-1015.
- L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730-27744.
- Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
- R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018. URL https://openreview. net/forum?id=HkAClQgA-.
- X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. · arXiv
- J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
- R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview. net/forum?id=8aHzds2uUyB.
- M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. CoRR, abs/1511.06732, 2015. · arXiv
- A. Saha, A. Pacchiano, and J. Lee. Dueling rl: Reinforcement learning with trajectory preferences. In F. Ruiz, J. Dy, and J.-W. van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceed- ings of Machine Learning Research, pages 6263-6289. PMLR, 25-27 Apr 2023. URL https://proceedings.mlr.press/v206/saha23a.html.
- V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chh- ablani, N. Nayak, D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. A. Fries, R. Teehan, T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush. Multi- task prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4.
- H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. · arXiv
- S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319, 2019. · arXiv
- Y. Wu and B. Hu. Learning to extract coherent summary via deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'18/IAAI'18/EAAI'18. AAAI Press, 2018. ISBN 978-1-57735-800-8.
- X. Yan, C. Luo, C. L. A. Clarke, N. Craswell, E. M. Voorhees, and P. Castells. Human preferences as dueling bandits. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 567-577, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450387323. doi: 10.1145/3477495.3531991. URL https://doi.org/10.1145/3477495.3531991. · DOI
Cited by
Sorted by citations
Items 201–300 of 318
BabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories?
Zhao, X., Wang, T., Osborn, S., & Rios, A. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.16 · Cited by 4
Iterative improvements from feedback for language modelsPreprint
Li, Y. (2023) · doi:10.14293/pr2199.000220.v1 · Cited by 3
Latent Diffusion for Conditional Generation of MoleculesPreprint
Kaufman, B., Williams, E., Pederson, R. D., Underkoffler, C., Panjwani, Z., Wang-Henderson, M., Mardirossian, N., Katcher, M. H., Strater, Z., Grandjean, J.-M., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.08.22.609169 · Cited by 4
Prabha, P. L., & Parvathy, M. (2025) · Journal of the Chinese Institute of Engineers · doi:10.1080/02533839.2025.2466647 · Cited by 2
Symphony: Edge-powered Decentralized Multi-agent Framework for Autonomous Co-evolving IntelligencePreprint
Wang, J., Feng, Y., Ai, L., & Shi, B. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5361139 · Cited by 2
Ou, J., & Holve, E. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf081 · Cited by 1
Conditionally Combining Robot Skills using Large Language Models
Zentner, K. R., Julian, R., Ichter, B., & Sukhatme, G. S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611275 · Cited by 2
Tractable Ridge Regression for Paired Comparisons
Varin, C., & Firth, D. (2024) · Stat · doi:10.1002/sta4.70028 · Cited by 1
Animating the Past: Reconstruct Trilobite via Video Generation
Wu, X., Huang, Z., & Yu, C. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825790 · Cited by 1
Chen, Y.-C., & Huang, H. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i22.34534 · Cited by 1
Seo, J., Park, S., Byun, S., Choi, J., Choi, J. D., & Shin, H. (2025) · Healthcare Informatics Research · doi:10.4258/hir.2025.31.2.166 · Cited by 1
The superalignment of superhuman intelligence with large language models
Huang, M., Wang, Y., Cui, S., Ke, P., & Tang, J. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4348-6 · Cited by 3
Al-Zuraiqi, A., & Greer, D. (2025) · doi:10.1145/3727582.3728687 · Cited by 1
Large Language Model Annotation Bias in Hate Speech Detection
Okpala, E., & Cheng, L. (2025) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v19i1.35879 · Cited by 4
Li, J., Huang, H., Xia, Y., Xu, X., Wu, Y., Guo, Q., Liu, Y., Zhong, Y., Min, J., Son, S., et al. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3583442 · Cited by 1
Temporal consistent multi-view perception for robust embodied manipulation
Chen, H., Yang, R., Zhang, J., Wen, X., Chen, Y., Yu, D., Bai, C., & Wang, Z. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112177 · Cited by 2
Diffusion Preference Alignment via Attenuated Kullback–Leibler Regularization
Zhang, X., & Xiang, W. (2025) · Electronics · doi:10.3390/electronics14152939 · Cited by 2
Yang, Y., Li, C., Han, Z., Li, R., Xu, K., Li, Q., Zhong, W., Shen, Z., Fei, Z., Ge, J., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3764584 · Cited by 7
Recent advances in finetuning multimodal large language models
Wang, Z., Li, L., & Chen, L. (2025) · AI Magazine · doi:10.1002/aaai.70025 · Cited by 3
TimeJudge: empowering video-LLMs as zero-shot judges for temporal consistency in video captions
Hu, Y., Song, Z., Yu, J., Chen, Y. P., & Yang, W. H. (2025) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2500412 · Cited by 1
A Checks-and-Balances Framework for Context-Aware Ethical AI Alignment
Chang, E. Y. (2025) · ACM eBooks · doi:10.1145/3749421.3749434 · Cited by 1
Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testing
Meimandi, K. J., Rust, P. U. N., Eddy, D., Fraser, R., Vasan, N., Djordjevic, D., Dadlani, A., Lamparth, M., Kim, E., & Kochenderfer, M. J. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812332 · Cited by 5
Empowering clinical trial design with agentic intelligence and real-world data
Li, H., Pan, W., Rajendran, S., Zang, C., & Wang, F. (2026) · Nature Communications · doi:10.1038/s41467-026-74501-2 · Cited by 9
The Mechanics of Frictionless Reproducibility
Recht, B. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.f0f013d4 · Cited by 1
Hu, J., Wu, P., Wang, S., Wang, B., & Yang, G. (2024) · Pharmaceutics · doi:10.3390/pharmaceutics16081014 · Cited by 3
Weighted Strategy Optimization Approach for Discrete Sequence Generation
Jiang, X., Li, D., Zhang, H., Zhou, Y., Liu, J., & Xiang, X. (2024) · International Symposium on Artificial Intelligence and Intelligent Manufacturing (AIIM) · doi:10.1109/aiim64537.2024.10934534 · Cited by 1
MAPLE: A Framework for Active Preference Learning Guided by Large Language Models
Mahmud, S., Nakamura, M., & Shlomo Zilberstein, B. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i26.34964 · Cited by 1
Liu, K., Long, Q., Shi, Z., Su, W., & Xiao, J. (2026) · The Annals of Statistics · doi:10.1214/26-aos2643 · Cited by 23
Zhang, Z., Zhang, Y., Chen, G., Qu, L., Zhou, X. S., Wang, H., & Xu, Z. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130572 · Cited by 1
Pasupuleti, M., & Satapathy, S. M. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02539-7 · Cited by 2
BLiMP-NL: A corpus of Dutch minimal pairs and acceptability judgements for language model evaluationPreprint
Suijkerbuijk, M., Prins, Z., Kloots, M. D. H., Zuidema, W. H., & Frank, S. L. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/mhjbx · Cited by 3
Hirano, M., & Imajo, K. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4796245 · Cited by 2
Hu, J., & Wu, P. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-c24c3 · Cited by 1
Two-phase Framework Clinical Question-Answering – Autocorrection for Guideline-concordancePreprint
Tariq, A., Yu, N. Y., Patel, B. N., & Banerjee, I. (2024) · medRxiv · doi:10.1101/2024.11.04.24316718 · Cited by 1
Self-Pluralising Culture Alignment for Large Language ModelsPreprint
Xu, S., Leng, Y., Yu, L., & Xiong, D. (2024) · Qeios · doi:10.32388/q7ywfv · Cited by 1
AlignAb: Pareto-Optimal Energy Alignment for Designing Nature-Like AntibodiesPreprint
Wen, Y., Xu, C., Hu, J. Y.-C., & Liu, H. (2025) · Qeios · doi:10.32388/ovbkgz · Cited by 1
The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility?Preprint
Zhang, Y., Chen, X., Chen, K., Du, Y., Dang, X., & Heng, P. (2025) · Qeios · doi:10.32388/cw5qru · Cited by 2
Personalizing Language Models for Generative TargetingPreprint
Yin, M., Gao, R., & Cong, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5329729 · Cited by 1
Auto-Formulating Dynamic Programming Problems with Large Language ModelsPreprint
Zhou, C., Jing-yuan, Y., Li, X., Chen, Y., He, Z., & Ge, D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5350161 · Cited by 16
Probing Hidden States for Calibrated, Alignment-Resistant Predictions in LLMsPreprint
Berkowitz, J. S., Kivelson, S., Srinivasan, A., Gisladottir, U., Tsang, K. K., Cortina, J. M. A., Kuchi, A., Patock, J. R., Czarny, R., & Tatonetti, N. P. (2025) · medRxiv · doi:10.1101/2025.09.17.25336018 · Cited by 2
How Sparse Attention Approximates Exact Attention? Your Attention is Naturally $n^C$-Sparse
Deng, Y., Song, Z., Xiong, J., & Yang, C. (2024) · arXiv · doi:10.48550/arxiv.2404.02690 · Cited by 1
PAACE: A Plan-Aware Automated Agent Context Engineering Framework
Yuksel, K. A. (2025) · arXiv · doi:10.48550/arxiv.2512.16970
Paech, S., Roush, A., Goldfeder, J., & Shwartz-Ziv, R. (2025) · arXiv · doi:10.48550/arxiv.2510.15061
A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation
Chen, J., Chen, W., Lin, Z., & Vong, C. M. (2026) · arXiv · doi:10.48550/arxiv.2608.24419
Uhler, L., Jordan, V., Buder, J., Huff, M., & Papenmeier, F. (2026) · Communications Psychology · doi:10.1038/s44271-026-00403-0
Discit Ergo Est: Training Data Provenance and Fair UsePreprint
Mahari, R., & Longpre, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4795277
Feedback-aligned Mixed LLMs for Machine Language-Molecule TranslationPreprint
Gkoumas, D., & Liakata, M. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-p39s0
A Chit-Chat between Llama2 and Chatgpt for the Automated Creation of ExploitsPreprint
Romano, S. P., Caturano, F., & Ciotola, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4886571
From Large Language Models to Generative AI Systems
Tomczak, J. M. (2024) · Springer eBooks · doi:10.1007/978-3-031-64087-2_11
Are Large Language Models Immune to Representativeness Bias? Evidence and ConjecturesPreprint
Li, S., & Rui, H. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4976447
Detoxifying Large Language Models via Kahneman-Tversky Optimization
Li, Q., Du, W., & Liu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9443-0_36 · Cited by 1
Development of a Large-scale Korean Language Model in the Field of Geosciences
Lee, S. (2024) · Economic and Environmental Geology · doi:10.9719/eeg.2024.57.5.539
Modeling the Structural and Semantic Features for Japanese Lyrics Generation of J-Pop Songs
Han, Y., & Wang, H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0119-6_31
Ishida, T. (2024) · Denshi Jouhou Tsuushin Gakkai Tsuushin Sosaieti magajin/B-plus/Denshi Jouhou Tsuushin Gakkai Tsuushin Sosaeti magajin · doi:10.1587/bplus.18.232
Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model AlignmentPreprint
Cai, Y., Yuan, Y., Shi, J., & Lin, Q. (2024) · Qeios · doi:10.32388/slo0cb
Second language learning of degree expressions: A computational approach
Cong, Y. (2024) · Natural language processing. · doi:10.1017/nlp.2024.56
Filho, M. P. F., de Almeida, J. D. S., & de Paiva, A. C. (2024) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2024.245458
Zuo, L., Zhang, D., Zhao, Y., & Wang, G. (2025) · Electronics · doi:10.3390/electronics14020294
Are Vision-Language Models Truly Understanding Multi-vision Sensor?Preprint
Chung, S., Yu, Y., Chee, Y., Kim, S. Y., Lee, B., & Ro, Y. M. (2025) · Qeios · doi:10.32388/vn42c2
Enhancing Code LLMs with Reinforcement Learning in Code Generation: A SurveyPreprint
He, Y., Shi, T., Li, Y., & Chen, Q. (2025) · Qeios · doi:10.32388/8g8tb2
Filtered Direct Preference Optimization
Morimura, T. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.355
Ma, L., Ding, Y.-X., Zhang, Z., & Zhou, Z. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709279
Debate Helps Weak-to-Strong Generalization
Lang, H., Huang, F., & Li, Y. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i26.34952 · Cited by 13
Behaviour Preference Regression for Offline Reinforcement Learning
Srinivasan, P., & Knottenbelt, W. J. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i19.34267
Imran, A., & Gopalakrishnan, K. (2025) · Apress eBooks · doi:10.1007/979-8-8688-0989-7_4
CoreViz: Context-Aware Reasoning and Visualization Engine for Business Intelligence DashboardsPreprint
Xie, M., & Chen, S. (2025) · Research Square · doi:10.21203/rs.3.rs-6675880/v1
CoreViz: Context-Aware Reasoning and Visualization Engine for Business Intelligence DashboardsPreprint
Xie, M., & Chen, S. (2025) · doi:10.36227/techrxiv.174803320.08603240/v1
Efficient Algorithms for Leveraging LLMs for Generative and Predictive Recommender Systems
Behdin, K., Dai, Y., Dexter, G., Gupta, A., Mazumder, R., Saha, A., Song, Q., Tang, S., Zhu, S., & Hsu, P.-L. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715868
Large Language Model Training and Reinforcement LearningPreprint
Alonso, M. N. I. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5194064
TestFlow: Advancing Mobile UI Testing through Multi-Step Reinforcement Learning
Tang, X., Chen, X., Chen, D., Zhou, S., Jiang, W., & Li, Y. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3732930
Patch the Leak: Strengthening CodeLLMs Against Privacy Extraction Threats
Guo, Y., Ma, W., Xiao, X., Wen, S., Di, P., & Zhu, X. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3732931
Responsible Prompt Engineering: An Embedding Based Approach to Secure LLM Interactions
Sharma, P. S. K. (2025) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2025.72157
A deep reinforcement learning framework for optimized dummy pad placement in PCB electroplating
Zhou, Q., Li, Y., Wang, C., & Wang, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128639
Zhang, B., Wang, S., Jiang, Y., Sui, D., Tu, Z., & Chu, D. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729898
LLM powered Spatial Enrichment of Message Sequence Charts and its Applications
Ramrakhiyani, N., Pawar, S. S., Palshikar, G. K., & Varma, V. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129103
Enhancing Clinical Reasoning in Medical Vision-Language Model through Structured PromptsPreprint
Prakash, K. D., Kim, K., & Han, Y. (2025) · medRxiv · doi:10.1101/2025.07.31.25331901
AI Surrogacy in Psychological ResearchPreprint
D’Alessandro, W., & Thompson, J. A. F. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/6emg9_v1
Morris, R. J. T. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-28969-9.00008-x
"Give a Positive Review Only": Can Hidden PDF Prompts Sway LLMs?Preprint
Guo, X., Huang, Z., Song, Z., & Zhang, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5368453
LLMs Can Overcome Lack of Embodied Context Through Explicit Prompting
Lakkananithiphan, P., Sritriratanarak, W., & Garcia, P. (2025) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1109/iccai66501.2025.00061
Shankarnarayanan, A., Syed, T., Shapsough, S., Zualkernan, I. A., & Kochmar, E. (2025) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1109/iccai66501.2025.00057
Data to Dose: Efficient Synthetic Data Generation with Expert Guidance for Personalized DosingPreprint
Verma, H. S., Wiberg, H., Zhu, W., & TAYUR, S. R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5400895
Agile3D: Adaptive Contention- and Content-Aware 3D Object Detection for Embedded GPUs
Wang, P., Liu, Z., Bagchi, S., Xu, R., Bagchi, S., Li, Y., & Chaterji, S. (2025) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3711875.3729147 · Cited by 1
Large language models – the future of fundamental physics?
Heneka, C. S., Nieser, F., Ore, A., Plehn, T., & Schiller, D. (2026) · SciPost Physics · doi:10.21468/scipostphys.20.3.070 · Cited by 13
AI as a Financial Coach—Promoting Sustainable Financial Management with Generative AI
Salo-Lahti, M., Ranta, M., Toivonen, N., & Haapio, H. (2025) · Perspectives in law, business and innovation · doi:10.1007/978-981-95-2058-9_4
Balancing Engagement and Polarization: Multi-Objective Alignment of News Content Using LLMsPreprint
Cheng, M., Ofek, E., & Yoganarasimhan, H. (2025) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2504.13444 · Cited by 6
Wang, C., Meng, F., Pei, J., Xia, L., Liu, J., Yuan, X., & Di, X. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105881
The Future of Higher Education in the Era of Generative AI
Williams, J. S. (2026) · Springer eBooks · doi:10.1007/978-981-95-1875-3_15
Zuo, J., Wang, Y., & Yibulayin, X. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1732609
Structure-derived synthetic sequences guide a protein language model toward metalloproteinsPreprint
Peteani, G., Sgueglia, G., Lemmin, T., & Chino, M. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.04.30.722007
Automated Generation of Synthetic Medical Dialogues Using Large Language Models
Bahrololloomi, F., Luderschmidt, J., & Fu, B. (2026) · ACM Southeast Regional Conference (ACMSE) · doi:10.1145/3746467.3801512
Hybrid Supervised Fine-Tuning Method for Medical Language Models via Explicit Reasoning Modeling
Wang, X., Wei, T., Nan, Z., & Wan, S. (2026) · Journal of Shanghai Jiaotong University (Science) · doi:10.1007/s12204-026-2929-6
Zaiats, V. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20496248
TRIAD-T2 Clean Shell 5.3: Sovereign Immunoceptive Architecture for Silicon IntelligencePreprint
Zaiats, V. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20534506
Zaiats, V. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20534949
Evaluating and Enhancing an AI Mental Health Guide in High-Risk Mental Health ContextsPreprint
Swirbul, M. S., Peake, E. E., McDonald, P., Rodio, M. J., Padmanabhan, A., & Ben‐Zion, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-9588991/v1
Evaluation of DPO Configurations for LLM Alignment
Majkutewicz, J., & Szymański, J. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-0068-9_10
Rozado, D., & Tetlock, P. E. (2026) · Theory and Society · doi:10.1007/s11186-026-09719-6
Measuring Creativity at Scale via Multimodal Large Language Models
Lewis, A., & Ochôa, X. (2026) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3774398.3811607
Metadata
- Authors
- Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C.
- Affiliations
- Chan Zuckerberg Biohub San Francisco (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.18290
- OpenAlex
- W4378771755
- Cited by
- 293
- References
- 0
- Citations by year
- 2026: 36 · 2025: 149 · 2024: 97 · 2023: 10
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Canadian Institute for Advanced Research
- Office of Naval Research (N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- active inference, reward modeling, language models, trauma, affect labeling, amygdala reactivity, Landauer limit, information thermodynamics, deception, allostasis, free energy principle, distributed cognition, extended mind, interpersonal neural synchronization, criticality, integrated information, mode collapse, reinforcement learning from human feedback, AI alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
