Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Capability internalization is achieved by learning from reasoning traces, through supervised fine-tuning or preference-guided optimization methods such as DPO and GRPO.
“Technically, capability internalization is achieved by learning from reasoning traces, through supervised fine-tuning (Wei et al., 2022; Zelikman et al., 2022; Schick et al., 2023; Mukherjee et al., 2023) or preference-guided optimization methods such as DPO (Rafailov et al., 2023; Tunstall et al., 2023; Yuan et al., 2024c; Grattafiori et al., 2024) and GRPO (Shao et al., 2024; DeepSeek-AI et al.,”
From Memory in the Age of AI Agents (2025) · cites this paper as Rafailov et al., 2023
RL is a complex and often unstable procedure.
“However, RL is a complex and often unstable procedure (Mnih et al., 2015; Rafailov et al., 2023)”
From ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification (2025) · cites this paper as Rafailov et al., 2023
Preference optimization allows structured preference learning without relying on reinforcement learning.
“This allows us to perform structured preference learning without relying on reinforcement learning (RL), which can be computationally extensive and unstable (Rafailov et al., 2023).”
From ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification (2025) · cites this paper as Rafailov et al., 2023
GIER offers a lightweight alternative that aligns model reasoning through prompts rather than fine-tuning or reward signals.
“Our approach also complements alignment methods like RLHF (Ouyang et al., 2022) and preference modeling (Rafailov et al., 2023), offering a lightweight alternative that aligns model reasoning through prompts rather than fine-tuning or reward signals.”
From GIER: Gap-Driven Self-Refinement for Large Language Models (2025) · cites this paper as Rafailov et al., 2023
Current state-of-the-art LLMs are aligned with honesty and helpfulness via training processes including reinforcement learning with human feedback (RLHF).
“Current state-of-the-art LLMs are aligned with these principles via training processes8,10,13, including reinforcement learning with human feedback14 (RLHF).”
From When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior (2025) · cites this paper as [13]
DPO simplifies RL on language models for the special case of learning from preference rankings over possible model outputs.
“The DPO algorithm simplifies RL on language models for this special case (Rafailov et al., 2023)”
From Fine-tuning Language Models for Factuality (2023) · cites this paper as Rafailov et al., 2023
DPO enables learning πθ from a fixed dataset of preferences without fitting an explicit reward function or sampling from the policy in the loop of training.
“DPO enables learning π θ from a fixed dataset of preferences, without fitting an explicit reward function or sampling from the policy in the loop of training (as is required in PPO).”
From Fine-tuning Language Models for Factuality (2023) · cites this paper as Rafailov et al., 2023
Verifiers, self-training, preference optimization, and verbal feedback turn evaluated trajectories into signals for selection and learning.
“Verifiers, self-training, preference optimization, and verbal feedback turn evaluated trajectories into signals for selection and learning [1-5].”
From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [4]
DPO tends to make model responses longer.
“Previous work (Gu et al., 2024;Ivison et al., 2023) has shown that DPO (Rafailov et al., 2023) tends to make model responses longer”
From Explaining Length Bias in LLM-Based Preference Evaluations (2025) · cites this paper as Rafailov et al., 2023
DPO aligns the model on preference pairs without relying on reward models.
“Our work closely connects to preference-optimization methods like Direct Preference Optimization [Rafailov et al., 2023], which align the model on preference pairs without relying on reward models.”
From Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Models (2025) · cites this paper as Rafailov et al., 2023
Cited by
Sorted by citations
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580 · see citing works
Self-citationEmpowering biomedical discovery with AI agents
Gao, S., Fang, A., Huang, Y., Giunchiglia, V., Noori, A., Schwarz, J. R., Ektefaie, Y., Kondic, J., & Zitnik, M. (2024) · Cell · doi:10.1016/j.cell.2024.09.022 · Cited by 315 · see citing works
A generalist medical language model for disease diagnosis assistance
Liu, X., Liu, H., Yang, G., Jiang, Z., Cui, S., Zhang, Z., Wang, H., Tao, L., Sun, Y., Song, Z., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03416-6 · Cited by 183 · see citing works
Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities
Ferrag, M. A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisztray, T., & Debbah, M. (2025) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2025.01.001 · Cited by 148 · see citing works
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024) · Findings of the Association for Computational Linguistics: ACL 2024 · doi:10.18653/v1/2024.findings-acl.624 · Cited by 147 · see citing works
Chen, S., Gao, M., Sasse, K., Hartvigsen, T., Anthony, B., Fan, L., Aerts, H., Gallifant, J., & Bitterman, D. S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02008-z · Cited by 112 · see citing works
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
“A different family of methods augments the language modeling objective to utilize feedback provided by the reward model [Korbak et al., 2023, Yuan et al., 2023, Rafailov et al., 2023].”
MethodGraph Retrieval-Augmented Generation for Large Language Models: A Survey
Procko, T. T., & Ochoa, O. (2024) · 2024 Conference on AI, Science, Engineering, and Technology (AIxSET) · doi:10.1109/aixset62544.2024.00030 · Cited by 72 · see citing works
Dahlgren Lindström, A., Methnani, L., Krause, L., Ericson, P., de Rituerto de Troya, Í. M., Coelho Mollo, D., & Dobbe, R. (2025) · Ethics and Information Technology · doi:10.1007/s10676-025-09837-2 · Cited by 54 · see citing works
Qwen2.5-Coder Technical Report
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.12186 · Cited by 65 · see citing works
“5-Coder does not produce inflated results due to test set leakage, we performed decontamination on all data, including both pre-training and post-training datasets.”
MethodLLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40 · see citing works
PlanGenLLMs: A Modern Survey of LLM Planning Capabilities
Wei, H., Zhang, Z., He, S., Xia, T., Pan, S., & Liu, F. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.958 · Cited by 36 · see citing works
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
Chhabra, A., Datta, S., Nahin, S. K., & Mohapatra, P. (2026) · IEEE Access · doi:10.1109/access.2026.3675554 · Cited by 28 · see citing works
“Similarly, in [180], the authors introduce SecAlign , which leverages alignment training via direct preference optimization (DPO) [181] to align agents toward preferring benign instructions over adversarial ones.”
Token-Budget-Aware LLM Reasoning
Han, T., Wang, Z., Fang, C., Zhao, S., Ma, S., & Chen, Z. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.1274 · Cited by 31 · see citing works
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
Vashurin, R., Fadeeva, E., Vazhentsev, A., Rvanova, L., Vasilev, D., Tsvigun, A., Petrakov, S., Xing, R., Sadallah, A., Grishchenkov, K., et al. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00737 · Cited by 21 · see citing works
A Survey on the Integration and Optimization of Large Language Models in Edge Computing Environments
Bhardwaj, S., Singh, P., & Pandit, M. K. (2024) · 2024 16th International Conference on Computer and Automation Engineering (ICCAE) · doi:10.1109/iccae59995.2024.10569285 · Cited by 18 · see citing works
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 17 · see citing works
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17 · see citing works
“Previous work has shown that GPT-4 accurately simulates human preferences in both summarisation (Rafailov et al., 2023) and instruction following (Dubois et al.”
BackgroundMethodInfluentialWang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X., Yang, J., Zhang, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01939 · Cited by 14
“Offline meth-ods like DPO (Rafailov et al., 2023), SimPO (Meng et al., 2024), and KTO (Ethayarajh et al., 2024) optimize policies using pre-collected preferences, typically from human annotators or LLMs.”
Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
Ye, J., Wang, Y., Huang, Y., Chen, D., Zhang, Q., Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P.-Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.02736 · Cited by 11 · see citing works
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
Yao, W., Heinecke, S., Niebles, J. C., Liu, Z., Feng, Y., Xue, L., Murthy, R., Chen, Z., Zhang, J., Arpit, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.02151 · Cited by 10 · see citing works
“Among the prevalent transformer RL methods are online RL algorithms such as Proximal Policy Optimization or PPO (Schulman et al., 2017), and offline RL techniques such as Implicit Language Q-Learning or ILQL (Snell et al., 2022) and Direct Preference Optimization or DPO (Rafailov et al., 2023).”
MethodFine-tuning Language Models for Factuality
Tian, K., Mitchell, E., Yao, H., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2311.08401 · Cited by 14 · see citing works
“Rafailov et al. (2023) show that the optimal policy π ∗ for the problem in Eq.”
InfluentialSelf-citationAutomated Design of Agentic Systems
Hu, S., Lu, C., & Clune, J. (2024) · arXiv · doi:10.48550/arxiv.2408.08435 · Cited by 10 · see citing works
“…et al., 2023) instead of manual design; in LLM alignment, learned loss functions (Lu et al., 2024a) out-perform most hand-designed ones such as DPO (Rafailov et al., 2024); The AI Scientist (Lu et al., 2024b) demonstrates an automated research pipeline, including the development of novel ML…”
A Survey of Post-Training Scaling in Large Language Models
Lai, H., Liu, X., Gao, J., Cheng, J., Qi, Z., Xu, Y., Yao, S., Zhang, D., Du, J., Hou, Z., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.140 · Cited by 8 · see citing works
Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.10759 · Cited by 8 · see citing works
“We employ DPO (Rafailov et al., 2024) to further optimize models to follow human preferences.”
MethodExternalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Zhou, C., Chai, H., Chen, W., Guo, Z., Shan, R., Song, Y., Xu, T., Yang, Y., Yu, A., Zhang, W., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.08224 · Cited by 1 · see citing works
Fang, J., Peng, Y., Zhang, X., Wang, Y., Yi, X., Zhang, G., Xu, Y., Wu, B., Liu, S., Li, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.07407 · Cited by 6 · see citing works
Mitigating Sycophancy in Large Language Models via Direct Preference Optimization
Khan, A. A., Alam, S., Wang, X., Khan, A. F., Neog, D. R., & Anwar, A. (2024) · 2024 IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825538 · Cited by 6 · see citing works
Explaining Length Bias in LLM-Based Preference Evaluations
Hu, Z., Song, L., Zhang, J., Xiao, Z., Wang, T., Chen, Z., Yuan, N. J., Lian, J., Ding, K., & Xiong, H. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.358 · Cited by 4 · see citing works
“• Using both AlpacaEval and AdapAlpaca, we analyze the impact of length bias in DPO.”
InfluentialDoes Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., & Huang, G. (2025) · arXiv · doi:10.48550/arxiv.2504.13837 · Cited by 3 · see citing works
“Previously, a reward model or preferences between responses were employed for reward modeling (Ouyang et al., 2022; Rafailov et al., 2023).”
Token-Budget-Aware LLM Reasoning
Han, T., Wang, Z., Fang, C., Zhao, S., Ma, S., & Chen, Z. (2024) · arXiv · Cited by 31 · see citing works
“Another way to in-centivize M θ to learn the token-budget preference is applying the DPO algorithm (Rafailov et al., 2023) to post-train the model.”
Gao, H.-A., Geng, J., Hua, W., Hu, M., Juan, X., Liu, H., Liu, S., Qiu, J., Qi, X., Wu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.21046 · Cited by 3 · see citing works
Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
Chen, X., Xu, J., Liang, T., He, Z., Pang, J., Yu, D., Song, L., Liu, Q., Zhou, M., Zhang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.21187 · Cited by 3 · see citing works
Seed-Coder: Let the Code Model Curate Data for Itself
Seed, B., Zhang, Y., Su, J., Sun, Y., Xi, C., Xiao, X., Zheng, S., Zhang, A., Liu, K., Zan, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.03524 · Cited by 3 · see citing works
“To build Seed-Coder-8B-Instruct, we first applied supervised fine-tuning (SFT) to adapt our base models to a wide range of real-world tasks, followed by direct preference optimization (DPO; Rafailov et al. [2023]) to further enhance specific capabilities such as code generation and reasoning.”
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence
Feng, S., Wang, Z., Wang, Y., Ebrahimi, S., Palangi, H., Miculicich, L., Kulshrestha, A., Rauschmayr, N., Choi, Y., Tsvetkov, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.11163 · Cited by 2 · see citing works
“This indicates that M ODEL S WARMS outputs are consistently preferred by both automatic metrics and human users, indicating M ODEL S WARMS ’ great potential to produce domain-specialized and community-specific LLM experts.”
Preference Leakage: A Contamination Problem in LLM-as-a-judge
Li, D., Sun, R., Huang, Y., Zhong, M., Jiang, B., Han, J., Zhang, X., Wang, W., & Liu, H. (2025) · arXiv · doi:10.48550/arxiv.2502.01534 · Cited by 2 · see citing works
“We also compare three learning methods, supervised fine-tuning (SFT), direct preference optimization (DPO) [51], and in-context learning (ICL) [10], to explore the different influences to them under preference leakage.”
Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy
Feng, Z., Chen, Z., Ma, J., Po, Y. T., Chersoni, E., & Li, B. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1126 · Cited by 2 · see citing works
Lee, J., Sakaguchi, K., & Bak, J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.528 · Cited by 2 · see citing works
Oh, G., Kim, S., Park, S., & Kim, B.-H. (2026) · Journal of Medical Internet Research · doi:10.2196/90693 · Cited by 1 · see citing works
“UltraMedical [27] is a medical LLM fine-tuned using SFT followed by preference optimization techniques such as direct preference optimization [22] or Kahneman-Tversky optimization [50].”
Natural Emergent Misalignment from Reward Hacking in Production RL
MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., Bouscal, N., Bowman, S., Bricken, T., Cloud, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.18397 · Cited by 1 · see citing works
DebateQA: Evaluating Question Answering on Debatable Knowledge
Xu, R., Qi, X., Qi, Z., Xu, W., & Guo, Z. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.44 · Cited by 1 · see citing works
Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Jiang, L., Chai, Y., Li, M., Liu, M., Fok, R., Dziri, N., Tsvetkov, Y., Sap, M., Albalak, A., & Choi, Y. (2025) · arXiv · doi:10.48550/arxiv.2510.22954 · Cited by 1 · see citing works
LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment
Li, L., Li, D., Chen, C., Ma, R., Yu, R., Lin, M., Yin, R., Fan, L., Shyr, C., Ma, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.25273 · Cited by 1 · see citing works
“This may involve supervised fine-tuning on expert-labeled rubric scores [71, 166], parameter-efficient adaptation such as LoRA [54], or preference optimization [114] using human or model-generated judgments.”
A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework
Li, C., Akhtar, Z., Kwak, M., Ji, Y., Zhang, H., Obi, T., Ren, Y., Wu, X., Sivarajkumar, S., Lehmann, H. P., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.25933 · Cited by 1 · see citing works
AdvAgent: Controllable Blackbox Red-teaming on Web Agents
Xu, C., Kang, M., Zhang, J., Liao, Z., Mo, L., Yuan, M., Sun, H., & Li, B. (2024) · arXiv · doi:10.48550/arxiv.2410.17401 · Cited by 1 · see citing works
Memory in the Age of AI Agents
Hu, Y., Liu, S., Yue, Y., Zhang, G., Liu, B., Zhu, F., Lin, J., Guo, H., Dou, S., Xi, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.13564 · Cited by 1 · see citing works
Ou, J., & Holve, E. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf081 · Cited by 1 · see citing works
CAPTURE: Disentangling Preference Drift from Memory Poisoning in Personalized LLM Agents
Hossain, S. M. A., Shayoni, R. K., & Morol, M. K. (2026) · arXiv · doi:10.48550/arxiv.2609.02265
“Alignment from human feedback fits a single population preference (Ouyang et al., 2022; Rafailov et al., 2023), and a growing line fits individuals instead, through profiles (Salemi et al., 2024), latent preferences learned from user edits (Gao et al., 2024), system-message generalization (Lee et…”
Liu, Q., Wang, M., & Xie, J. (2026) · 2026 6th International Conference on Machine Learning and Intelligent Systems Engineering (MLISE) · doi:10.1109/mlise70044.2026.11607496
Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
Tao, Y., Wang, T., Dong, Y., Liu, H., Zhang, K., Hu, X., & Li, G. (2025) · arXiv · doi:10.48550/arxiv.2510.09259 · Cited by 1 · see citing works
“Investigating the effectiveness of Self-Critique on these frontier models represents an important next step in ensuring the reliability of the entire LLM ecosystem.”
Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression
Johnson, W. (2026) · arXiv
“Future work could incorporate compression-degraded prompts into the preference learning process, drawing on Constitutional AI principles (Bai et al., 2022) to encode conciseness constraints, or leveraging Direct Preference Optimization (Rafailov et al., 2024) to train models that prefer proportionate responses regardl…”
ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification
Lee, H., Oh, S., Kim, J., Shin, J., & Tack, J. (2025) · arXiv · doi:10.48550/arxiv.2502.14565
“However, RL is a complex and often unstable procedure (Mnih et al., 2015; Rafailov et al., 2023), and it does not explicitly model the verification of intermediate reasoning steps, making it difficult to assess whether a model is confident in its current trajectory or prone to deviating toward…”
Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges
Motger, Q., Oriol, M., Marco, J., & Franch, X. (2026) · arXiv · doi:10.48550/arxiv.2607.26212
GIER: Gap-Driven Self-Refinement for Large Language Models
Dewri, R. (2025) · arXiv · doi:10.48550/arxiv.2509.00325
“…on aligning models with human preferences via reinforcement learning with human feedback (RLHF) (Ouyang et al., 2022; Bai et al., 2022a), preference modeling (Rafailov et al., 2023), critiques (Bai et al., 2022b), or by using high-quality examples for supervised fine-tuning (Zhou et al., 2023).”
Ai, Q., Zhan, J., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2501.02842
“Recently, there are also some offline reinforcement learning algorithms that do not necessaite training a reward model, such as DPO [45].”
Zhou, C., Zhang, X., Song, D., Chen, X., Gu, W., Ma, H., Tian, Y., Zhang, M., & Hu, L. (2025) · arXiv · doi:10.48550/arxiv.2502.09183
“DPO leads to a significant performance decline due to the intrinsic risk of reducing the likelihood of correct code during training (Feng et al., 2024; Tao et al., 2024; Pal et al., 2024).”
Beyond Similarity: Trustworthy Memory Search for Personal AI Agents
Zhang, J., Chen, K., Ma, J., Hu, Y., He, L., Zhang, Y., Liu, J., Yang, X., Zhang, T., & Jia, R. (2026) · arXiv · doi:10.48550/arxiv.2606.06054
“We formulate retrieval alignment as a Direct Preference Optimization (DPO) task [32].”
Purified OPSD: On-Policy Self-Distillation Without Losing How to Think
Shen, Z., Tong, J., Yan, S., Shen, C., Chen, H., Ye, W., Hu, X., Miao, R., Wang, H., Zhao, J., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.02234
“…reward while remaining close to the clean base distribution P 0 : This objective is the token-level analogue of KL-regularized policy improvement used in RLHF and DPO-style derivations [Rafailov et al., 2024], with P 0 serving as the reference policy and ∆ it serving as the implicit reward.”
FlowReasoner: Reinforcing Query-Level Meta-Agents
Gao, H., Liu, Y., He, Y., Dou, L., Du, C., Deng, Z., Hooi, B., Lin, M., & Pang, T. (2025) · arXiv · doi:10.48550/arxiv.2504.15257
Skin Tokens: A Learned Compact Representation for Unified Autoregressive Rigging
Zhang, J.-P., Pu, C.-F., Guo, M.-H., Cao, Y.-P., & Hu, S.-M. (2026) · arXiv · doi:10.48550/arxiv.2602.04805
RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation
Wu, S., Ma, Y., Zhao, Y., Hu, T., Jiang, O., Patwardhan, M., & Cohan, A. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1696
ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization
Wang, Y., Yang, L., Li, G., Wang, M., & Aragam, B. (2025) · arXiv · doi:10.48550/arxiv.2502.04306
Paech, S., Roush, A., Goldfeder, J., & Shwartz-Ziv, R. (2025) · arXiv · doi:10.48550/arxiv.2510.15061
“Our work closely connects to preference-optimization methods like Direct Preference Optimization [Rafailov et al., 2023], which align the model on preference pairs without relying on reward models.”
Does Spatial Cognition Emerge in Frontier Models?
Ramakrishnan, S. K., Wijmans, E., Kraehenbuehl, P., & Koltun, V. (2024) · arXiv · doi:10.48550/arxiv.2410.06468
MAPLE: Multi-Aspect Panels of LLM Evaluators for Open-Ended Questions
Jinji, M., Atarashi, K., Takeuchi, K., & Kashima, H. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1351
“Preference-based post-training methods such as Reinforcement Learning from Human Feed-back (RLHF) (Stiennon et al., 2020; Ouyang et al., 2022) and Direct Preference Optimization (DPO) (Rafailov et al., 2023; Song et al., 2024) have been central to aligning Large Language Models (LLMs) with human preferences.”
A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation
Chen, J., Chen, W., Lin, Z., & Vong, C. M. (2026) · arXiv · doi:10.48550/arxiv.2608.24419
“Every one is an invariance test; the label-set side has no headroom metric, no cross-set audit, no reporting convention Preference optimisation [43, 47] a preference signal distils into a policy, and a proxy reward is gameable in the limit that the signal means what it says.”
YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
D'Souza, J., Giglou, H. B., & Münch, Q. (2025) · arXiv · doi:10.48550/arxiv.2505.14279
Untitled work
Ovcharov, V. (2026) · arXiv
“Standard RLHF [17] and DPO [20] rely on human-generated preference pairs.”
TTRL: Test-Time Reinforcement Learning
Zuo, Y., Zhang, K., Sheng, L., Qu, S., Cui, G., Zhu, X., Li, H., Zhang, Y., Long, X., Hua, E., et al. (2025) · arXiv · Cited by 7 · see citing works
“However, these works primarily focus on open-domain instruction following [5, 58] rather than mathematical reasoning or employ preference-based optimization strategies [32] such as DPO [33] instead of online reinforcement learning algorithms.”
RSI-Anything: Unlocking Recursive Self-Improvement with World Models
Cui, T., Zeng, P., Xu, W., Xiao, H., Xue, S., Gao, Q., Ouyang, W., Heng, P. A., Yin, Z., Wu, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2610.13912
Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editing
Wu, Y., Ding, L., Shen, L., & Tao, D. (2025) · arXiv · doi:10.48550/arxiv.2511.12661
Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities
Bei, Y., Zhang, W., Wang, S., Chen, W., Zhou, S., Chen, H., Li, Y., Bu, J., Pan, S., Yu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.18019
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · see citing works
Self-citationAutoPal: Autonomous Adaptation to Users for Personal AI Companionship
Cheng, Y., Liu, W., Xu, K., Hou, W., Ouyang, Y., Leong, C. T., Li, W., Wu, X., & Zheng, Y. (2024) · arXiv · doi:10.48550/arxiv.2406.13960 · Cited by 2 · see citing works
T2I-Eval-R1: Reinforcement Learning-Driven Reasoning for Interpretable Text-to-Image Evaluation
Ma, Z.-A., Lan, T., Tu, R.-C., Liu, S.-H., Huang, H., Wu, Z., Xu, C., & Mao, X.-L. (2025) · arXiv · doi:10.48550/arxiv.2505.17897
- Authors
- Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 53728-53741
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133208539
- Cited by
- 686
- References
- 0
- Citations by year
- 2026: 486 · 2025: 127 · 2024: 72 · 2023: 1
- FWCI
- 121.54
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- neural information processing systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- preference optimization, reward modeling, language models, reward models, preference data, reinforcement learning from human feedback
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
