Preference Ranking Optimization for Human Alignment
Open accessAbstract
Large language models (LLMs) often contain misleading content, emphasizing the need to align them with human values to ensure secure AI systems. Reinforcement learning from human feedback (RLHF) has been employed to achieve this alignment. However, it encompasses two main drawbacks: (1) RLHF exhibits complexity, instability, and sensitivity to hyperparameters in contrast to SFT. (2) Despite massive trial-and-error, multiple sampling is reduced to pair-wise contrast, thus lacking contrasts from a macro perspective. In this paper, we propose Preference Ranking Optimization (PRO) as an efficient SFT algorithm to directly fine-tune LLMs for human alignment. PRO extends the pair-wise contrast to accommodate preference rankings of any length. By iteratively contrasting candidates, PRO instructs the LLM to prioritize the best response while progressively ranking the rest responses. In this manner, PRO effectively transforms human alignment into aligning the probability ranking of n responses generated by LLM with the preference ranking of humans towards these responses. Experiments have shown that PRO outperforms baseline algorithms, achieving comparable results to ChatGPT and human responses through automatic-based, reward-based, GPT-4, and human evaluations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Automated metrics like BLEU and ROUGE often do not correlate with human judgment, making them unreliable for evaluation and training.
“Automated metrics like BLEU and ROUGE [99] often do not correlate with human judgment [103, 164, 166, 177], making them unreliable for evaluation and training.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [177]
The Human and GPT-4 Evaluation section says reward models may distort human preferences, and Evaluation Metrics calls human evaluation the gold standard. But the paper does not say BLEU or ROUGE often fail to correlate with human judgment or are unreliable for evaluation and training; it uses BLEU to assess text quality. Page numbers are unavailable in our copy. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,682 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons
Bradley, R. A., & Terry, M. E. (1952) · Biometrika · doi:10.2307/2334029 · Cited by 2,412 · see citing works
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · see citing works
Self-citationDeep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · see citing works
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,028 · see citing works
Plackett, R. L. (1975) · Applied Statistics · doi:10.2307/2346567 · Cited by 731 · see citing works
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550 · see citing works
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · see citing works
Fine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413 · see citing works
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · see citing works
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise ComparisonsPreprint
Zhu, B.-H., Jiao, J.-T., & Jordan, M. I. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2301.11270 · Cited by 314 · see citing works
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293 · see citing works
LIMA: Less Is More for Alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2400 · Cited by 275 · see citing works
WebGPT: Browser-assisted question-answering with human feedback
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.09332 · Cited by 272 · see citing works
Peng, B., Li, C., He, P., Galley, M., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2304.03277 · Cited by 183 · see citing works
Interactive Learning from Policy-Dependent Human Feedback
MacGlashan, J., Ho, M. K., Loftin, R., Peng, B., Wang, G., Roberts, D., Taylor, M. E., & Littman, M. L. (2017) · arXiv · doi:10.48550/arxiv.1701.06049 · Cited by 163 · see citing works
Individual Choice Behavior A Theoretical Analysis.
Suppes, P., & Luce, R. D. (1961) · Journal of the American Statistical Association · doi:10.2307/2282347 · Cited by 135 · see citing works
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., & Li, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.187 · Cited by 124 · see citing works
Self-citationLee, K., Smith, L., & Abbeel, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05091 · Cited by 86 · see citing works
RRHF: Rank Responses to Align Language Models with Human Feedback without tears
Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., & Huang, F. (2023) · arXiv · doi:10.48550/arxiv.2304.05302 · Cited by 41 · see citing works
Self-citationFine-Grained Human Feedback Gives Better Rewards for Language Model Training
Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P., Smith, N. A., Ostendorf, M., & Hajishirzi, H. (2023) · arXiv · doi:10.48550/arxiv.2306.01693 · Cited by 40 · see citing works
RAFT: Reward rAnked FineTuning for Generative Foundation Model AlignmentPreprint
Dong, H., Xiong, W. H., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., & Zhang, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.06767 · Cited by 38 · see citing works
Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces
Warnell, G., Waytowich, N. R., Lawhern, V. J., & Stone, P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11485 · Cited by 37 · see citing works
Large Language Models are not Fair Evaluators
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., & Sui, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.17926 · Cited by 34 · see citing works
Chain of Hindsight Aligns Language Models with Feedback
Liu, H., Sferrazza, C., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2302.02676 · Cited by 30 · see citing works
Reinforcement Learning from Diverse Human Preferences
Xue, W., An, B., 颜水成, & Xu, Z. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/586 · Cited by 9 · see citing works
Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue PolicyPreprint
Lei, W., Zhang, Y., Song, F., Liang, H., Mao, J., Lv, J., Yang, Z., & Chua, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.07433 · Cited by 9 · see citing works
Self-citationOffline RL for Natural Language Generation with Implicit Language Q LearningPreprint
Snell, C., Kostrikov, I., Su, Y., Yang, M., & Levine, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.11871 · Cited by 9 · see citing works
Fine-Tuning Language Models with Advantage-Induced Policy Alignment
Zhu, B., Sharma, H., Frujeri, F. V., Dong, S., Zhu, C., Jordan, M. I., & Jiao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.02231 · Cited by 6 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
3 of the 41 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Gao, L.; Schulman, J.; and Hilton, J. 2022. Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760. · arXiv
- Precup, D.; and Teh, Y. W., eds., Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, 2285-2294.
- PMLR.
- Philadel- phia, Pennsylvania, USA: Association for Computational Linguistics.
- Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H. 2022. Self-Instruct: Aligning Language Model with Self Generated Instructions. arXiv:2212.10560. · arXiv
Cited by
Sorted by citations
A review of large language models and autonomous agents in chemistry
Ramos, M. C., Collison, C. J., & White, A. D. (2025) · Chemical Science · doi:10.1039/d4sc03921a · Cited by 274 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 173 · see citing works
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., & Li, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.187 · Cited by 124 · see citing works
Self-citationLearning from models beyond fine-tuning
Zheng, H., Shen, L., Tang, A., Luo, Y., Hu, H., Du, B., Wen, Y., & Tao, D. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00961-0 · Cited by 114 · see citing works
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
When Search Engine Services Meet Large Language Models: Visions and Challenges
Xiong, H., Bian, J., Li, Y., Li, X., Du, M., Wang, S., Yin, D., & Helal, S. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3451185 · Cited by 82 · see citing works
Self-citationHan, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 72 · see citing works
Foundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 76 · see citing works
Self-citationDahlgren Lindström, A., Methnani, L., Krause, L., Ericson, P., de Rituerto de Troya, Í. M., Coelho Mollo, D., & Dobbe, R. (2025) · Ethics and Information Technology · doi:10.1007/s10676-025-09837-2 · Cited by 54 · see citing works
Franceschelli, G., & Musolesi, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15278 · Cited by 52 · see citing works
Shi, H. H., Fang, L., Chen, X., Gu, C., Ma, K., Zhang, X., Zhang, Z., Gu, J., & Lim, E. G. (2024) · IET Smart Grid · doi:10.1049/stg2.12191 · Cited by 46 · see citing works
Liu, Q., Song, Z., Liang, Y., Xie, Z., Zhang, S., Zhang, J., & Li, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130113 · Cited by 40 · see citing works
Self-citationUnderstand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Dong, G., Zhu, Y., Zhang, C., Wang, Z., Wen, J.-R., & Dou, Z. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714717 · Cited by 42 · see citing works
Generative AI Meets Wireless Networking: An Interactive Paradigm for Intent-Driven Communications
Qin, X., Sun, M., Dai, J., Ma, P., Cao, Y., Zhang, J., Wang, J., Xu, X., Zhang, P., & Niyato, D. T. (2025) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2025.3567613 · Cited by 32 · see citing works
Llama 2: Early Adopters' Utilization of Meta's New Open-Source Pretrained ModelPreprint
Roumeliotis, K. I., Tselikas, N. D., & Nasiopoulos, D. Κ. (2023) · Preprints.org · doi:10.20944/preprints202307.2142.v2 · Cited by 29 · see citing works
JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration
Lin, Y., Lin, Z., Chen, H., Pan, P., Li, C., Chen, S., Wen, K., Jin, Y., Li, W., & Ding, X. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02084 · Cited by 27 · see citing works
Llama 2: Early Adopters' Utilization of Meta's New Open-Source Pretrained ModelPreprint
Roumeliotis, K. I., Tselikas, N. D., & Nasiopoulos, D. Κ. (2023) · Preprints.org · doi:10.20944/preprints202307.2142.v1 · Cited by 21 · see citing works
Duan, R., Zhu, C., Chen, Y., Hu, Y., Shi, J., & Ramani, K. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735824 · Cited by 19 · see citing works
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · doi:10.31219/osf.io/ac4sp · Cited by 51 · see citing works
Self-citationKellogg, K. C., Lifshitz‐Assaf, H., Randazzo, S., Mollick, E., Dell’Acqua, F., McFowland, E., Candelon, F., & Lakhani, K. R. (2025) · Information and Organization · doi:10.1016/j.infoandorg.2025.100559 · Cited by 18 · see citing works
A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applications
Shang, W., & Huang, X. S. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3609877 · Cited by 19 · see citing works
Zhu, C., Hu, X., Wu, H., Qin, C., Zhu, H., & Xiong, H. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104261 · Cited by 16 · see citing works
Kirk, H. R., Bean, A., Vidgen, B., Röttger, P., & Hale, S. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.148 · Cited by 26 · see citing works
Yager, K. G. (2024) · Digital Discovery · doi:10.1039/d4dd00178h · Cited by 12 · see citing works
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., & Li, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.08244 · Cited by 124 · see citing works
Self-citationRevolutionizing optical networks: The integration and impact of large language models
Cruzes, S. (2025) · Optical Switching and Networking · doi:10.1016/j.osn.2025.100812 · Cited by 9 · see citing works
MedicalCLIP: Anomaly-Detection Domain Generalization with Asymmetric Constraints
Hua, L., Luo, Y., Qi, Q., & Long, J. (2024) · Biomolecules · doi:10.3390/biom14050590 · Cited by 9 · see citing works
LLM-Sentry: A Model-Agnostic Human-in-the-Loop Framework for Securing Large Language Models
Irtiza, S., Akbar, K. A., Yasmeen, A., Khan, L. R., Daescu, O., & Thuraisingham, B. M. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00036 · Cited by 8 · see citing works
Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
Winata, G. I., Zhao, H., Das, A., Tang, W., Yao, D. D. W., Zhang, S.-X., & Sahu, S. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.17541 · Cited by 11 · see citing works
Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation
Gao, X., Xiong, Y., Wang, D., Guan, Z., Shi, Z., Wang, H., & Li, S. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694987 · Cited by 6 · see citing works
Self-citationMitigating Sycophancy in Large Language Models via Direct Preference Optimization
Khan, A. A., Alam, S., Wang, X., Khan, A. F., Neog, D. R., & Anwar, A. (2024) · 2024 IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825538 · Cited by 6 · see citing works
Human in the loop reinforcement learning framework for adaptive phishing detection
Khan, S. (2025) · Discover Computing · doi:10.1007/s10791-025-09849-y · Cited by 5 · see citing works
Learning to Compare Hardware Designs for High-Level Synthesis
Bai, Y., Sohrabizadeh, A., Ding, Z., Liang, R., Li, W., Wang, D., Ren, H. M., Sun, Y., & Cong, J. (2024) · ACM/IEEE Workshop on Machine Learning for CAD (MLCAD) · doi:10.1145/3670474.3685940 · Cited by 4 · see citing works
Diverse AI Feedback For Large Language Model Alignment
Yu, T., Lin, T.-E., Wu, Y., Yang, M., Huang, F., & Li, Y. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00746 · Cited by 3 · see citing works
Self-citationLearning to Compare Hardware Designs for High-Level Synthesis
Bai, Y., Sohrabizadeh, A., Ding, Z., Liang, R., Li, W., Wang, D., Ren, H. M., Sun, Y., & Cong, J. (2024) · ACM/IEEE Workshop on Machine Learning for CAD (MLCAD) · doi:10.1109/mlcad62225.2024.10740257 · Cited by 2 · see citing works
Amirizaniani, M., Martin, E., Mashhadi, A., & Shah, C. S. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744598 · Cited by 2 · see citing works
Feng, Y., Gui, L., Jiang, Y., Jian-feng, H., Ou, D., Liu, Q., Lv, F., & Xu, Y. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737204 · Cited by 3 · see citing works
Evaluating and Improving the Effectiveness of Synthetic Chest X-Rays for Medical Image Analysis
Prakash, E., Valanarasu, J. M. J., Chen, Z., Reis, E. P., Johnston, A., Pareek, A., Blüthgen, C., Gatidis, S., Olsen, C., Chaudhari, A. S., et al. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00463 · Cited by 2 · see citing works
Using Large Language Models to Automate Annotation and Part-of-Math Tagging of Math Equations
Shan, R., & Youssef, A. S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-66997-2_1 · Cited by 2 · see citing works
Embedding an Ethical Mind: Aligning Text-to-Image Synthesis via Lightweight Value Optimization
Wang, X., Yi, X., Xie, X., & Jia, J. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681652 · Cited by 2 · see citing works
VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning
Lee, J. S., Kim, J., Na, J., Park, J., & Kim, H. J. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i4.32474 · Cited by 3 · see citing works
Kellogg, K. C., Lifshitz‐Assaf, H., Randazzo, S., Mollick, E., Dell’Acqua, F., McFowland, E., Candelon, F., & Lakhani, K. R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4857373 · Cited by 2 · see citing works
ID-GMLM: Intelligent Decision-Making with Integrated Graph Models and Large Language Models
Meng, Z., Meng, F., Lin, R., & Wu, B. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i18.34142 · Cited by 1 · see citing works
The superalignment of superhuman intelligence with large language models
Huang, M., Wang, Y., Cui, S., Ke, P., & Tang, J. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4348-6 · Cited by 1 · see citing works
The Fair Game: Auditing & debiasing AI algorithms over time
Basu, D., & Das, U. (2025) · Cambridge Forum on AI Law and Governance · doi:10.1017/cfl.2025.8 · Cited by 1 · see citing works
Push and Pull: A Framework for Measuring Attentional Agency on Digital Platforms
Wojtowicz, Z., Jain, S., & Vincent, N. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732043 · Cited by 1 · see citing works
Noise-Aware Direct Preference Optimization for RLAIF
Toleu, A., Tolegen, G., Pak, A., & Jaxylykova, A. (2025) · Applied Sciences · doi:10.3390/app151910328 · Cited by 1 · see citing works
DOM-AKG: Few-Shot Construction of Domain Academic Knowledge Graph Based on LLM
Tang, W., Mao, X., Lv, K., & Liu, M. (2025) · International Conference on Electronic Information Engineering and Computer Science (EIECS) · doi:10.1109/eiecs67708.2025.11283248 · Cited by 1 · see citing works
Uni-Layout: Integrating Human Feedback in Unified Layout Generation and Evaluation
Lu, S., Chen, Y., Feng, W., Fan, J., Li, F., Zhang, Z., Lv, J., Shen, J., Law, C., & Liang, J. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755064 · Cited by 1 · see citing works
Defending Against Jailbreak Through Early Exit Generation of Large Language Models
Zhao, C., Dou, Z., & Huang, K. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-4445-5_36 · Cited by 1 · see citing works
Jia, Z., Huang, C., Zhu, Y., Fei, H., Deng, Y., Yuan, Z., Zhang, J., Zhang, J., & Zhou, J. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01764 · Cited by 1 · see citing works
MAPLE: Multi-Aspect Panels of LLM Evaluators for Open-Ended Questions
Jinji, M., Atarashi, K., Takeuchi, K., & Kashima, H. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1351
“…post-training methods such as Reinforcement Learning from Human Feed-back (RLHF) (Stiennon et al., 2020; Ouyang et al., 2022) and Direct Preference Optimization (DPO) (Rafailov et al., 2023; Song et al., 2024) have been central to aligning Large Language Models (LLMs) with human preferences.”
Optimizing Personalized Robot Actions with Ranking of Trajectories
Huang, H. M., Liu, Y., Yuan, S., Wen, C., Hao, Y., & Fang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78110-0_1
Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model AlignmentPreprint
Cai, Y., Yuan, Y., Shi, J., & Lin, Q. (2024) · Qeios · doi:10.32388/slo0cb
Persona Extraction and Integration with Large Language Models Towards Personalized Dialogues
Qin, X., Mu, K., & Li, J. J. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4276-2_12
Patch the Leak: Strengthening CodeLLMs Against Privacy Extraction Threats
Guo, Y., Ma, W., Xiao, X., Wen, S., Di, P., & Zhu, X. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3732931
Consideration Sets Based Hierarchical Product Configurator Design: A Theoretical Analysis
Wang, Y., & Mo, D. Y. W. (2025) · International Conference on Advanced Algorithms and Control Engineering (ICAACE) · doi:10.1109/icaace65325.2025.11020269
Multi-Objective Kahneman-Tversky Optimization (Mo-Kto): A Novel Human Alignment Technique for LlmPreprint
Xie, E., Hu, H., & Zhang, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5356556
BP-LLM : Belief Propagation for Binary Feedback in Large Language Model Alignment
Liang, J. E. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.734
Training Generative Judge with Hard Negative Mining: A Metric Learning Perspective
Chen, C. (2025) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai66417.2025.00112
Preference Ranking Optimization for Human Alignment
Association for Artificial Intelligence 2024, Huang, F., Li, Y., Li, M., Song, F., Wang, H., Yu, H., & Yu, B. (2024) · Underline Science Inc. · doi:10.48448/zz0t-yf58
Self-citationA Study on Diabetes Health Management Dialogue Data Annotation Method Based on Few-shot-CoT
Sheng, G., & Yang, J. (2025) · International Conference on Big Data and Intelligent Medicine (BDIMed) · doi:10.1109/bdimed67722.2025.11309335
Mohammadi, F., Maghool, S., & Ceravolo, P. (2025) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa66935.2025.11315321
Nexus scissor: enhance open-access language model safety by connection pruning
Mai, P., Yang, Y., Yan, R., & Pang, Y. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-025-00046-2
Jiao, R., Zhang, Y., Li, J., & Ma, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00071
Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
Song, H., Choi, J., & Kim, M. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3777939
Partial Pairwise Preference-Driven Task Allocation in Volunteer Crowdsourcing
Li, J., Fu, X., Ding, J., Li, J., & Li, Y. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-21171-2_13
Self-citationAligning Query Rewriting with Human Cognition and Preference in E-Commerce Search
Ou, R., Wang, K., Shao, J., Zhang, T., & Huo, C. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792806
Zhang, Z., Yi, X., Dai, Z., Yu, X., Zhang, S., Liu, B., Cao, J., & Zhao, H. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791926
Mabunda, C., Amadi-Echendu, A. P., & Ntoyanto-Tyatyantsi, N. (2026) · Journal of Transport and Supply Chain Management · doi:10.4102/jtscm.v20i0.1299 · Cited by 1 · see citing works
Zhang, Q., 吴贞东, Lin, Q., Chen, H., & Yang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7078-2_31
RankNB: Ranking-Aware Direct Preference Optimization for Alignment of a Nanobody Diffusion Model
Wu, Y., Wu, D., Shu, J., Linker, S. M., Hrovatin, K., Winkel, M., & Tan, F. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462703
Mitigating Security Risks in Large Language Models: A Full Lifecycle Perspective
Wang, Y. K., Bai, Z., Huo, J., Wang, B., Liu, J., Cao, H., Meng, F., Yang, X., & Gao, Y. (2026) · Machine Learning · doi:10.1007/s10994-025-06930-x
What Makes a Good Response? Learning Personal Preferences from Interpretable Features
Afsharrad, A., Soroka, E., O’Neill, D., & Lall, S. (2026) · ICON · doi:10.1109/icnlp69856.2026.11527497
Unifying Data, Memory, and Compute Efficiency in LLM Training: A Survey
Schmidt, V., Nguyen, H. H., Jung, C., Salehi, S., & Schmeink, A. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3702625
Online Communities as Collaborative Testbeds for Collective Alignment
Zent, M., Khadar, M., Yarosh, S., Chancellor, S., & Kaur, H. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812370
URLcoat: Exploiting Web Search Capability to Jailbreak Large Language Models
Sun, Y., Du, L., Su, Z., Wang, Y., & Liu, H. (2026) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp63933.2026.00082
VALUE: Value-Aware Large Language Model for Query Rewriting via Weighted Trie in Sponsored Search
Zhang, X., Chen, G., Zuo, B., Li, F., Wang, P., Xu, J., & Zheng, B. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770854.3783951 · Cited by 1 · see citing works
RaTA-Tool: Retrieval-Based Tool Selection with Multimodal Large Language Models
Mattioli, G., Turri, E., Sarto, S., Baraldi, L., Cornia, M., Baraldi, L., & Cucchiara, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31397-3_21
Wang, P., Huang, H., Sun, J., Wang, H., Jiang, W., Ge, T., Cao, J., & He, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37083-9_29
Self-citation
- Authors
- Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., & Wang, H.
- Affiliations
- Peking University (CN)
- Alibaba Group (China) (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Volume
- 38
- Issue
- 17
- Pages
- 18990-18998
- ISSN
- 2159-5399 (ISSN-L)
- arXiv
- 2306.17492
- OpenAlex
- W4393160410
- Cited by
- 77
- References
- 41
- Citations by year
- 2026: 18 · 2025: 35 · 2024: 18 · 2023: 6
- FWCI
- 20.95
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62036001)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- human alignment, large language models, preference ranking, supervised fine-tuning, reinforcement learning from human feedback, AI alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
