Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 17

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization

Xiao, Jiancong, Li, Ziniu, Xie, Xingyu, Getzen, Emily, Fang, Cong, Long, Qi, Su, Weijie J.·2025·Journal of the American Statistical Association
DOI 10.1080/01621459.2025.2555067
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1080/01621459.2025.2555067

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 37

  1. Anthropic, A. (2024), “The Claude 3 Model Family: Opus, Sonnet, Haiku,” Claude-3 Model Card.
  2. 10.52202/075280-2321 · DOI
  3. Arrow, K. J. (2012), Social Choice and Individual values (Vol. 12), New Haven, CT: Yale University Press.
  4. Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. (2024), “A General Theoretical Paradigm to Understand Learning from Human Preferences,” in International Conference on Artificial Intelligence and Statistics, PMLR, pp. 4447–4455.
  5. Bai Y. Jones A. Ndousse K. Askell A. Chen A. DasSarma N. Drain D. Fort S. Ganguli D. Henighan T. Joseph N. Kadavath S. Kernion J. Conerly T. El-Showk S. Elhage N. Hatfield-Dodds Z. Hernandez D. Hume T. Johnston S. Kravec S. Lovitt L. Nanda N. Olsson C. Amodei D. Brown T. Clark J. McCandlish S. Olah C. Mann B. and Kaplan J. (2022) “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback ” available at https://arxiv.org/abs/2204.05862 · arXiv
  6. Bubeck S. Chandrasekaran V. Eldan R. Gehrke J. Horvitz E. Kamar E. Lee P. Lee Y. T. Li Y. Lundberg S. et al. (2023) “Sparks of Artificial General Intelligence: Early Experiments with gpt-4 ” arXiv preprint arXiv:2303.12712. · arXiv
  7. Casper S. Davies X. Shi C. Gilbert T. K. Scheurer J. Rando J. Freedman R. Korbak T. Lindner D. Freire P. et al. (2023) “Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback ” arXiv preprint arXiv:2307.15217. · arXiv
  8. Chakraborty S. Qiu J. Yuan H. Koppel A. Huang F. Manocha D. Bedi A. S. and Wang M. (2024) “Maxmin-RLHF: Towards Equitable Alignment of Large Language Models with Diverse Human Preferences ” arXiv preprint arXiv:2402.08925. · arXiv
  9. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2023), “Palm: Scaling Language Modeling with Pathways,” Journal of Machine Learning Research, 24, 1–113.
  10. Dong H. Xiong W. Pang B. Wang H. Zhao H. Zhou Y. Jiang N. Sahoo D. Xiong C. and Zhang T. (2024) “RLHF Workflow: From Reward Modeling to Online RLHF ” arXiv e-prints pp. arXiv–2405.
  11. Duetting P. Mirrokni V. Leme R. P. Xu H. and Zuo S. (2023) “Mechanism Design for Large Language Models ” arXiv preprint arXiv:2310.10826. · DOI
  12. Eloundou T. Manning S. Mishkin P. and Rock D. (2023) “Gpts are gpts: An Early Look at the Labor Market Impact Potential of Large Language Models ” arXiv preprint arXiv:2303.10130. · DOI · Paper page
  13. 10.1007/978-94-015-9273-4_9 · DOI
  14. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2020), “The Curious Case of Neural Text Degeneration,” in International Conference on Learning Representations.
  15. 10.1121/1.2016299 · DOI · Paper page
  16. Kaufmann T. Weng P. Bengs V. and Hüllermeier E. (2023) “A Survey of Reinforcement Learning from Human Feedback.” arXiv preprint arXiv:2312.14925. · arXiv
  17. Kenton, J. D. M.-W. C., and Toutanova, L. K. (2019), “Bert: Pre-Training of deep Bidirectional Transformers for Language Understanding,” in Proceedings of NAACL-HLT, pp. 4171–4186.
  18. Li, Z., Chen, C., Xu, T., Qin, Z., Xiao, J., Luo, Z.-Q., and Sun, R. (2025), “Preserving diversity in Supervised Fine-Tuning of Large Language Models,” in The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=NQEe7B7bSw
  19. Liang W. Izzo Z. Zhang Y. Lepp H. Cao H. Zhao X. Chen L. Ye H. Liu S. Huang Z. et al. (2024) “Monitoring AI-Modified Content at Scale: A Case Study on the Impact of Chatgpt on AI Conference Peer Reviews ” arXiv preprint arXiv:2403.07183. · arXiv
  20. Liu K. Long Q. Shi Z. Su W. J. and Xiao J. (2025) “Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium ” arXiv preprint arXiv:2503.10990. · DOI
  21. Luce, R. D. (2012), Individual Choice Behavior: A Theoretical Analysis, North Chelmsford, MA: Courier Corporation.
  22. OpenAI. (2023) “Gpt-4 Technical Report ” arXiv preprint arXiv:2303.08774. · arXiv
  23. Rafailov R. Sharma A. Mitchell E. Ermon S. Manning C. D. and Finn C. (2023) “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” · DOI · Paper page
  24. Rényi, A. (1961), “On Measures of Entropy and Information,” in Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics (Vol. 4), University of California Press, pp. 547–562.
  25. Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., and Hashimoto, T. (2023), “Whose Opinions Do Language Models Reflect?” in International Conference on Machine Learning, PMLR, pp. 29971–30004.
  26. Shi Z. Liu K. Long Q. Su W. J. and Xiao J. (2025) “Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching ” arXiv preprint arXiv:2505.20627. · arXiv
  27. Song Z. Cai T. Lee J. D. and Su W. J. (2023) “Reward Collapse in Aligning Large Language Models ” arXiv preprint arXiv:2305.17608. · arXiv
  28. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. (2020), “Learning to Summarize with Human Feedback,” Advances in Neural Information Processing Systems (Vol. 33), pp. 3008–3021.
  29. Team G. Anil R. Borgeaud S. Wu Y. Alayrac J.-B. Yu J. Soricut R. Schalkwyk J. Dai A. M. Hauth A. et al. (2023) “Gemini: A Family of Highly Capable Multimodal Models ” arXiv preprint arXiv:2312.11805. · arXiv
  30. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017), “Attention Is All You Need,” in Advances in Neural Information Processing Systems 30, pp. 5998–6008.
  31. Wang H. Lin Y. Xiong W. Yang R. Diao S. Qiu S. Zhao H. and Zhang T. (2024) “Arithmetic Control of LLMS for Diverse User Preferences: Directional Preference Alignment with Multi-objective Rewards ” arXiv preprint arXiv:2402.18571. · DOI
  32. Xiao J. Shi Z. Liu K. Long Q. and Su W. J. (2025) “Theoretical tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory ” arXiv preprint arXiv:2506.12350. · arXiv
  33. Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. (2023), “Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under kl-constraint,” in ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models.
  34. Xu, W., Dong, S., Lu, X., Lam, G., Wen, Z., and Van Roy, B. (2024), “Rlhf and IIA: Perverse Incentives,” in ICML 2024 Workshop on Models of Human Feedback for AI Alignment.
  35. Yao Z. Aminabadi R. Y. Ruwase O. Rajbhandari S. Wu X. Awan A. A. Rasley J. Zhang M. Li C. Holmes C. et al. (2023) “Deepspeed-chat: Easy Fast and Affordable rlhf Training of Chatgpt-Like Models at All Scales ” arXiv preprint arXiv:2308.01320. · arXiv
  36. Zhang S. Roller S. Goyal N. Artetxe M. Chen M. Chen S. Dewan C. Diab M. Li X. Lin X. V. et al. (2022) “Opt: Open Pre-Trained Transformer Language Models ” arXiv preprint arXiv:2205.01068. · arXiv
  37. Zhong H. Deng Z. Su W. J. and Wu Z. S. and Zhang L. (2024) “Provable Multi-Party Reinforcement Learning with Diverse Human Feedback ” arXiv preprint arXiv:2403.05006. · arXiv

Cited by

Metadata

Authors
Xiao, J., Li, Z., Xie, X., Getzen, E., Fang, C., Long, Q., & Su, W. J.
Affiliations
  • Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania
  • School of Data Science, The Chinese University of Hong Kong
  • Department of Mathematics, National University of Singapore
  • School of Intelligence Science and Technology, Peking University
  • Department of Statistics and Data Science, University of Pennsylvania
ORCID
Su, Weijie J.
Venue
Journal of the American Statistical Association
Publisher
Informa UK Limited
Published
2025-10-02
Volume
120
Issue
552
Pages
2154-2164
ISSN
  • 0162-1459
  • 1537-274X
DOI
10.1080/01621459.2025.2555067
Cited by
17
References
5
Language
en
Licence
doi.org/10.1080/policies-text-data-mining (tdm)
Funders
NIH grants (R01-EB036016, RF1AG063481, U01CA274576, NSF DMS-2310679)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.