CoChat
Sign Up
PreprintCited by 3Topic ModelingAcademic Writing and PublishingRecommender Systems and Techniques

REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning

Open access
Taechoyotin, Pawin, Acuna, Daniel·2025·arXiv
DOI 10.48550/arxiv.2505.11718arXiv 2505.11718
Open this paper in CoChat — free accountOpen PDF

Abstract

AI-based peer review systems tend to produce shallow and overpraising suggestions compared to human feedback. Here, we evaluate how well a reasoning LLM trained with multi-objective reinforcement learning (REMOR) can overcome these limitations. We start by designing a multi-aspect reward function that aligns with human evaluation of reviews. The aspects are related to the review itself (e.g., criticisms, novelty) and the relationship between the review and the manuscript (i.e., relevance). First, we perform supervised fine-tuning of DeepSeek-R1-Distill-Qwen-7B using LoRA on PeerRT, a new dataset of high-quality top AI conference reviews enriched with reasoning traces. We then apply Group Relative Policy Optimization (GRPO) to train two models: REMOR-H (with the human-aligned reward) and REMOR-U (with a uniform reward). Interestingly, the human-aligned reward penalizes aspects typically associated with strong reviews, leading REMOR-U to produce qualitatively more substantive feedback. Our results show that REMOR-U and REMOR-H achieve more than twice the average rewards of human reviews, non-reasoning state-of-the-art agentic multi-modal AI review systems, and general commercial LLM baselines. We found that while the best AI and human reviews are comparable in quality, REMOR avoids the long tail of low-quality human reviews. We discuss how reasoning is key to achieving these improvements and release the Human-aligned Peer Review Reward (HPRR) function, the Peer Review Reasoning-enriched Traces (PeerRT) dataset, and the REMOR models, which we believe can help spur progress in the area.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.11718

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Remor employs multi-objective reinforcement learning to address the common failure mode of AI-generated reviews being shallow.

    “A direct application of this principle is seen in Remor (Taechoyotin and Acuna, 2025), which employs multi-objective reinforcement learning to address the common failure mode of AI-generated reviews being shallow (Wei et al., 2025).”

    From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Taechoyotin and Acuna, 2025

  • Frameworks like REMOR frame review generation as a policy optimized to maximize a multi-objective reward function designed to encapsulate human preferences for a helpful review.

    “Frameworks like REMOR (Taechoyotin and Acuna, 2025) frame review generation as a policy that is optimized to maximize a multi-objective reward function, which is carefully designed to encapsulate human preferences for what constitutes a helpful review.”

    From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Taechoyotin and Acuna, 2025

  • Remor's Human-aligned Peer Review Reward is a composite function that quantifies multiple facets of review quality, including criticism, relevance, and actionable suggestions.

    “The framework's core innovation is its Human-aligned Peer Review Reward, a composite function that quantifies multiple facets of review quality, including criticism, relevance, and actionable suggestions.”

    From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Taechoyotin and Acuna, 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 52

  1. [1]Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005.
  2. [2]Alberto Bartoli, Andrea De Lorenzo, Eric Medvet, and Fabiano Tarlao. Your paper has been accepted, rejected, or whatever: Automatic generation of scientific paper reviews. In Availability, Reliability, and Security in Information Systems: IFIP WG 8.4, 8.9, TC 5 International Cross- Domain Conference, CD-ARES 2016, and Workshop on Privacy Aware Machine Learning for Health Data Science, PAML 2016, Salzburg, Austria, August 31-September 2, 2016, Proceedings, pages 19–28. Springer, 2016.
  3. [3]Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. ISSN 00063444, 14643510. URL http://www.jstor.org/stable/2334029.
  4. [4]Marijke Breuning, Jeremy Backstrom, Jeremy Brannon, Benjamin Isaak Gross, and Michael Widmeier. Reviewer fatigue? why scholars decline to review their peers' work. PS: Political Science & Politics, 48(4):595–600, 2015.
  5. [5]Andrea Castelletti, Francesca Pianosi, and Marcello Restelli. A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run. Water Resources Research, 49(6):3476–3486, 2013.
  6. [6]A Checco, L Bracciale, P Loreti, S Pinfield, and G Bianchi. Ai-assisted peer review. humanities and social sciences communications, 8 (1), 1-11, 2021.
  7. [7]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, 2024.
  8. [8]Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
  9. [9]Mike D'Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. Marg: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259, 2024. · arXiv
  10. [10]Rafael D'Andrea and James P O'Dwyer. Can editors save peer review from peer reviewers? PloS one, 12(10):e0186111, 2017.
  11. [11]Nady El-Guebaly, John Foster, Anees Bahji, and Matilda Hellman. The critical role of peer reviewers: Challenges and future steps. Nordic Studies on Alcohol and Drugs, 40(1):14–21, 2023.
  12. [12]Stephen Gallo, Lisa Thompson, Karen Schmaling, and Scott Glisson. Risk evaluation in peer review of grant applications. Environment Systems and Decisions, 38:216–229, 2018.
  13. [13]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. · arXiv
  14. [14]Cigdem Kadaifci, Erkan Isikli, and Y Ilker Topcu. Fundamental problems in the peer-review process and stakeholders' perceptions of potential suggestions for improvement. Learned Publishing, 38(1):e1637, 2025. http://www.jstor.org/stable/2334029
  15. [15]Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine Van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. A dataset of peer reviews (peerread): Collection, insights and nlp applications. arXiv preprint arXiv:1804.09635, 2018. · arXiv
  16. [16]Ingvar Karlberg. On peer review–the cornerstone of scientific publication, 2015.
  17. [17]Jacalyn Kelly, Tara Sadeghieh, and Khosrow Adeli. Peer review in scientific publications: benefits, critiques, & a survival guide. Ejifcc, 25(3):227, 2014.
  18. [18]Chhavi Kirtani, Madhav Krishan Garg, Tejash Prasad, Tanmay Singhal, Murari Mandal, and Dhruv Kumar. Revieweval: An evaluation framework for ai-generated reviews. arXiv preprint arXiv:2502.11736, 2025. · arXiv
  19. [19]Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective opti- mization. IEEE transactions on cybernetics, 51(6):3103–3114, 2020.
  20. [21]Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.
  21. [22]Ryan Liu and Nihar B Shah. Reviewergpt? an exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622, 2023. · arXiv
  22. [23]Patrice Lopez. Grobid: Combining automatic bibliographic data recognition and term extraction for scholarship publications. In International conference on theory and practice of digital libraries, pages 473–474. Springer, 2009.
  23. [24]Christopher P Morley and Sam Grammer. Now more than ever: reflections on the state and importance of peer review. PRiMER: Peer-review reports in medical education research, 5:36, 2021.
  24. [25]Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025. · arXiv
  25. [26]Syavash Nobarany and Kellogg S Booth. Use of politeness strategies in signed open peer review. Journal of the Association for Information Science and Technology, 66(5):1048–1064, 2015.
  26. [27]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/ 2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
  27. [28]Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020.
  28. [29]David B Resnik, Christina Gutierrez-Ford, and Shyamal Peddada. Perceptions of ethical problems with scientific journal peer review: an exploratory study. Science and engineering ethics, 14:305–310, 2008.
  29. [31]Abu-Ubaida Sani. Journal peer review: Meaning, types, and processes. Journal Peer Review: Meaning, 2023. https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf
  30. [32]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. · arXiv
  31. [33]Anna Severin, Michaela Strinzel, Matthias Egger, Tiago Barros, Alexander Sokolov, Julia Vil- strup Mouatt, and Stefan Müller. Relationship between journal impact factor and the thorough- ness and helpfulness of peer reviews. PLOS Biology, 21(8):1–18, 08 2023. doi: 10.1371/journal. pbio.3002238. URL https://doi.org/10.1371/journal.pbio.3002238. · DOI
  32. [34]Nihar B Shah. Challenges, experiments, and computational solutions in peer review. Communi- cations of the ACM, 65(6):76–87, 2022.
  33. [35]Minghao Shao, Abdul Basit, Ramesh Karri, and Muhammad Shafique. Survey of different large language model architectures: Trends, benchmarks, and challenges. IEEE Access, 2024.
  34. [36]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. · arXiv
  35. [37]David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017.
  36. [38]Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv. org/abs/2408.03314.
  37. [39]Cecilia Superchi, José Antonio González, Ivan Solà, Erik Cobo, Darko Hren, and Isabelle Boutron. Tools used to assess the quality of peer review reports: a methodological systematic review. BMC Medical Research Methodology, 19(1):48, Mar 2019. ISSN 1471-2288. doi: 10.1186/s12874-019-0688-x. URL https://doi.org/10.1186/s12874-019-0688-x. · DOI · Paper page
  38. [40]Cecilia Superchi, Darko Hren, David Blanco, Roser Rius, Alessandro Recchioni, Isabelle Boutron, and José Antonio González. Development of arcadia: a tool for assessing the quality of peer-review reports in biomedical research. BMJ Open, 10(6), 2020. ISSN 2044-6055. doi: 10.1136/bmjopen-2019-035604. URL https://bmjopen.bmj.com/content/10/6/ e035604. · DOI
  39. [41]Pawin Taechoyotin, Guanchao Wang, Tong Zeng, Bradley Sides, and Daniel Acuna. Mamorx: Multi-agent multi-modal scientific review generation with external knowledge. In Neurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges, 2024.
  40. [42]Misha Teplitskiy, Hao Peng, Andrea Blasco, and Karim R Lakhani. Is novel research worth doing? evidence from peer review at 49 journals. Proceedings of the National Academy of Sciences, 119(47):e2118046119, 2022.
  41. [43]Gang Wang, Qi Peng, Yanfeng Zhang, and Mingyang Zhang. What have we learned from openreview? World Wide Web, 26(2):683–708, Mar 2023. ISSN 1573-1413. doi: 10.1007/ s11280-022-01109-z. URL https://doi.org/10.1007/s11280-022-01109-z. · DOI
  42. [44]Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. Reviewrobot: Explainable paper review generation based on knowledge synthesis. arXiv preprint arXiv:2010.06119, 2020. · arXiv
  43. [45]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/ 2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf.
  44. [46]Christie Wilcox. Rude reviews are pervasive and sometimes harmful, study finds. Science, 366 (6472):1433–1433, 2019. https://doi.org/10.1371/journal.pbio.3002238 https://arxiv.org/abs/2408.03314 https://arxiv.org/abs/2408.03314 https://doi.org/10.1186/s12874-019-0688-x https://bmjopen.bmj.com/content/10/6/e035604 https://bmjopen.bmj.com/content/10/6/e035604 https://doi.org/10.1007/s11280-022-01109-z https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf · DOI
  45. [47]Weizhe Yuan, Pengfei Liu, and Graham Neubig. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171–212, 2022.
  46. [48]Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. · arXiv
  47. [49]Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39–57, 2024.
  48. [50]Youming Zhang. A novel multi-objective deep q-network: Addressing immediate and delayed rewards in multi-objective q-learning. IEEE Access, 12:144932–144949, 2024.
  49. [51]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
  50. [52]Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL http://arxiv.org/abs/2403.13372. · arXiv
  51. [53]Ruiyang Zhou, Lu Chen, and Kai Yu. Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks. In Proceedings of the 2024 Joint International Con- ference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340–9351, 2024.
  52. [54]Zhenzhen Zhuang, Jiandong Chen, Hongfeng Xu, Yuwen Jiang, and Jialiang Lin. Large lan- guage models for automated scholarly paper review: A survey. arXiv preprint arXiv:2501.10326, 2025. http://arxiv.org/abs/2403.13372 · arXiv

Cited by

Sorted by citations

Authors
Taechoyotin, P., & Acuna, D.
ORCID
Acuna, Daniel
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2505.11718
arXiv
2505.11718
OpenAlex
W4417299935
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
automated peer review, large language models, multi-objective reinforcement learning, Group Relative Policy Optimization, LLM reasoning, reward function design, low-rank adaptation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review