The Good, the Bad and the Constructive: Automatically Measuring Peer Review’s Utility for Authors
Open accessAbstract
Providing constructive feedback to paper authors is a core component of peer review.With reviewers increasingly having less time to perform reviews, automated support systems are required to ensure high reviewing quality, thus making the feedback in reviews useful for authors.To this end, we identify four key aspects of review comments (individual points in weakness sections of reviews) that drive the utility for authors: Actionability, Grounding & Specificity, Verifiability, and Helpfulness.To enable evaluation and development of models assessing review comments, we introduce the RevUtil dataset.We collect 1,430 human-labeled review comments and scale our data with 10k synthetically labeled comments for training purposes.The synthetic data additionally contains rationales, i.e., explanations for the aspect score of a review comment.Employing the RevUtil dataset, we benchmark fine-tuned models for assessing review comments on these aspects and generating rationales.Our experiments demonstrate that these fine-tuned models achieve agreement levels with humans comparable to, and in some cases exceeding, those of powerful closed models like GPT-4o.Our analysis further reveals that machinegenerated reviews generally underperform human reviews on our four aspects. 1
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The prompt design for paper-level and category-level metrics of CoCoReviewBench with LLM-as-a-Judge largely follows Garg et al. (2025) and Sadallah et al. (2025).
“The prompt design largely follows Garg et al. (2025) and Sadallah et al. (2025).”
From CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers (2026) · cites this paper as Sadallah et al. (2025)
Smaller open weights LLMs seem to be powerful enough for the task of LLM review support, allowing more security than when using cloud-based versions.
“Smaller open weights LLMs seem to be powerful enough for this task (Sadallah et al., 2025), allowing more security than when using cloud-based versions.”
From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Sadallah et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 33
- Lutz Bornmann, Rüdiger Mutz, and Robin Haunschild. 2020. Growth rates of modern science: A latent piecewise growth curve approach to model publi- cation numbers from established and new literature databases. CoRR, abs/2012.07675. · arXiv
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language Models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Aoife Cahill, Leon Derczynski, and Kokil Jaidka. 2024. ACL Policy on Publication Ethics. https: //www.aclweb.org/adminwiki/index.php/ACL_ Policy_on_Publication_Ethics#Guidelines_ for_Generative_Assistance_in_Peer_Review. [Online; accessed 04-May-2025].
- Nitay Calderon, Roi Reichart, and Rotem Dror. 2025. The alternative annotator test for LLM-as-a-Judge: How to statistically justify replacing human anno- tators with LLMs. In Proceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vi- enna, Austria, July 27 -August 1, 2025, pages 16051- 16081. Association for Computational Linguistics.
- Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or par- tial credit. Psychological Bulletin, 70:213-220.
- CVPR. 2025. CVPR 2025 Reviewer Guide- lines . https://cvpr.thecvf.com/Conferences/ 2025/ReviewerGuidelines. [Online; accessed 02- July-2025].
- Tri Dao. 2024. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient fine- tuning of quantized llms. In Advances in Neural Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
- Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Ku- mar, Ramaneswaran S., Deepali Aneja, Zeyu Jin, Ramani Duraiswami, and Dinesh Manocha. 2024. A closer look at the limitations of instruction tuning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
- ICCV. 2025. ICCV 2025 Reviewer Guide- lines. https://iccv.thecvf.com/Conferences/ 2025/ReviewerGuidelines. [Online; accessed 02- July-2025].
- Tom Jefferson, Philip Alderson, Elizabeth Wager, and Frank Davidoff. 2002. Effects of editorial peer re- view: A systematic review. Jama, 287(21):2784- 2786.
- Dhiraj D. Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. 2019. A study of BFLOAT16 for deep learning train- ing. CoRR, abs/1905.12322. · arXiv
- Aditya Khosla, Derek Hoiem, and Serge J. Belongie. 2012. Analysis of Reviews for CVPR 2012.
- Seungone Kim, Jamin Shin, Yejin Choi, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. Prometheus: Inducing fine- grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Nino Künzli, Anke Berger, Katarzyna Czabanowska, Raquel Lucas, Andrea Madarasova Geckova, Sarah Mantwill, and Olaf von Dem Knesebeck. 2022. «I Do Not Have Time»-is this the end of peer review in public health sciences? Public health reviews, 43:1605407. · DOI
- Sihang Li, Jin Huang, Jiaxi Zhuang, Yaorui Shi, Xi- aochen Cai, Mingjun Xu, Xiang Wang, Linfeng Zhang, Guolin Ke, and Hengxing Cai. 2025. SciL- itLLM: How to adapt LLMs for scientific literature 28991 understanding. In The Thirteenth International Con- ference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
- Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Hao- tian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, and James Y. Zou. 2024a. Monitoring AI-Modified content at scale: A case study on the impact of Chat- GPT on AI conference peer reviews. In Forty-first In- ternational Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenRe- view.net.
- Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A. McFarland, and James Zou. 2024b. Can Large Lan- guage Models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196. · DOI · Paper page
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Nature. 2025. Peer Review. https://www.nature. com/nature-portfolio/editorial-policies/ peer-review. [Online; accessed 02-July-2025].
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 -December 9, 2022.
- PLOS. 2025. How to Write a Peer Re- view. https://plos.org/resource/ how-to-write-a-peer-review/. [Online; accessed 02-July-2025].
- Tony Ross-Hellauer. 2017. What is open peer review? A systematic review. F1000 Research, 6:588.
- Anna Severin, Michaela Strinzel, Matthias Egger, Tiago Barros, Alexander Sokolov, Julia Vilstrup Mouatt, and Stefan Müller. 2022. Journal impact factor and peer review thoroughness and helpful- ness: A supervised machine learning study. CoRR, abs/2207.09821. · DOI
- Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023. Large Language Models can be easily distracted by irrelevant context. In Interna- tional Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 31210-31227.
- PMLR.
- Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024a. HelpSteer2: Open-source dataset for train- ing top-performing reward models. In Advances in Neural Information Processing Systems, volume 37, pages 1474-1501.
- Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. 2024b. HelpSteer: Multi- attribute helpfulness dataset for SteerLM. In Pro- ceedings of the 2024 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3371-3384, Mexico City, Mexico. Association for Computational Linguistics. · DOI
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought prompt- ing elicits reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 -December 9, 2022.
- Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeon- bin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2024. FLASK: fine-grained language model evaluation based on alignment skill sets. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Sys- tems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Or- leans, LA, USA, December 10 -16, 2023.
Cited by
Sorted by citations
Sandström, U., & Thelwall, M. (2026) · Journal of Data and Information Science · doi:10.1515/jdis-2026-0048 · Cited by 1 · see citing works
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Wu, S., Jiang, O., Zhao, Y., Hu, T., Ma, Y., Zhang, K., Patwardhan, M., & Cohan, A. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1504 · Cited by 1 · see citing works
Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers
Yang, Y., Thelwall, M., & He, G. (2026) · arXiv · doi:10.48550/arxiv.2609.01895
CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
Deng, H., Ke, X., Li, Y., Hu, R., Huang, D., Wong, D. F., Wang, Y., Liu, X., & Zhang, M. (2026) · arXiv · doi:10.48550/arxiv.2605.07905
RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation
Wu, S., Ma, Y., Zhao, Y., Hu, T., Jiang, O., Patwardhan, M., & Cohan, A. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1696
PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing
Żurawicki, K., Farganus, J., Gaweł, A., Bystroński, M., & Kajdanowicz, T. J. (2026) · arXiv · doi:10.48550/arxiv.2605.29815
Artificial Intelligence in Research and Academic Publication
Zhang, Y., & Xie, Q. (2026) · IEEE Intelligent Systems · doi:10.1109/mis.2026.3726702
PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality
Chen, Z., Yang, Z., Ma, Y., Backes, M., & Zhang, Y. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1170
- Authors
- Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T.
- Affiliations
- Mohamed bin Zayed University of Artificial Intelligence (AE)
- Technische Universität Darmstadt (DE)
- Venue
- Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
- Published
- 2025
- Pages
- 28979-29009
- OpenAlex
- W4416035330
- Cited by
- 1
- References
- 0
- Citations by year
- 2026: 1
- FWCI
- 8.52
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- TUbilio (Technical University of Darmstadt) (acceptedVersion)
- Technischen Universität Darmstadt (submittedVersion)
- Funders
- European Commission
- Deutsche Forschungsgemeinschaft
- HORIZON EUROPE European Research Council (101054961)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- peer review, reviewer comments, author feedback, synthetic data generation, explanation generation, actionability, helpfulness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
