Can We Automate Scientific Reviewing?
Open accessAbstract
The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications. At the same time, the review of each paper is a laborious process that must be carried out by subject matter experts. Thus, providing high-quality reviews of this growing number of papers is a significant challenge. In this work, we ask the question “can we automate scientific reviewing? ”, discussing the possibility of using natural language processing (NLP) models to generate peer reviews for scientific papers. Because it is non-trivial to define what a “good” review is in the first place, we first discuss possible evaluation metrics that could be used to judge success in this task. We then focus on the machine learning domain and collect a dataset of papers in the domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers as input and generate reviews as output. Comprehensive experimental results on the test set show that while system-generated reviews are comprehensive, touching upon more aspects of the paper than human-written reviews, the generated texts are less constructive and less factual than human-written reviews for all aspects except the explanation of the core ideas of the papers, which are largely factually correct. Given these results, we pose eight challenges in the pursuit of a good review generation system together with potential solutions, which, hopefully, will inspire more future research in this direction. We make relevant resource publicly available for use by future research: https://github. com/neulab/ReviewAdvisor. In addition, while our conclusion is that the technology is not yet ready for use in high-stakes review settings we provide a system demo, ReviewAdvisor (http://review.nlpedia.ai/), showing the current capabilities and failings of state-of-the-art NLP models at this task (see demo screenshot in A.2). A review of this paper written by the system proposed in this paper can be found in A.1.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Numerous samples of peer reviews and representative cross-sections of scores are critical to minimize bias and avoid high-frequency pattern generation in machine-learning-based review generation.
“numerous samples of peer reviews and representative cross-sections of scores are critical to minimize any bias possibly existing in the documents and avoid high-frequency pattern generation.”
From The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics (2023) · cites this paper as [Yuan et al., 2022]
Yuan et al. use an extract-then-generate method that reduces text length to within 1,024 tokens.
“Yuan et al. (2022) use an extract-then-generate method that reduces the text length to within the processing limits of 1,024 tokens.”
From MOPRD: A multidisciplinary open peer review dataset (2022) · cites this paper as Yuan et al., 2022
LLMs are increasingly used to assist with manuscript preparation, review drafting, and editorial support tasks.
“LLMs are increasingly used to assist with manuscript preparation, review drafting, and editorial support tasks (Liang et al., 2024;Yuan et al., 2022;Sadeghian et al., 2026).”
From Peerify: Benchmarking Peer-Review Claim Verification (2026) · cites this paper as Yuan et al., 2022
Existing work on LLM-assisted peer review focuses on review generation or quality assessment.
“Existing work on LLMassisted peer review focuses on review gen-eration or quality assessment (Yuan et al., 2022;Liang et al., 2024;Hua et al., 2019;Rogers and Augenstein, 2020;Ebrahimi et al., 2025a).”
From Peerify: Benchmarking Peer-Review Claim Verification (2026) · cites this paper as Yuan et al., 2022
Prior work on LLMs for peer review primarily aims to produce, evaluate, or characterize reviews rather than to determine whether specific claims made within a review are supported by evidence contained in the reviewed manuscript.
“However, their primary objective is to produce, evaluate, or characterize reviews rather than to determine whether specific claims made within a review are supported by evidence contained in the reviewed manuscript.”
From Peerify: Benchmarking Peer-Review Claim Verification (2026) · cites this paper as Yuan et al., 2022
Early exploratory studies tested whether GPT-4 could detect inserted errors and produce structured first-pass feedback.
“Early exploratory studies tested whether GPT-4 could detect inserted errors and produce structured first-pass feedback”
From On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists (2026) · cites this paper as Yuan et al., 2022
Automatic review generation methods neglect the fine-grained decision-making factors in humans and LLMs that shape complex assessment.
“While these methods offer valuable insights into downstream performance in peer review, they neglect the fine-grained decision-making factors in humans and LLMs that shape the complex assessment.”
From STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond (2025) · cites this paper as Yuan et al., 2022
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Bootstrap methods: another look at the jackknife
Efron, B. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344552 · Cited by 17,602 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · see citing works
Bornmann, L., & Mutz, R. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23329 · Cited by 1,576 · see citing works
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207 · see citing works
A Deep Reinforced Model for Abstractive Summarization
Paulus, R., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1705.04304 · Cited by 1,182 · see citing works
Rubinstein, R. Y., & Kroese, D. P. (2004) · Medical Entomology and Zoology · Cited by 1,074 · see citing works
Peer Review: A Flawed Process at the Heart of Science and Journals
Smith, R. (2006) · Journal of the Royal Society of Medicine · doi:10.1177/014107680609900414 · Cited by 880 · see citing works
Nallapati, R. M., Zhai, F., & Zhou, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04230 · Cited by 742 · see citing works
Luan, Y., He, L., Mari Ostendorf, E., & Hajishirzi, H. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1360 · Cited by 714 · see citing works
Bottom-Up Abstractive Summarization
Gehrmann, S., Deng, Y., & Rush, A. M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1443 · Cited by 679 · see citing works
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., & Goharian, N. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2097 · Cited by 658 · see citing works
Reviewer bias in single- versus double-blind peer review
Tomkins, A., Zhang, M., & Heavlin, W. D. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1707323114 · Cited by 605 · see citing works
Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
Chen, Y.-C., & Bansal, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1063 · Cited by 601 · see citing works
Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics
Bellis, N. D. (2009) · IRIS UNIMORE (University of Modena and Reggio Emilia) · Cited by 548 · see citing works
Evaluating Gender Bias in Machine Translation
Stanovsky, G., Smith, N. A., & Zettlemoyer, L. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1164 · Cited by 492 · see citing works
S2ORC: The Semantic Scholar Open Research Corpus
Lo, K., Wang, L. L., Neumann, M., Kinney, R., & Weld, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.447 · Cited by 457 · see citing works
Fact or Fiction: Verifying Scientific Claims
Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., & Hajishirzi, H. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.609 · Cited by 382 · see citing works
Effects of Editorial Peer Review: A Systematic Review
Jefferson, T., Alderson, P., Wager, E., & Davidoff, F. (2002) · JAMA · doi:10.1001/jama.287.21.2784 · Cited by 347 · see citing works
On Extractive and Abstractive Neural Document Summarization with Transformer Language ModelsPreprint
Subramanian, S., Li, R., Pilault, J., & Pal, C. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.1909.03186 · Cited by 253 · see citing works
Scientific paper summarization using citation summary networks
Qazvinian, V., & Radev, D. (2008) · International Conference on Computational Linguistics · doi:10.3115/1599081.1599168 · Cited by 243 · see citing works
Measuring the quality of editorial peer review.
Jefferson, T. (2002) · JAMA · doi:10.1001/jama.287.21.2786 · Cited by 230 · see citing works
Angelidis, S., & Lapata, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1403 · Cited by 192 · see citing works
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
Kang, D., Ammar, W., Dalvi, B., Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1149 · Cited by 171 · see citing works
SciREX: A Challenge Dataset for Document-Level Information Extraction
Jain, S., van Zuylen, M., Hajishirzi, H., & Beltagy, I. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.670 · Cited by 159 · see citing works
Extractive Summarization of Long Documents by Combining Global and Local Context
Xiao, W., & Carenini, G. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1298 · Cited by 158 · see citing works
Using citations to generate surveys of scientific paradigms
Mohammad, S. M., Dorr, B. J., Egan, M., et al. (2009) · doi:10.3115/1620754.1620839 · Cited by 126 · see citing works
Gender Bias in Coreference Resolution: Evaluation and Debiasing MethodsPreprint
Zhao, J., Wang, T., Yatskar, M., Ordóñez, V., & Chang, K.-W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.06876 · Cited by 85 · see citing works
Literature Dynamics: Studies on Growth, Diffusion, and Epidemics
Tabah, A. N. (1999) · Cited by 77 · see citing works
Sentiment Analysis of Peer Review Texts for Scholarly Papers
Wang, K., & Wan, X. (2018) · doi:10.1145/3209978.3210056 · Cited by 72 · see citing works
Hou, Y., Jochim, C., Gleize, M., Bonin, F., & Ganguly, D. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1513 · Cited by 67 · see citing works
Unsupervised Query-Focused Multi-Document Summarization using the Cross Entropy Method
Feigenblat, G., Roitman, H., Boni, O., et al. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080690 · Cited by 63 · see citing works
Automatically Predicting Peer-Review Helpfulness
Xiong, W., & Litman, D. J. (2011) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 59 · see citing works
On Testing for Biases in Peer ReviewPreprint
Stelmakh, I., Shah, N. B., & Singh, A. (2019) · Neural Information Processing Systems · doi:10.48550/arxiv.1912.13188 · Cited by 55 · see citing works
Inducing Document Structure for Aspect-based Summarization
Frermann, L., & Klementiev, A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1630 · Cited by 52 · see citing works
The arbitrariness of reviews, and advice for school administrators
Langford, J., & Guzdial, M. (2015) · Communications of the ACM · doi:10.1145/2732417 · Cited by 49 · see citing works
Does My Rebuttal Matter? Insights from a Major NLP Conference
Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., & Miyao, Y. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1129 · Cited by 48 · see citing works
Yasunaga, M., Kasai, J., Zhang, R., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01716 · Cited by 42 · see citing works
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · arXiv · doi:10.48550/arxiv.1903.10676 · Cited by 41 · see citing works
Automatic Generation of Citation Texts in Scholarly Papers: A Pilot Study
Xing, X., Fan, X., & Wan, X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.550 · Cited by 40 · see citing works
A decision support tool using Order Weighted Averaging for conference review assignment
Nguyen, J., Sánchez-Hernández, G., Agell, N., et al. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.09.020 · Cited by 34 · see citing works
Uncovering Latent Biases in Text: Method and Application to Peer Review
Manzoor, E., & Shah, N. B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i6.16608 · Cited by 28 · see citing works
Integrating the Trend of Research Interest for Reviewer Assignment
Jin, J., Geng, Q., Zhao, Q., et al. (2017) · doi:10.1145/3041021.3053053 · Cited by 24 · see citing works
A System for Summarizing Scientific Topics Starting from Keywords
Jha, R., Abu-Jbara, A., & Radev, D. (2013) · Cited by 21 · see citing works
Deep Paper GestaltPreprint
Huang, J.-B. (2018) · arXiv.org · doi:10.48550/arxiv.1812.08775 · Cited by 20 · see citing works
Surveyor: A System for Generating Coherent Survey Articles for Scientific Topics
Jha, R., Coke, R., & Radev, D. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9495 · Cited by 19 · see citing works
Qiao, F., Xu, L., & Han, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-02934-0_7 · Cited by 13 · see citing works
GSum: A General Framework for Guided Neural Abstractive SummarizationPreprint
Dou, Z.-Y., Liu, P., Hayashi, H., Jiang, Z., & Neubig, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08014 · Cited by 13 · see citing works
Self-citationContent Models for Survey Generation: A Factoid-Based Evaluation
Jha, R. K., Finegan‐Dollak, C., King, B., et al. (2015) · doi:10.3115/v1/p15-1043 · Cited by 12 · see citing works
Scientific Article Summarization Using Citation-Context and Article's Discourse StructurePreprint
Cohan, A., & Goharian, N. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.06619 · Cited by 3 · see citing works
Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments
Jecmen, S., Zhang, H., Liu, R., Shah, N. B., Conitzer, V., & Fang, F. (2020) · arXiv · doi:10.48550/arxiv.2006.16437
Flagged first, then by citations
Cited by
Sorted by citations
MetaWriter: Exploring the Potential and Perils of AI Writing Support in Scientific Peer Review
Sun, L., Tao, S., Hu, J., & Dow, S. P. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637371 · Cited by 37 · see citing works
Large language models for automated scholarly paper review: A survey
Zhuang, Z., Chen, J., Xu, H., Jiang, Y., & Lin, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103332 · Cited by 34 · see citing works
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23 · see citing works
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Narayanan Venkit, P., Zhang, N., Srinath, M., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.292 · Cited by 23 · see citing works
Large language models in peer review: challenges and opportunities
Sun, Z. (2025) · Scientometrics · doi:10.1007/s11192-025-05440-w · Cited by 15 · see citing works
MOPRD: A multidisciplinary open peer review dataset
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2022) · arXiv · doi:10.48550/arxiv.2212.04972 · Cited by 11 · see citing works
“The authors train proposition segmentation and classification models with this dataset to analyze the patterns and evaluate the utilities of argumentative propositions.”
BackgroundMethodInfluentialOpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews
Idahl, M., & Ahmadi, Z. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations) · doi:10.18653/v1/2025.naacl-demo.44 · Cited by 10 · see citing works
L-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2307.11088 · Cited by 6 · see citing works
“We use the processed PDF files from Yuan et al. (2021).”
MethodLin, T.-L., Chen, W.-C., Hsiao, T.-F., Liu, H.-I., Yeh, Y.-H., Chan, Y.-K., Lien, W.-S., Kuo, P.-Y., Yu, P. S., & Shuai, H.-H. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.259 · Cited by 6
An Analysis of Tasks and Datasets in Peer Reviewing
Staudinger, M., Kusa, W., Piroi, F., & Hanbury, A. (2024) · Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024) · doi:10.18653/v1/2024.sdp-1.24 · Cited by 6 · see citing works
Chen, S., Brumby, D., & Cox, A. (2025) · Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work · doi:10.1145/3729176.3729196 · Cited by 5 · see citing works
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Venkit, P. N., Zhang, N., Srinath, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16253 · Cited by 2 · see citing works
Chang, Y., Li, Z., Zhang, H., Kong, Y., Wu, Y., So, H. K.-H., Guo, Z., Zhu, L., & Wong, N. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.790 · Cited by 2 · see citing works
Szumega, J., Bougueroua, L., Gkotse, B., Jouvelot, P., & Ravotti, F. (2023) · arXiv · doi:10.48550/arxiv.2312.04576 · Cited by 1 · see citing works
“Otherwise, various weaknesses related to artificial text would be visible in the generated reviews, rendering them useless ([Yuan et al., 2022]).”
BackgroundSciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
Xiang, Y., Yan, H., Ouyang, S., Gui, L., & He, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.00255 · Cited by 1 · see citing works
“For instance, Si et al. (2025); Gu & Krenn (2025) demonstrated how LLMs can be prompted to generate novel research ideas, while Yuan et al. (2022); Du et al. (2024) explored their use in producing literature reviews for idea evaluation.”
Kim, S., Yoon, D., Gashteovski, K., Suk, J., Baek, J., Aggarwal, P., Wu, I., Zaverkin, V., Petkoski, S., Schrider, D. R., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.20668 · Cited by 1 · see citing works
Self-citationCan AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Wu, S., Jiang, O., Zhao, Y., Hu, T., Ma, Y., Zhang, K., Patwardhan, M., & Cohan, A. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1504 · Cited by 1 · see citing works
STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond
Dycke, N., Zečević, M., Kuznetsov, I., Suess, B., Kersting, K., & Gurevych, I. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1107 · Cited by 1 · see citing works
Dycke, N., & Gurevych, I. (2025) · arXiv · doi:10.48550/arxiv.2508.21422 · Cited by 1
The Good, the Bad and the Constructive: Automatically Measuring Peer Review’s Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1476 · Cited by 1 · see citing works
OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews
Idahl, M., & Ahmadi, Z. (2024) · arXiv · doi:10.48550/arxiv.2412.11948 · Cited by 1 · see citing works
Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions
Zhang, D., Bao, Z., Du, S., Zhao, Z., Zhang, K., Bao, D., & Yang, Y. (2025) · arXiv · doi:10.48550/arxiv.2505.07920
“A few other works [Kang et al., 2018, Shen et al., 2021, Yuan et al., 2022, Dycke et al., 2022, D’Arcy et al., 2023, Gao et al., 2024] incorporate reviews from additional conferences like NeurIPS and ACL.”
CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
Deng, H., Ke, X., Li, Y., Hu, R., Huang, D., Wong, D. F., Wang, Y., Liu, X., & Zhang, M. (2026) · arXiv · doi:10.48550/arxiv.2605.07905
Peerify: Benchmarking Peer-Review Claim Verification
Daghighfarsoodeh, A., Ebrahimi, S., Ghorbanpour, A., Sadeghian, S., Cheraghi, R., Arabzadeh, N., & Bagheri, E. (2026) · arXiv · doi:10.48550/arxiv.2609.25046
RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation
Wu, S., Ma, Y., Zhao, Y., Hu, T., Jiang, O., Patwardhan, M., & Cohan, A. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1696
How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
Li, M., Wang, C., Li, X., Zeng, X., Li, D., Shi, P., Zhou, D., & Zhou, T. (2026) · arXiv · doi:10.48550/arxiv.2608.08975
Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review
Yuan, S., Zhang, X., Ni, W., Xiao, T., Di, S., Zhu, J., Jiang, Z., Tan, R., Chen, L., & Ren, K. (2026) · arXiv · doi:10.48550/arxiv.2609.05947
FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
Xi, S., Rao, V., Payan, J., & Shah, N. B. (2025) · arXiv · doi:10.48550/arxiv.2511.21843
- Authors
- Yuan, W., Liu, P., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2022-09-29
- Volume
- 75
- Pages
- 171-212
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W3128455716
- Cited by
- 96
- References
- 57
- Citations by year
- 2026: 13 · 2025: 22 · 2024: 21 · 2023: 20 · 2022: 11 · 2021: 9
- FWCI
- 6.09
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Artificial Intelligence Research (publishedVersion, cc-by)
- arXiv (Cornell University) (acceptedVersion)
- arXiv (Cornell University) (public-domain)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- automated peer review, review generation, scholarly publishing, summarization, constructive feedback, factual accuracy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
