Automatic Analysis of Substantiation in Scientific Peer Reviews
Open accessAbstract
With the increasing amount of problematic peer reviews in top AI conferences, the community is urgently in need of automatic quality control measures. In this paper, we restrict our attention to substantiation -- one popular quality aspect indicating whether the claims in a review are sufficiently supported by evidence -- and provide a solution automatizing this evaluation process. To achieve this goal, we first formulate the problem as claim-evidence pair extraction in scientific peer reviews, and collect SubstanReview, the first annotated dataset for this task. SubstanReview consists of 550 reviews from NLP conferences annotated by domain experts. On the basis of this dataset, we train an argument mining system to automatically analyze the level of substantiation in peer reviews. We also perform data analysis on the SubstanReview dataset to obtain meaningful insights on peer reviewing quality in NLP conferences over recent years.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Peer reviews contain argumentative elements, such as evaluations, requests, facts, references, quotations, and claim-evidence relations.
“Substantiation has long been treated as a component of review quality, and computational studies show that peer reviews contain argumentative elements, such as evaluations, requests, facts, references, quotations, and claim-evidence relations (Guo et al., 2023;Hua et al., 2019;van Rooyen et al., 1999).”
From Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers (2026) · cites this paper as Guo et al., 2023
The Introduction and the annotation guidelines describe peer reviews as containing evaluations and claim–evidence relations, and the Introduction treats substantiation as a review-quality criterion. The paper does not establish the full list of argumentative elements attributed to the citations: it does not identify requests, references, or quotations as such elements. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThis reliability is consistent with prior studies reporting human agreement on similar annotation tasks.
“This reliability is consistent with prior studies reporting human agreement on similar annotation tasks (Guo et al., 2023; Dycke and Gurevych, 2025).”
From FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers (2025) · cites this paper as Guo et al., 2023
Peer reviews contain argumentative propositions and claim-evidence relations.
“It also aligns with computational studies showing that peer reviews contain argumentative propositions and claim-evidence relations (Fromm et al., 2021;Guo et al., 2023;Hua et al., 2019).”
From Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers (2026) · cites this paper as Guo et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Argument Mining for Understanding Peer Reviews
Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1219 · Cited by 86 · cited 3× · see citing works
“Hua et al. (2019) introduced the AMPERE dataset containing 400 reviews annotated for proposition segmentation and proposition classification (evaluation, request, fact, reference, quote, or non argument) and trained neural models to perform the two tasks.”
BackgroundMethod
Flagged first, then by citations
Research works printed but not linked: 34
- Francesco Barbieri, Jose Camacho-Collados, Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644-1650, Online. Association for Computational Linguistics. · DOI
- Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguistics. · DOI · Paper page
- Alessandro Checco, Lorenzo Bracciale, Pierpaolo Loreti, Stephen Pinfield, and Giuseppe Bianchi. 2021. Ai-assisted peer review. Humanities and social sci- ences communications, 8(1):1-11.
- Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. 2020. APE: Argument pair extraction from peer review and rebuttal via multi-task learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7000-7011, Online. Association for Computa- tional Linguistics. · DOI · Paper page
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI · Paper page
- Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2022. Yes-yes-yes: Proactive data collection for ACL rolling review and beyond. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, pages 300-318, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023. NLPeer: A unified resource for the computational study of peer review. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5049- 5073, Toronto, Canada. Association for Computa- tional Linguistics. · DOI · Paper page
- Liat Ein-Dor, Eyal Shnarch, Lena Dankin, Alon Halfon, Benjamin Sznajder, Ariel Gera, Carlos Alzate, Mar- tin Gleize, Leshem Choshen, Yufang Hou, Yonatan Bilu, Ranit Aharonov, and Noam Slonim. 2020. Cor- pus wide argument mining-a working solution. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 34(05):7683-7691. · DOI
- James B Freeman. 2011a. Argument Structure:: Repre- sentation and Theory, volume 18. Springer Science & Business Media.
- James B Freeman. 2011b. Dialectics and the macrostructure of arguments. In Dialectics and the Macrostructure of Arguments. De Gruyter Mouton.
- Michael Fromm, Evgeniy Faerman, Max Berrendorf, Siddharth Bhargava, Ruoxia Qi, Yao Zhang, Lukas Dennert, Sophia Selle, Yang Mao, and Thomas Seidl. 2021. Argument mining driven analysis of peer- reviews. Proceedings of the AAAI Conference on Artificial Intelligence, 35(6):4758-4766. · DOI · Paper page
- Tirthankar Ghosal, Sandeep Kumar, Prabhat Kumar Bharti, and Asif Ekbal. 2022. Peer review analyze: A novel benchmark resource for computational analysis of peer reviews. Plos one, 17(1):e0259238.
- Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span- BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77. · DOI · Paper page
- Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long Papers), pages 1647-1661, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI · Paper page
- Neha Kennard, Tim O'Gorman, Rajarshi Das, Ak- shay Sharma, Chhandak Bagchi, Matthew Clinton, Pranay Kumar Yelugam, Hamed Zamani, and An- drew McCallum. 2022. DISAPERE: A dataset for discourse structure in peer review discussions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1234-1249, Seattle, United States. Association for Computational Linguistics. · DOI · Paper page
- Klaus Krippendorff, Yann Mathet, Stéphane Bouvry, and Antoine Widlöcher. 2016. On the reliability of unitizing textual continua: Further developments. Quality & Quantity, 50(6):2347-2364.
- John Lawrence and Chris Reed. 2019. Argument min- ing: A survey. Computational Linguistics, 45(4):765- 818. · DOI
- Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35.
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. · arXiv
- Henrique Lopes Cardoso, Rui Sousa-Silva, Paula Car- valho, and Bruno Martins. 2023. Argumentation models and their use in corpus annotation: Practice, prospects, and challenges. Natural Language Engi- neering, page 1-38. · DOI
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
- Simon Price and Peter A Flach. 2017. Computational support for academic peer review: A perspective from artificial intelligence. Communications of the ACM, 60(3):70-79.
- Lance Ramshaw and Mitch Marcus. 1995. Text chunk- ing using transformation-based learning. In Third Workshop on Very Large Corpora.
- Gil Rocha, Luís Trigo, Henrique Lopes Cardoso, Rui Sousa-Silva, Paula Carvalho, Bruno Martins, and Miguel Won. 2022. Annotating arguments in a cor- pus of opinion articles. In Proceedings of the Thir- teenth Language Resources and Evaluation Confer- ence, pages 1890-1899, Marseille, France. European Language Resources Association.
- Alessio Russo. 2021. Some ethical issues in the re- view process of machine learning conferences. arXiv preprint arXiv:2106.00810. · arXiv
- Anna Severin, Michaela Strinzel, Matthias Egger, Tiago Barros, Alexander Sokolov, Julia Vilstrup Mouatt, and Stefan Müller. 2022. Journal impact factor and peer review thoroughness and helpfulness: A su- pervised machine learning study. arXiv preprint arXiv:2207.09821. · arXiv
- Stephen E Toulmin. 2003. The uses of argument. Cam- bridge university press.
- David Tran, Alex Valtchanov, Keshav Ganapathy, Ray- mond Feng, Eric Slud, Micah Goldblum, and Tom Goldstein. 2020. An open review of openreview: A critical analysis of the machine learning conference review process. arXiv preprint arXiv:2010.05137. · arXiv
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30.
- Rajeev Verma, Rajarshi Roychoudhury, and Tirthankar Ghosal. 2022. The lack of theory is painful: Mod- eling harshness in peer review comments. In Pro- ceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 925-935, Online only. Association for Computational Linguistics.
- Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2):165-210.
- Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171-212.
- Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
- Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206. · arXiv
Cited by
Sorted by citations
Large language models for automated scholarly paper review: A survey
Zhuang, Z., Chen, J., Xu, H., Jiang, Y., & Lin, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103332 · Cited by 34 · see citing works
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Venkit, P. N., Zhang, N., Srinath, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16253 · Cited by 2 · see citing works
“…such as argument (Kennard et al., 2022; Hua et al., 2019; Yuan et al., 2022; Cheng et al., 2020; Ruggeri et al., 2023), politeness (Bharti et al., 2023), uncertainty detection (Ghosal et al., 2022b), contradictions in review pairs (Kumar et al., 2023), and substantiation (Guo et al., 2023).”
BackgroundFLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
Xi, S., Rao, V., Payan, J., & Shah, N. B. (2025) · arXiv · doi:10.48550/arxiv.2511.21843
“This reliability is consistent with prior studies reporting human agreement on similar annotation tasks (Guo et al., 2023; Dycke and Gurevych, 2025).”
Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers
Yang, Y., Thelwall, M., & He, G. (2026) · arXiv · doi:10.48550/arxiv.2609.01895
“…has long been treated as a component of review quality, and computational studies show that peer reviews contain argumentative elements, such as evaluations, requests, facts, references, quotations, and claim-evidence relations (Guo et al., 2023; Hua et al., 2019; van Rooyen et al., 1999).”
The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · arXiv · doi:10.48550/arxiv.2509.04484
“Prior work analyses peer reviews on different aspects, including harshness (Verma et al., 2022), thoroughness and helpfulness (Severin et al., 2022), completeness (Yuan et al., 2022), substantiation (Guo et al., 2023), and politeness (Bharti et al., 2024).”
- Authors
- Guo, Y., Shang, G., Rennard, V., Vazirgiannis, M., & Clavel, C.
- Affiliations
- École Polytechnique (FR)
- Linagora (France) (FR)
- Télécom Paris (FR)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 10198-10216
- arXiv
- 2311.11967
- OpenAlex
- W4389520304
- Cited by
- 5
- References
- 1
- Citations by year
- 2026: 2 · 2025: 3
- FWCI
- 0.89
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- HAL (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Agence Nationale de la Recherche
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- peer review quality, substantiation, argument mining, automated quality control
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
