Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework
Open accessAbstract
Large Language Models (LLMs) have great potential to accelerate and support scholarly peer review and are increasingly used as fully automatic review generators (ARGs). However, potential biases and systematic errors may pose significant risks to scientific integrity; understanding the specific capabilities and limitations of state-of-the-art ARGs is essential. We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic. This involves evaluating the internal consistency between a paper's results, interpretations, and claims. We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions. Testing a range of ARG approaches, we find that, contrary to expectation, flaws in research logic have no significant effect on their output reviews. Based on our findings, we derive three actionable recommendations for future work and release our counterfactual dataset and evaluation framework publicly.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This reliability is consistent with prior studies reporting human agreement on similar annotation tasks.
“This reliability is consistent with prior studies reporting human agreement on similar annotation tasks (Guo et al., 2023; Dycke and Gurevych, 2025).”
From FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers (2025) · cites this paper as Dycke and Gurevych, 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 39
- Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905v1. · arXiv
- Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022. A review on language models as knowledge bases. arXiv preprint arXiv:2204.06031v1. Anthropic. 2024. Claude 3.5 sonnet. https:// www.anthropic.com/news/claude-3-5 -sonnet. Accessed: 2025-11-05. · arXiv
- Francis Bacon. 1878. Novum Organum. Clarendon press.
- Changyu Chen, Xiting Wang, Ting-En Lin, Ang Lv, Yuchuan Wu, Xin Gao, Ji-Rong Wen, Rui Yan, and Yongbin Li. 2024. Masked thought: Simply masking partial reasoning steps can improve mathematical reasoning learning of language models. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 5872-5900, Bangkok, Thailand. Associ- ation for Computational Linguistics. https:// doi.org/10.18653/v1/2024.acl-long .320 · DOI
- Subhabrata Dutta, Timo Kaufmann, Goran Glavaš, Ivan Habernal, Kristian Kersting, Frauke Kreuter, Mira Mezini, Iryna Gurevych, Eyke Hüllermeier, and Hinrich Schuetze. 2025. Problem solving through human-AI preference-based cooperation. Computational Linguistics, pages 1-36. https://doi.org /10.1162/COLI.a.19 · DOI
- Xi Fang, Weijie Xu, Fiona Anting Tan, Ziqing Hu, Jiani Zhang, Yanjun Qi, Srinivasan H. Sengamedu, and Christos Faloutsos. 2024. Large language models (LLMs) on tabular data: Prediction, generation, and understanding -a survey. Transactions on Machine Learning Research.
- Zhaolin Gao, Kianté Brantley, and Thorsten Joachims. 2024. Reviewer2: Optimizing review generation through prompt generation. arXiv preprint arXiv:2402.10886v2. · arXiv
- Yair Ori Gat, Nitay Calderon, Amir Feder, Alexander Chapanin, Amit Sharma, and Roi Reichart. 2024. Faithful explanations of black-box NLP models using LLM-generated counterfactuals. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. 2019. Explaining classifiers with causal concept effect (CaCE). arXiv preprint arXiv: 1907.07165v2. · arXiv
- Chhavi Kirtani, Madhav Krishan Garg, Tejash Prasad, Tanmay Singhal, Murari Mandal, and Dhruv Kumar. 2025. ReviewEval: An evalua- tion framework for AI-generated reviews. arXiv preprint arXiv:2502.11736v3. · arXiv
- Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A. Smith, Thamar Solorio, Jingyan Wang, Xiaodan Zhu, Anna Rogers, Nihar B. Shah, and Iryna Gurevych. 2024. What can natural language processing do for peer review? arXiv preprint arXiv:2405.06563v1. · arXiv
- Jiatao Li, Yanheng Li, Xinyu Hu, Mingqi Gao, and Xiaojun Wan. 2025. Aspect-guided multi-level perturbation analysis of large language mod- els in automated peer review. arXiv preprint arXiv:2502.12510v1. · arXiv
- Yongqi Li, Mayi Xu, Xin Miao, Shen Zhou, and Tieyun Qian. 2024. Prompting large lan- guage models for counterfactual generation: An empirical study. In Proceedings of the 2024 Joint International Conference on Computa- tional Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 13201-13221. ELRA and ICCL. https://doi.org/10.63317 /3xeiq2g96jjy · DOI
- Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel Mcfarland, and James Y. Zou. 2024a. Monitoring AI-modified content at scale: A case study on the impact of ChatGPT on AI conference peer reviews. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Ma- chine Learning Research, pages 29575-29620.
- PMLR.
- Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A. McFarland, and James Zou. 2024b. Can large language models pro- vide useful feedback on research papers? A large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196. https://doi.org/10 .1056/AIoa2400196 · DOI · Paper page
- Guan-Ting Lin, Cheng-Han Chiang, and Hung-yi Lee. 2024. Advancing large language mod- els to capture varied speaking styles and respond properly in spoken conversations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6626-6642, Bangkok, Thailand. Association for Computa- tional Linguistics. https://doi.org/10 .18653/v1/2024.acl-long.358 · DOI
- Ryan Liu and Nihar B. Shah. 2023. Review- erGPT? An exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622v1. · arXiv
- Renze Lou, Hanzi Xu, Sijia Wang, Jiangshu Du, Ryo Kamoi, Xiaoxin Lu, Jian Xie, Yuxuan Sun, Yusen Zhang, Jihyun Janice Ahn, Hongchao Fang, Zhuoyang Zou, Wenchao Ma, Xi Li, Kai Zhang, Congying Xia, Lifu Huang, and Wenpeng Yin. 2025. AAAR-1.0: Assessing AI's potential to assist research. In Forty-second International Conference on Machine Learning.
- Sheng Lu, Ilia Kuznetsov, and Iryna Gurevych. 2025. Identifying aspects in peer reviews. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 6145-6167, Suzhou, China. Association for Computational Linguistics. https://doi.org/10.18653 /v1/2025.findings-emnlp.326 · DOI
- Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, volume 36, pages 46534-46594.
- Alison McCook. 2006. Is peer review broken? Submissions are up, reviewers are overtaxed, and authors are lodging complaint after com- plaint about the process at top-tier journals. What's wrong with peer review? The Scientist, 20(2):26-35.
- Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. GPT-4 technical report. arXiv preprint arXiv:2303.08774v6. · arXiv
- Judea Pearl. 2000. Causality: Models, Reasoning and Inference, 2 edition, volume 7. Cambridge University Press.
- James Phoenix and Mike Taylor. 2024. Prompt Engineering for Generative AI. O'Reilly Media, Inc.
- Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. A systematic survey of prompt engineering in large language mod- els: Techniques and applications. arXiv preprint arXiv:2402.07927v2. · arXiv
- Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language mod- els' sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations.
- D. Sculley, Jasper Snoek, and Alex Wiltschko. 2018. Avoiding a tragedy of the commons in the peer review process. arXiv preprint arXiv:1901.06246v1. · arXiv
- Guijin Son, Jiwoo Hong, Honglu Fan, Heejeong Nam, Hyunwoo Ko, Seungwon Lim, Jinyeop Song, Jinha Choi, Gonc ¸alo Paulo, Youngjae Yu, and Stella Biderman. 2025. When AI co-scientists fail: SPOT-a benchmark for auto- mated verification of scientific research. arXiv preprint arXiv:2505.11855v1. · arXiv
- Madeleine Udell, Dov Te'eni, and Iddo Drori. 2024. AI-driven review systems: Evaluating LLMs in scalable and bias-aware academic reviews. arXiv preprint arXiv:2408.10365v1. · arXiv
- Yongjie Wang, Xiaoqi Qiu, Yu Yue, Xu Guo, Zhiwei Zeng, Yuhong Feng, and Zhiqi Shen. 2024a. A survey on natural language counter- factual generation. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2024, pages 4798-4818, Miami, Florida, USA. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024 .findings-emnlp.276 · DOI
- Ziao Wang, Xiaofeng Zhang, and Hongwei Du. 2024b. Beyond what if: Advancing counter- factual text generation with structural causal modeling. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 6522-6530.
- Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2025. CycleResearcher: Improv- ing automated research via automated review. In The Thirteenth International Conference on Learning Representations.
- Amelie Wuehrl, Dustin Wright, Roman Klinger, and Isabelle Augenstein. 2024. Understanding fine-grained distortions in reports of scien- tific findings. In Findings of the Associa- tion for Computational Linguistics: ACL 2024, pages 6175-6191, Bangkok, Thailand. Associ- ation for Computational Linguistics. https:// doi.org/10.18653/v1/2024.findings-acl .369 · DOI
- Shengwei Xu, Yuxuan Lu, Grant Schoenebeck, and Yuqing Kong. 2025. Benchmarking LLMs' judgments with no gold standard. In The Thir- teenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
- Tianmai M. Zhang and Neil F. Abernethy. 2025. Reviewing scientific papers for criti- cal problems with reasoning LLMs: Baseline approaches and automatic evaluation. arXiv preprint arXiv:2505.23824v2. · arXiv
- Curran Asso- ciates, Inc. https://doi.org/10.52202 /075280-2020 · DOI · Paper page
- Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language mod- els are human-level prompt engineers. In The Eleventh International Conference on Learning Representations.
- Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. 2025. Deepreview: Improving LLM-based paper review with human-like deep thinking process. arXiv preprint arXiv: 2503.08569v1. 12 https://api-docs.deepseek.com. 13 https://github.com/allenai/marg -reviewer/tree/master (last accessed on 18.07.25). 14 https://github.com/YuanChang98/tree -review. 15 https://huggingface.co/GitBag /Reviewer2_Mr and https://huggingface.co /GitBag/Reviewer2_Mp (last accessed on 18.07.25). 16 https://huggingface.co/WestlakeNLP /DeepReviewer-14B. · arXiv
Cited by
Sorted by citations
FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
Xi, S., Rao, V., Payan, J., & Shah, N. B. (2025) · arXiv · doi:10.48550/arxiv.2511.21843
“This reliability is consistent with prior studies reporting human agreement on similar annotation tasks (Guo et al., 2023; Dycke and Gurevych, 2025).”
PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers
Loc, N. P. P., La Viet, T. H., Khanh, T. T., Nguyen, D. A., Pham, T. A. N., Nguyen, T., Chawla, N. V., Buntine, W., Wong, K.-S., Doan, K. D., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.26730
Rethinking Social Computing Research in the Age of AI: Agency, Collaboration, and Governance
Huang, Y., Contractor, N., Peng, T., Wang, D., Shen, C., Li, M., Liu, Y., Zhang, H., & Suh, S. (2026) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3785651.3831580
- Authors
- Dycke, N., & Gurevych, I.
- Affiliations
- Technische Universität Darmstadt (DE)
- ORCID
- Gurevych, Iryna
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- Volume
- 14
- Pages
- 465-488
- ISSN
- 2307-387X (ISSN-L)
- arXiv
- 2508.21422
- OpenAlex
- W4415990061
- Cited by
- 1
- References
- 0
- Citations by year
- 2026: 1
- FWCI
- 8.51
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- TUbilio (Technical University of Darmstadt) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- Technischen Universität Darmstadt (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- European Commission
- Bundesministerium für Bildung und Forschung
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Keywords
- large language models, research logic, counterfactual evaluation, scholarly peer review, scientific integrity, evaluation dataset, large language model bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
