CoChat
Sign Up
Cited by 1Academic Writing and PublishingTopic ModelingPublishing and Scholarly Communication

The DOI matches its Crossref record.

Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework

Open access
Dycke, Nils, Gurevych, Iryna·2026·Transactions of the Association for Computational Linguistics
DOI 10.1162/tacl.a.642
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have great potential to accelerate and support scholarly peer review and are increasingly used as fully automatic review generators (ARGs). However, potential biases and systematic errors may pose significant risks to scientific integrity; understanding the specific capabilities and limitations of state-of-the-art ARGs is essential. We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic. This involves evaluating the internal consistency between a paper’s results, interpretations, and claims. We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions. Testing a range of ARG approaches, we find that, contrary to expectation, flaws in research logic have no significant effect on their output reviews. Based on our findings, we derive three actionable recommendations for future work and release our counterfactual dataset and evaluation framework publicly.1

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/tacl.a.642

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Dycke and Gurevych (2026) deliberately disrupts the relationships among results, interpretations, and claims, using meaning-preserving edits as controls, to test whether automatic reviewers detect the resulting substantive defects.

    “Dycke and Gurevych (2026) deliberately disrupts the relationships among results, interpretations, and claims, using meaning-preserving edits as controls, to test whether automatic reviewers detect the resulting substantive defects.”

    From How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (2026) · cites this paper as Dycke and Gurevych (2026)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 71

  1. Abdin, 2024, Phi-4 technical report, arXiv preprint arXiv:2412.08905v1 · arXiv
  2. AlKhamissi, 2022, A review on language models as knowledge bases, arXiv preprint arXiv:2204.06031v1 · arXiv
  3. Armstrong, 2022, The Scientific Method: A Guide to Finding Useful Knowledge, 10.1017/9781009092265 · DOI
  4. Association for the Advancement of Artificial Intelligence. 2025. AAAI launches AI-powered peer review assessment system. https://aaai.org/aaai-launches-ai-powered-peer-review-assessment-system/. Accessed: 2025-07-18.
  5. Bacon, 1878, Novum Organum
  6. Benjamini, 1995, Controlling the false discovery rate: A practical and powerful approach to multiple testing, Journal of the Royal Statistical Society: Series B (Methodological), 57, 289, 10.1111/j.2517-6161.1995.tb02031.x · DOI · Paper page
  7. Bennett, 1954, Communications through limited response questioning, Public Opinion Quarterly, 303, 10.1086/266520 · DOI
  8. Birukou, 2011, Alternatives to peer review: Novel approaches for research evaluation, Frontiers in Computational Neuroscience, 5, 10.3389/fncom.2011.00056 · DOI
  9. Bornmann, 2011, Scientific peer review, Annual Review of Information Science and Technology, 45, 197, 10.1002/aris.2011.1440450112 · DOI · Paper page
  10. Chamoun, 2024, Automated focused feedback generation for scientific writing assistance, 9742, 10.18653/v1/2024.findings-acl.580 · DOI
  11. Chang, 2025, TreeReview: A dynamic tree of questions framework for deep and efficient LLM-based scientific peer review, 15662, 10.18653/v1/2025.emnlp-main.790 · DOI · Paper page
  12. Chen, 2024, Masked thought: Simply masking partial reasoning steps can improve mathematical reasoning learning of language models, 5872, 10.18653/v1/2024.acl-long.320 · DOI
  13. D’Arcy, 2024, MARG: Multi-agent review generation for scientific papers, arXiv preprint arXiv:2401.04259v1 · arXiv
  14. DeepSeek-AI, 2024, Deepseek-v3 technical report, arXiv preprint arXiv:2412.19437v2 · arXiv
  15. Jiangshu, 2024, LLMs assist NLP researchers: Critique paper (meta-)reviewing, 5081, 10.18653/v1/2024.emnlp-main.292 · DOI · Paper page
  16. Dutta, 2025, Problem solving through human–AI preference-based cooperation, Computational Linguistics, 1, 10.1162/COLI.a.19 · DOI
  17. Dycke, 2023, NLPeer: A unified resource for the computational study of peer review, 5049, 10.18653/v1/2023.acl-long.277 · DOI · Paper page
  18. Dycke, 2023, Overview of PragTag-2023: Low-resource multi-domain pragmatic tagging of peer reviews, 187, 10.18653/v1/2023.argmining-1.21 · DOI
  19. Dycke, 2025, STRICTA: Structured reasoning in critical text assessment for peer review and beyond, 22687, 10.18653/v1/2025.acl-long.1107 · DOI · Paper page
  20. Xi, 2024, Large language models (LLMs) on tabular data: Prediction, generation, and understanding - a survey, Transactions on Machine Learning Research
  21. Feinstein, 1990, High agreement but low kappa: I. the problems of two paradoxes, Journal of Clinical Epidemiology, 43, 543, 10.1016/0895-4356(90)90158-L · DOI · Paper page
  22. Gallegos, 2024, Bias and fairness in large language models: A survey, Computational Linguistics, 50, 1097, 10.1162/coli_a_00524 · DOI · Paper page
  23. Gao, 2024, Reviewer2: Optimizing review generation through prompt generation, arXiv preprint arXiv:2402.10886v2 · arXiv
  24. Gat, 2024, Faithful explanations of black-box NLP models using LLM-generated counterfactuals
  25. Goyal, 2019, Explaining classifiers with causal concept effect (CaCE), arXiv preprint arXiv: 1907.07165v2 · arXiv
  26. Hsieh, 2024, RULER: What’s the real context size of your long-context language models?
  27. Huang, 2025, From pixels to insights: A survey on automatic chart understanding in the era of large foundation models, IEEE Transactions on Knowledge and Data Engineering, 37, 2550, 10.1109/TKDE.2024.3513320 · DOI
  28. Idahl, 2025, OpenReviewer: A specialized large language model for generating critical scientific paper reviews, 550, 10.18653/v1/2025.naacl-demo.44 · DOI · Paper page
  29. Jefferson, 2002, Measuring the quality of editorial peer review, JAMA, 287, 2786, 10.1001/jama.287.21.2786 · DOI · Paper page
  30. Ji, 2023, Survey of hallucination in natural language generation, ACM Computing Surveys, 55, 10.1145/3571730 · DOI · Paper page
  31. Kirtani, 2025, ReviewEval: An evaluation framework for AI-generated reviews, arXiv preprint arXiv:2502.11736v3 · arXiv
  32. Kuznetsov, 2024, What can natural language processing do for peer review?, arXiv preprint arXiv:2405.06563v1 · arXiv
  33. Li, 2025, Aspect-guided multi-level perturbation analysis of large language models in automated peer review, arXiv preprint arXiv:2502.12510v1 · arXiv
  34. Li, 2024, Prompting large language models for counterfactual generation: An empirical study, 13201, 10.63317/3xeiq2g96jjy · DOI
  35. Liang, 2024, Monitoring AI-modified content at scale: A case study on the impact of ChatGPT on AI conference peer reviews, 29575
  36. Liang, 2024, Can large language models provide useful feedback on research papers? A large-scale empirical analysis, NEJM AI, 1, AIoa2400196, 10.1056/AIoa2400196 · DOI · Paper page
  37. Lin, 2024, Advancing large language models to capture varied speaking styles and respond properly in spoken conversations, 6626, 10.18653/v1/2024.acl-long.358 · DOI
  38. Lindstrom, 1988, Newton—Raphson and EM algorithms for linear mixed-effects models for repeated-measures data, Journal of the American Statistical Association, 83, 1014, 10.1080/01621459.1988.10478693 · DOI
  39. Liu, 2023, ReviewerGPT? An exploratory study on using large language models for paper reviewing, arXiv preprint arXiv:2306.00622v1 · arXiv
  40. Lou, 2025, AAAR-1.0: Assessing AI’s potential to assist research
  41. Sheng, 2025, Identifying aspects in peer reviews, 6145, 10.18653/v1/2025.findings-emnlp.326 · DOI
  42. Madaan, 2023, Self-refine: Iterative refinement with self-feedback, 46534, 10.52202/075280-2019 · DOI · Paper page
  43. McCook, 2006, Is peer review broken? Submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. What’s wrong with peer review?, The Scientist, 20, 26
  44. Molnar, 2025, Interpretable Machine Learning, 10.21105/joss.00786 · DOI
  45. Feiteng, 2024, A causal approach for counterfactual reasoning in narratives, 6556, 10.18653/v1/2024.acl-long.354 · DOI
  46. OpenAI, Josh Achiam,, 2024, GPT-4 technical report, arXiv preprint arXiv:2303.08774v6 · arXiv
  47. Pearl, 2000, Causality: Models, Reasoning and Inference
  48. Phoenix, 2024, Prompt Engineering for Generative AI
  49. Rao, 2023, Can ChatGPT assess human personalities? A general evaluation framework, 1184, 10.18653/v1/2023.findings-emnlp.84 · DOI
  50. Sahoo, 2024, A systematic survey of prompt engineering in large language models: Techniques and applications, arXiv preprint arXiv:2402.07927v2 · arXiv
  51. Sainz, 2023, NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark, 10776, 10.18653/v1/2023.findings-emnlp.722 · DOI · Paper page
  52. Sclar, 2024, Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting
  53. Sculley, 2018, Avoiding a tragedy of the commons in the peer review process, arXiv preprint arXiv:1901.06246v1 · arXiv
  54. Shin, 2025, Mind the blind spots: A focus-level evaluation framework for LLM reviews, arXiv preprint arXiv:2502.17086v4, 10.18653/v1/2025.emnlp-main.1805 · DOI · Paper page
  55. Son, 2025, When AI co-scientists fail: SPOT-a benchmark for automated verification of scientific research, arXiv preprint arXiv:2505.11855v1 · arXiv
  56. Teufel, 2009, Towards domain-independent argumentative zoning: Evidence from chemistry and computational linguistics, 1493, 10.3115/1699648.1699696 · DOI
  57. Touvron, 2023, Llama 2: Open foundation and fine-tuned chat models, CoRR
  58. Tyser, 2024, AI-driven review systems: Evaluating LLMs in scalable and bias-aware academic reviews, arXiv preprint arXiv:2408.10365v1 · arXiv
  59. Waltman, 2023, How to improve scientific peer review: Four schools of thought, Learned Publishing, 36, 334, 10.1002/leap.1544 · DOI · Paper page
  60. Wang, 2024, A survey on natural language counterfactual generation, 4798, 10.18653/v1/2024.findings-emnlp.276 · DOI
  61. Wang, 2024, Beyond what if: Advancing counterfactual text generation with structural causal modeling, 6522
  62. Weng, 2025, CycleResearcher: Improving automated research via automated review
  63. Tongshuang, 2021, Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models, 6707, 10.18653/v1/2021.acl-long.523 · DOI
  64. Wuehrl, 2024, Understanding fine-grained distortions in reports of scientific findings, 6175, 10.18653/v1/2024.findings-acl.369 · DOI
  65. Shengwei, 2025, Benchmarking LLMs’ judgments with no gold standard
  66. Jianxiang, 2024, Automated peer reviewing in paper SEA: Standardization, evaluation, and analysis, 10164, 10.18653/v1/2024.findings-emnlp.595 · DOI
  67. Yuan, 2022, Can we automate scientific reviewing?, Journal of Artificial Intelligence Research, 75, 171, 10.1613/jair.1.12862 · DOI · Paper page
  68. Zhang, 2025, Reviewing scientific papers for critical problems with reasoning LLMs: Baseline approaches and automatic evaluation, arXiv preprint arXiv:2505.23824v2 · arXiv
  69. Zheng, 2023, Judging LLM-as-a-judge with MT-bench and chatbot arena, 46595, 10.52202/075280-2020 · DOI · Paper page
  70. Zhou, 2023, Large language models are human-level prompt engineers
  71. Zhu, 2025, Deepreview: Improving LLM-based paper review with human-like deep thinking process, arXiv preprint arXiv: 2503.08569v1 · arXiv

Cited by

Sorted by citations

Metadata

Authors
Dycke, N., & Gurevych, I.
Affiliations
Technische Universität Darmstadt (DE)
ORCID
Gurevych, Iryna
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2026
Volume
14
Pages
465-488
ISSN
2307-387X (ISSN-L)
DOI
10.1162/tacl.a.642
OpenAlex
W4415990061
Cited by
1
References
0
Citations by year
2026: 1
FWCI
8.51
Type
journal article
Language
en
Open access
Licence
Locations
  • Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
  • TUbilio (Technical University of Darmstadt) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Technischen Universität Darmstadt (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • European Commission
  • Bundesministerium für Bildung und Forschung
Topics
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
Keywords
large language models, research logic, counterfactual evaluation, scholarly peer review, scientific integrity, evaluation dataset, large language model bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review