OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews
Open accessAbstract
Maximilian Idahl, Zahra Ahmadi. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations). 2025.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The annotation is bidirectional: AI and human review items are subject to the same scrutiny rather than human reviews being treated as the implicit reference for AI.
“The annotation is bidirectional, in that AI and human review items are subject to the same scrutiny rather than human reviews being treated as the implicit reference for AI [Chang et al., 2025, Garg et al., 2025, Idahl and Ahmadi, 2025, Bougie and Watanabe, 2025].”
From On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists (2026) · cites this paper as Idahl and Ahmadi, 2025
Section 4.2, “Review Arena,” says the evaluator judges which AI-generated review aligns better with human expert reviews. The judge prompt reproduced in the Ethics and Broader Impact Statement explicitly calls the expert reviews the “groundtruth of reviews.” This treats human reviews as the reference, contrary to the claimed bidirectional scrutiny. (LLM, conf=0.91)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredOpenReviewer fine-tunes a Llama-8B model on a curated dataset of 79,000 expert reviews, and its high performance stemmed from curating reviews with diverse critique patterns such as methodological flaws, reproducibility issues and balanced scores.
“OpenReviewer (Idahl and Ahmadi, 2025) fine-tunes a Llama-8B model (AI@Meta, 2024) on a curated dataset of 79,000 expert reviews.Its high performance stemmed from curating reviews with diverse critique patterns such as methodological flaws, reproducibility issues and balanced scores.”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Idahl and Ahmadi, 2025
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredThe datasets cover an early peer-review corpus, the ACL Rolling Review process, and public ICLR review records, respectively, and differ in field structure, text granularity, and label sources.
“These datasets cover an early peer-review corpus, the ACL Rolling Review process, and public ICLR review records, respectively, and differ in field structure, text granularity, and label sources.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Idahl and Ahmadi 2025
Section 6.3, “Datasets and Resources,” identifies PeerRead as a peer-review dataset from machine-learning conferences and NLPeer as focused on computational-linguistics venues. Section 3.2.1, “Training Data,” describes reviews collected from ICLR and NeurIPS through OpenReview. Neither passage identifies NLPeer specifically with the ACL Rolling Review process or compares the datasets’ field structure, text granularity, or label sources; page numbers are not available in our copy. (LLM, conf=0.83)
Experimental — accuracy not yet measuredExisting LLM-based peer-review systems align prompts or metrics with reviewer guidelines, but they do not treat CfPs themselves as retrieval sources.
“Existing LLM-based peer-review systems like SEA (Yu et al., 2024), OpenReviewer (Idahl and Ahmadi, 2025) and ReviewEval (Garg et al., 2024) align prompts or metrics with reviewer guidelines, but they do not treat CfPs themselves as retrieval sources.”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Idahl and Ahmadi, 2025
Process variables are open-ended texts with no single correct expression.
“these process variables are open-ended texts with no single correct expression.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Idahl and Ahmadi 2025
Peer review is a process in which domain experts analyze a paper, identify its main contributions and problems, provide suggestions for improvement, and form an overall assessment.
“Peer review is a process in which domain experts analyze a paper, identify its main contributions and problems, provide suggestions for improvement, and form an overall assessment (Liang et al. 2023;Idahl and Ahmadi 2025;Biswas et al. 2026).”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Idahl and Ahmadi 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. 2024. Nougat: Neural optical under- standing for academic documents. In The Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024.
- Laurent Charlin and Richard Zemel. 2013. The Toronto Paper Matching System: An automated paper-reviewer assignment system. In Proceedings of the 30th International Conference on Machine Learning, volume 28, Atlanta, Georgia, USA. Jour- nal of Machine Learning Research Workshop and Conference Proceedings.
- Tri Dao. 2024. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. 2023b. MOPRD: A multi- disciplinary open peer review dataset. Neural Com- put. Appl., 35(34):24191-24206. · DOI · Paper page
Cited by
Sorted by citations
Rethinking peer review: ethics and blind spots of scientific gatekeeping
Montero, J. A. (2025) · World Journal of Pediatrics · doi:10.1007/s12519-025-00960-0 · Cited by 4 · see citing works
Guo, B., Wen, Z., Yu, Y., Gao, P., Yang, R., & Shen, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3456-2_27 · Cited by 4 · see citing works
Torrielli, F., Locci, S., Rapp, A., & Di, L. (2026) · Scientometrics · doi:10.1007/s11192-026-05695-x · Cited by 4 · see citing works
Co-Reviewer: can AI review like a human? An agentic framework for LLM-human alignment in peer review
Bharti, P. K., Dalal, V., & Panchal, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05557-6 · Cited by 2 · see citing works
Kim, S., Yoon, D., Gashteovski, K., Suk, J., Baek, J., Aggarwal, P., Wu, I., Zaverkin, V., Petkoski, S., Schrider, D. R., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.20668 · Cited by 1 · see citing works
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Wu, S., Jiang, O., Zhao, Y., Hu, T., Ma, Y., Zhang, K., Patwardhan, M., & Cohan, A. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1504 · Cited by 1 · see citing works
Dycke, N., & Gurevych, I. (2025) · arXiv · doi:10.48550/arxiv.2508.21422 · Cited by 1
LimAgents: Multi-Agent LLMs for Generating Research Limitations
Azher, I. A., Guo, Z., & Alhoori, H. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846066 · Cited by 1 · see citing works
PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing
Żurawicki, K., Farganus, J., Gaweł, A., Bystroński, M., & Kajdanowicz, T. J. (2026) · arXiv · doi:10.48550/arxiv.2605.29815
CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
Deng, H., Ke, X., Li, Y., Hu, R., Huang, D., Wong, D. F., Wang, Y., Liu, X., & Zhang, M. (2026) · arXiv · doi:10.48550/arxiv.2605.07905
Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review
Yuan, S., Zhang, X., Ni, W., Xiao, T., Di, S., Zhu, J., Jiang, Z., Tan, R., Chen, L., & Ren, K. (2026) · arXiv · doi:10.48550/arxiv.2609.05947
Do Methods Support the Claims? Intra-Paper Verification for Peer Review
Ballakuraya, R. S., Mahyari, A., & Srinivasan, A. (2026) · arXiv · doi:10.48550/arxiv.2607.26066
Li, C., Lai, Z., An, D. U., Hu, J., & Wen, Z. (2026) · Journal of Scientific Computing · doi:10.1007/s10915-026-03493-4
LLM-Based Scholarly Paper Review for Master's Theses
Sun, P., Huang, D., Wang, Q., Na, Q., & Cai, L. (2025) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale66047.2025.11346794
RDoFlow: Automatically assessing under-specified statistical analyses in HCI
Grunde-McLaughlin, M., Liu, W., Patil, R., Migineishvili, N., Reif, E., Krishna, R., Weld, D. S., & Heer, J. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789160
Supporting Artifact Evaluation with LLMs: A Study with Published Security Research Papers
Heye, D., Kindermann, K., Decker, R., Lohmöller, J., Belova, A., Geisler, S., Wehrle, K., & Pennekamp, J. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401815
(2344960), Y. G., Yu, L., & Li, H. (2026) · Journal of Superintelligence · doi:10.3390/superintelligence1020008
Masrur, M. A. (2026) · IEEE Intelligent Systems · doi:10.1109/mis.2026.3682716
Evaluating automated refereeing with heterogeneous LLM panels
Douven, I. (2026) · Scientometrics · doi:10.1007/s11192-026-05783-y
- Authors
- Idahl, M., & Ahmadi, Z.
- Affiliations
- Leibniz University Hannover (DE)
- Venue
- Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations)
- Published
- 2025
- Pages
- 550-562
- OpenAlex
- W4411119960
- Cited by
- 10
- References
- 0
- Citations by year
- 2026: 6 · 2025: 4
- FWCI
- 9.53
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Funders
- Bundesministerium für Bildung und Forschung (01IS22093C)
- Gottfried Wilhelm Leibniz Universität Hannover (11-76251-114/2022)
- Leibniz-Gemeinschaft
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, peer review, critical reading, academic writing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
