The DOI matches its Crossref record; the references it cites are not checked yet.
Personal experience with AI-generated peer reviews: a case study
Open accessAbstract
BACKGROUND: While some recent studies have looked at large language model (LLM) use in peer review at the corpus level, to date there have been few examinations of instances of AI-generated reviews in their social context. The goal of this first-person account is to present my experience of receiving two anonymous peer review reports that I believe were produced using generative AI, as well as lessons learned from that experience. METHODS: This is a case report on the timeline of the incident, and my and the journal's actions following it. Supporting evidence includes text patterns in the reports, online AI detection tools and ChatGPT simulations; recommendations are offered for others who may find themselves in a similar situation. The primary research limitation of this article is that it is based on one individual's personal experience. RESULTS: After alleging the use of generative AI in December 2023, two months of back-and-forth ensued between myself and the journal, leading to my withdrawal of the submission. The journal denied any ethical breach, without taking an explicit position on the allegations of LLM use. Based on this experience, I recommend that authors engage in dialogue with journals on AI use in peer review prior to article submission; where undisclosed AI use is suspected, authors should proactively amass evidence, request an investigation protocol, escalate the matter as needed, involve independent bodies where possible, and share their experience with fellow researchers. CONCLUSIONS: Journals need to promptly adopt transparent policies on LLM use in peer review, in particular requiring disclosure. Open peer review where identities of all stakeholders are declared might safeguard against LLM misuse, but accountability in the AI era is needed from all parties.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
One documented incident described prolonged disputes between authors and journal editors following allegations of undisclosed AI-generated reviewing, ultimately resulting in manuscript withdrawal due to unclear investigation procedures and insufficient editorial policies.
“One documented incident described prolonged disputes between authors and journal editors following allegations of undisclosed AI-generated reviewing, ultimately resulting in manuscript withdrawal due to unclear investigation procedures and insufficient editorial policies (Lo Vecchio 2025).”
From Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review (2026) · cites this paper as Lo Vecchio 2025
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredOther case reports point to the use of generative AI during peer review.
“Other case reports point to the use of generative AI during peer review 3”
From Fine-Grained Detection of AI-Generated Writing in the Biomedical Literature (2026) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
The citation lists haven’t been fetched yet
References 25 · Cited by 15
Research works printed but not linked: 29
- Kousha K, Thelwall M. Artificial intelligence to support publishing and peer review: A summary and review. Learned Publishing. 2024;37:4–12. · DOI
- Levene A. Where next in peer review? Part 2: COPE Commentary. Committee on Publication Ethics. 16 November 2023. https://publicationethics.org/news/where-next-peer-review-ai
- Schintler LA, McNeely CL, Witte J. A critical examination of the ethics of AI-mediated peer review. arXiv; 2023. https://arxiv.org/abs/2309.12356 · arXiv
- Thelwall M. Can ChatGPT evaluate research quality? Journal of Data and Information Science. 2024;9(2):1–21. · DOI · Paper page
- Zou J. ChatGPT is transforming peer review—how can we use it responsibly? [online headline]. Nature. 2024;635(8037):10. · DOI
- Alvero AJ, Lee J, Regla-Vargas A, Kizilcec RF, Joachims T, Antonio AL. Large language models, social demography, and hegemony: Comparing authorship in human and synthetic text. Journal of Big Data. 2024;11:138. · DOI · Paper page
- Cheng K, Sun Z, Liu X, Wu H, Li C. Generative artificial intelligence is infiltrating peer review process. Crit Care. 2024;28:149. · DOI
- Flanagin A, Kendall-Taylor J, Bibbins-Domingo K. Guidance for authors, peer reviewers, and editors on use of AI, language models, and chatbots. JAMA. 2023;330(8):702–3. · DOI · Paper page
- Bagenal J, Biamis C, Boillot M, Brierley R, Chew M, Dehnel T, Frankish H, Grainger E, Pope J, Prowse J, Samuel D, Slogrove AL, Stacey J, Thapaliya G, Trethewey F, Wang HH, Varley-Reeves J, Kleinert S. Generative AI: Ensuring transparency and emphasising human intelligence and accountability. The Lancet. 2024;404(10468):2142–3. · DOI · Paper page
- Liang W, Izzo Z, Zhang Y, Lepp H, Cao H, Zhao X, Chen L, Ye H, Liu S, Huang Z, McFarland DA, Zou J. Monitoring AI-modified content at scale: A case study on the impact of ChatGPT on AI conference peer reviews. Proceedings of the 41st International Conference on Machine Learning. Proc Mach Learn Res. 2024;235:29575–620. https://proceedings.mlr.press/v235/liang24b.html.
- Ebadi S, Nejadghanbar H, Salman AR, Khosravi H. Exploring the impact of generative AI on peer review: Insights from journal reviewers. Journal of Academic Ethics. 2025. https://doi.org/10.1007/s10805-025-09604-4. · DOI · Paper page
- Grove J. “ChatGPT-generated reading list” sparks AI peer review debate. Times Higher Education. 5 April 2023. www.timeshighereducation.com/news/chatgpt-generated-reading-list-sparks-ai-peer-review-debate
- Russo Latona G, Horta Ribeiro M, Davidson TR, Veselovsky V, West R. The AI review lottery: Widespread AI-assisted peer reviews boost paper scores and acceptance rates. arXiv; 2024. https://arxiv.org/abs/2405.02150 · arXiv
- Majovsky M. AI-generated responses in peer review pose a growing challenge for reviewers and editors: Call for a reviewer rating system. J Clin Neurosci. 2025;133:111042. · DOI
- Anonymous review reports received by author on 19 December 2023, from guest editors for mediAzioni 2024;43. Accessible at www.nicospage.eu
- Liang W, Yuksekgonul M, Mao Y, Wu E, Zou J. GPT detectors are biased against non-native English writers. Patterns. 2023;4(7):100779. · DOI · Paper page
- Sheinman Orenstrakh M, Karnalim O, Aníbal Suárez C, Liut M. Detecting LLM-generated text in computing education: A comparative study for ChatGPT cases. arXiv; 2023. https://arxiv.org/abs/2307.07411 · arXiv
- Chakraborty S, Bedi A, Zhu S, An B, Manocha D, Huang F. Position: On the possibilities of AI-generated text detection. Proceedings of the 41st International Conference on Machine Learning. Proc Mach Learn Res. 2024;235:6093–115. https://proceedings.mlr.press/v235/chakraborty24a.html.
- Narayanan A, Kapoor S. AI snake oil: What artificial intelligence can do, what it can’t, and how to tell the difference. Princeton: Princeton University Press; 2024. · DOI
- Donker T. The dangers of using large language models for peer review [letter to the editors with supplementary appendix online]. The Lancet Infectious Diseases 2023;23(7):781. www.thelancet.com/journals/laninf/article/PIIS1473-3099(23)00290-6/fulltext · DOI
- mediAzioni. Journal response. Joint email communication from editors in chief and guest editors. 22 February 2024. Accessible at www.nicospage.eu
- mediAzioni website. About the Journal. https://mediazioni.unibo.it/about.
- Biswas S, Dobaria D, Cohen HL. ChatGPT and the future of journal reviews: A feasibility study. Yale Journal of Biology and Medicine. 2023;96(3):415–20. · DOI · Paper page
- Irfanullah H. Ending human-dependent peer review. The Scholarly Kitchen. 29 September 2023. https://scholarlykitchen.sspnet.org/2023/09/29/ending-human-dependent-peer-review/
- Bauchner H, Rivara FP. Use of artificial intelligence and the future of peer review. Health Aff Sch. 2024;2(5):qxae058. · DOI
- Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer review. J Am Soc Inform Sci Technol. 2013;64:2–17. · DOI · Paper page
- Ross-Hellauer T, Bouter LM, Horbach SPJM. Open peer review urgently requires evidence: A call to action. PLoS Biol. 2023;21(10):e3002255. · DOI
- Levene A. Where next in peer review? Part 1: COPE Commentary. Committee on Publication Ethics. 22 October 2023. https://publicationethics.org/news/where-next-peer-review-part-1
- Cooke SJ, Young N, Peiman KS, Roche DG, Clements JC, Kadykalo AN, Provencher JF, Raghavan R, DeRosa MC, Lennox RJ, Fayek AR, Cristescu ME, Murray SJ, Quinn J, Cobey KD, Browman HI. A harm reduction approach to improving peer review by acknowledging its imperfections. Facets. 2024;9:1–14.
Metadata
- Authors
- Lo Vecchio, N.
- ORCID
- Lo Vecchio, Nicholas
- Venue
- Research Integrity and Peer Review
- Published
- 2025-04-07
- Volume
- 10
- Issue
- 1
- Pages
- 4-4
- ISSN
- 2058-8615 (ISSN-L)
- PMC
- PMC11974187
- OpenAlex
- W4409201574
- Cited by
- 15
- References
- 25
- Citations by year
- 2026: 10 · 2025: 5
- FWCI
- 56.43
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Research Integrity and Peer Review (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- large language models, AI detection, author experience, AI disclosure, open peer review, accountability, research integrity, peer review ethics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
