Large language models in peer review: challenges and opportunities
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Taken together, these findings support current LLMs as reviewer assistants rather than autonomous reviewers (Sun, 2025).
“Taken together, these findings support current LLMs as reviewer assistants rather than autonomous reviewers (Sun, 2025).”
From AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality (2026) · cites this paper as Sun, 2025
LLMs can assist in multiple stages of peer review, including manuscript screening, reviewer selection, plagiarism detection, language improvement, summarization, recommendation generation and automated feedback creation.
“LLMs can assist in multiple stages of peer review, including manuscript screening, reviewer selection, plagiarism detection, language improvement, summarization, recommendation generation and automated feedback creation”
From Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review (2026) · cites this paper as Sun 2025
Existing AI-review systems often operate as opaque evaluators, making it difficult for editors to inspect why certain conclusions were generated.
“Existing AI-review systems often operate as opaque evaluators, making it difficult for editors to inspect why certain conclusions were generated (Sun 2025;Seghier 2025;Kocak et al. 2025).”
From Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review (2026) · cites this paper as Sun 2025
Current systems often suffer from hallucinations, unverifiable novelty assessments, superficial prior work comparisons and weak claim validation.
“Consequently, current systems often suffer from hallucinations, unverifiable novelty assessments, superficial prior work comparisons and weak claim validation (Sun 2025;Zhou et al. 2024;Doskaliuk et al. 2025).”
From Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review (2026) · cites this paper as Sun 2025
Existing LLM reviewer systems often generate unstructured outputs that vary substantially depending on prompting strategies, limiting reproducibility, traceability and editor interpretability.
“Existing LLM reviewer systems often generate unstructured outputs that vary substantially depending on prompting strategies, limiting reproducibility, traceability and editor interpretability (Sun 2025;Kocak et al. 2025).”
From Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review (2026) · cites this paper as Sun 2025
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1× · see citing works
“This lack of transparency can result in misunderstandings and misuse of LLM-generated outputs, particularly within academic publishing and peer review contexts (Bender et al., 2021; ICLR, 2025).”
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1× · see citing works
“Another important factor influencing LLM performance in peer review is the construction of well-designed prompts that provide the appropriate level of detail and context (Liu et al., 2023; Santu et al., 2024; Tao et al., 2024; Thelwall, 2024b).”
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · cited 1× · see citing works
“Several strategies can be considered to effectively construct an LLM–human collaboration framework for the peer review process (Yupeng Chang et al., 2024).”
Bornmann, L., & Mutz, R. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23329 · Cited by 1,576 · see citing works
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · see citing works
Transparency and the black box problem: why we do not trust AI
von Eschenbach, W. J. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00477-0 · Cited by 786 · see citing works
The TRIPOD-LLM reporting guideline for studies using large language models
Gallifant, J., Afshar, M., Ameen, S., Aphinyanaphongs, Y., Chen, S., Cacciamani, G., Demner-Fushman, D., Dligach, D., Daneshjou, R., Fernandes, C., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03425-5 · Cited by 615 · see citing works
Fabrication and errors in the bibliographic citations generated by ChatGPT
Walters, W. H., & Wilder, E. I. (2023) · Scientific Reports · doi:10.1038/s41598-023-41032-5 · Cited by 499 · see citing works
Expertise modeling for matching papers with reviewers
Mimno, D., & McCallum, A. (2007) · Knowledge Discovery and Data Mining · doi:10.1145/1281192.1281247 · Cited by 316 · see citing works
A billion-dollar donation: estimating the cost of researchers’ time spent on peer review
Aczel, B., Szaszi, B., & Holcombe, A. O. (2021) · Research Integrity and Peer Review · doi:10.1186/s41073-021-00118-2 · Cited by 306 · see citing works
Checco, A., Bracciale, L., Loreti, P., Pinfield, S., & Bianchi, G. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00703-8 · Cited by 286 · see citing works
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01783 · Cited by 243 · cited 13× · see citing works
“A primary challenge lies in assessing the novelty and significance of research findings (Huang et al., 2025a; Liang et al., 2024b; Si et al., 2024).”
InfluentialHosseini, M., & Horbach, S. P. J. M. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00133-5 · Cited by 243 · cited 3× · see citing works
“…emergence and rapid development of large language models (LLMs), including ChatGPT and its contemporaries, new approaches to improve the peer review process are being explored (Couto et al., 2024; Fecher et al., 2023; Hosseini & Horbach, 2023; Li et al., 2024a; Thelwall, 2024a; Zhao et al., 2024).”
Artificial intelligence in scientific writing: a friend or a foe?
Altmäe, S., Sola-Leyva, A., & Salumets, A. (2023) · Reproductive BioMedicine Online · doi:10.1016/j.rbmo.2023.04.009 · Cited by 174 · see citing works
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
Kang, D., Ammar, W., Dalvi, B., Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1149 · Cited by 171 · cited 1× · see citing works
“For example, PeerRead contains reviews from machine learning conferences (Kang et al., 2018); NLPeer focuses on computational linguistics venues (Dycke et al., 2023); and Openreviewer offers a larger and more recent collection of human reviews from leading machine learning conferences (Szumega et…”
Artificial intelligence to support publishing and peer review: A summary and review
Kousha, K., & Thelwall, M. (2023) · Learned Publishing · doi:10.1002/leap.1570 · Cited by 162 · see citing works
Using ChatGPT for language editing in scientific articles
Kim, S.-G. (2023) · Maxillofacial Plastic and Reconstructive Surgery · doi:10.1186/s40902-023-00381-x · Cited by 153 · see citing works
Computational support for academic peer review
Price, S., & Flach, P. A. (2017) · Communications of the ACM · doi:10.1145/2979672 · Cited by 149 · see citing works
A comparative study of several classification metrics and their performances on data
Obi, J. C. (2023) · World Journal of Advanced Engineering Technology and Sciences · doi:10.30574/wjaets.2023.8.1.0054 · Cited by 149 · see citing works
Evaluating the effectiveness of large language models in abstract screening: a comparative analysis
Li, M., Sun, J., & Tan, X. M. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02609-x · Cited by 135 · see citing works
Miao, J., Thongprayoon, C., Suppadungsuk, S., Garcia Valencia, O. A., Qureshi, F., & Cheungpasitporn, W. (2023) · Clinics and Practice · doi:10.3390/clinpract14010008 · Cited by 111 · see citing works
Tools used to assess the quality of peer review reports: A methodological systematic review
Superchi, C., González, J. A., Solà, I., Cobo, E., Hren, D., & Boutron, I. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0688-x · Cited by 110 · cited 1× · see citing works
“To mitigate subjective bias, independent domain experts evaluate both LLM-and human-generated reviews via a standardized Likert scale, with a focus on clarity, depth, and contribution to editorial decision-making (Superchi et al., 2019).”
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
The present and future of peer review: Ideas, interventions, and evidence
Aczél, B., Barwich, A., Diekman, A. B., Fishbach, A., Goldstone, R. L., Gómez, P., Gundersen, O. E., von Hippel, P. T., Holcombe, A. O., Lewandowsky, S., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401232121 · Cited by 104 · see citing works
Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models
Lai, H., Ge, L., Sun, M., Pan, B., Huang, J., Hou, L., Yang, Q., Liu, J., Liu, J., Ye, Z., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.12687 · Cited by 103 · cited 1× · see citing works
“Another promising direction is role-specific prompt design , where prompts are tailored to specific peer review tasks, such as detailed reviews, editorial summaries, or risk of bias assessment in systematic reviews (Eisele-Metzger et al., 2024; Lai et al., 2024, 2025).”
Can We Automate Scientific Reviewing?
Yuan, W., Liu, P., & Neubig, G. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12862 · Cited by 96 · see citing works
Exploring the potential of ChatGPT in the peer review process: an observational study
Saad, A., Jenko, N., Ariyaratne, S., Birch, N., Iyengar, K. P., Davies, A. M., Vaishya, R., & Botchu, R. (2024) · Diabetes & Metabolic Syndrome: Clinical Research & Reviews · doi:10.1016/j.dsx.2024.102946 · Cited by 80 · see citing works
Use of Artificial Intelligence in Peer Review Among Top 100 Medical Journals
Li, Z.-Q., Xu, H.-L., Cao, H.-J., Liu, Z.-L., Fei, Y.-T., & Liu, J.-P. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.48609 · Cited by 80 · see citing works
Friend or foe? Exploring the implications of large language models on the science system
Fecher, B., Hebing, M., Laufer, M., Pohle, J., & Sofsky, F. (2025) · AI & SOCIETY · doi:10.1007/s00146-023-01791-1 · Cited by 78 · cited 1× · see citing works
“…emergence and rapid development of large language models (LLMs), including ChatGPT and its contemporaries, new approaches to improve the peer review process are being explored (Couto et al., 2024; Fecher et al., 2023; Hosseini & Horbach, 2023; Li et al., 2024a; Thelwall, 2024a; Zhao et al., 2024).”
Use of artificial intelligence and the future of peer review
Bauchner, H., & Rivara, F. (2024) · Health affairs scholar · doi:10.1093/haschl/qxae058 · Cited by 77 · see citing works
Do Large Language Models Know What They Don’t Know?
Yin, Z., Sun, Q., Guo, Q., Wu, J., Qiu, X., & Huang, X. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.551 · Cited by 75 · see citing works
Can ChatGPT be used to generate scientific hypotheses?
Park, Y. J., Kaplan, D., Ren, Z., Hsu, C.-W., Li, C., Xu, H., Li, S., & Li, J. (2023) · Journal of Materiomics · doi:10.1016/j.jmat.2023.08.007 · Cited by 72 · see citing works
How trustworthy is ChatGPT? The case of bibliometric analyses
Farhat, F., Sohail, S. S., & Madsen, D. Ø. (2023) · Cogent Engineering · doi:10.1080/23311916.2023.2222988 · Cited by 69 · see citing works
Challenges, experiments, and computational solutions in peer review
Shah, N. B. (2022) · Communications of the ACM · doi:10.1145/3528086 · Cited by 66 · see citing works
Lehr, S. A., Caliskan, A., Liyanage, S., & Banaji, M. R. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2404328121 · Cited by 66 · see citing works
The dangers of using large language models for peer review.
Donker, T. (2023) · Lancet. Infectious Diseases (Print) · doi:10.1016/s1473-3099(23)00290-6 · Cited by 64 · see citing works
Mollaki, V. (2024) · Research Ethics · doi:10.1177/17470161231224552 · Cited by 64 · cited 2× · see citing works
“Although they excel at processing and generating text-based content (Miao et al., 2024; Mollaki, 2024), their ability to interpret and reason through intricate mathematical notation or multistep derivations remains significantly constrained (Das et al., 2024; Yan et al., 2024).”
Language models for data extraction and risk of bias assessment in complementary medicine
Lai, H., Liu, J., Bai, C., Liu, H., Pan, B., Luo, X., Hou, L., Zhao, W., Xia, D., Tian, J., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01457-w · Cited by 59 · see citing works
Take caution in using LLMs as human surrogates
Gao, Y., Lee, D., Burtch, G., & Fazelpour, S. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2501660122 · Cited by 52 · see citing works
Human-in-the-loop AI reviewing: Feasibility, opportunities, and risks
Drori, I., Boston University / Columbia University, Te'eni, D., & Tel Aviv University (2024) · Journal of the Association for Information Systems · doi:10.17705/1jais.00867 · Cited by 48 · see citing works
Peer Review in the Age of Generative AI
Kankanhalli, A., & National University of Singapore (2024) · Journal of the Association for Information Systems · doi:10.17705/1jais.00865 · Cited by 47 · see citing works
Benchmarking Human–AI collaboration for common evidence appraisal tools
Woelfle, T., Hirt, J., Janiaud, P., Kappos, L., Ioannidis, J. P. A., & Hemkens, L. G. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111533 · Cited by 46 · see citing works
ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing
Sun, L., Chan, A., Chang, Y., & Dow, S. P. (2024) · International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645159 · Cited by 44 · see citing works
Are large language models qualified reviewers in originality evaluation?
Huang, S., Huang, Y., Liu, Y., Luo, Z., & Lu, W. P. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103973 · Cited by 41 · see citing works
Farber, S. (2024) · Learned Publishing · doi:10.1002/leap.1638 · Cited by 39 · see citing works
Thelwall, M. A., & Yaghi, A. (2025) · Scientometrics · doi:10.1007/s11192-025-05287-1 · Cited by 36 · see citing works
Large language models for automated scholarly paper review: A survey
Zhuang, Z., Chen, J., Xu, H., Jiang, Y., & Lin, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103332 · Cited by 34 · cited 2× · see citing works
“A focused review of recent implementations of LLMs in both pre-peer review and formal peer review processes is presented, and key insights from the literature that have not been summarized in previous reviews are highlighted (Chen et al., 2025a; Luo et al., 2025b; Zhuang et al., 2025).”
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · arXiv · doi:10.48550/arxiv.2406.12708 · Cited by 34 · cited 3×
“…emergence and rapid development of large language models (LLMs), including ChatGPT and its contemporaries, new approaches to improve the peer review process are being explored (Couto et al., 2024; Fecher et al., 2023; Hosseini & Horbach, 2023; Li et al., 2024a; Thelwall, 2024a; Zhao et al., 2024).”
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.70 · Cited by 33 · see citing works
Generative artificial intelligence and scientific publishing: urgent questions, difficult answers
Bagenal, J. D. (2024) · The Lancet · doi:10.1016/s0140-6736(24)00416-1 · Cited by 33 · see citing works
Lee, S., Sim, W., Shin, D., Seo, W., Park, J., Lee, S. C., Hwang, S., Kim, S., & Kim, S. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3712701 · Cited by 33 · see citing works
DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process
Zhu, M., Weng, Y., Yang, L., & Zhang, Y. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1420 · Cited by 31 · see citing works
Research evaluation with ChatGPT: is it age, country, length, or field biased?
Thelwall, M. A., & Kurt, Z. (2025) · Scientometrics · doi:10.1007/s11192-025-05393-0 · Cited by 29 · see citing works
Suleiman, A., von Wedel, D., Munoz-Acuna, R., Redaelli, S., Santarisi, A., Seibold, E.-L., Ratajczak, N., Kato, S., Said, N., Sundar, E., et al. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108313 · Cited by 27 · see citing works
Affiliation Bias in Peer Review of Abstracts by a Large Language Model
von Wedel, D., Schmitt, R. A., Thiele, M., Leuner, R., Shay, D., Redaelli, S., & Schaefer, M. S. (2024) · JAMA · doi:10.1001/jama.2023.24641 · Cited by 27 · see citing works
Democratizing Knowledge Creation Through Human-AI Collaboration in Academic Peer Review
Sarker, S., University of Virginia, Susarla, A., Michigan State University, Gopal, R., University of Warwick, Thatcher, J. B., & University of Colorado / University of Manchester (2024) · Journal of the Association for Information Systems · doi:10.17705/1jais.00872 · Cited by 25 · see citing works
Scientific sinkhole: estimating the cost of peer review based on survey data with snowball sampling
LeBlanc, A. G., Barnes, J. D., Saunders, T. J., Tremblay, M. S., & Chaput, J. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00128-2 · Cited by 25 · see citing works
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Narayanan Venkit, P., Zhang, N., Srinath, M., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.292 · Cited by 23 · see citing works
Exploring ChatGPT's abilities in medical article writing and peer review
Kadı, G., & Aslaner, M. A. (2024) · Croatian Medical Journal · doi:10.3325/cmj.2024.65.93 · Cited by 23 · see citing works
Using GPT-4 to write a scientific review article: a pilot evaluation study
Wang, Z. P., Bhandary, P., Wang, Y., & Moore, J. H. (2024) · BioData Mining · doi:10.1186/s13040-024-00371-3 · Cited by 22 · see citing works
Huang, S., Wang, Q., Lu, W. P., Liu, L., Xu, Z., & Huang, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104225 · Cited by 22 · see citing works
NLPeer: A Unified Resource for the Computational Study of Peer Review
Dycke, N., Kuznetsov, I., & Gurevych, I. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.277 · Cited by 19 · see citing works
Reviewriter: AI-Generated Instructions For Peer Review Writing
Su, X., Wambsganß, T., Rietsche, R., Neshaei, S. P., & Kser, T. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.5 · Cited by 19 · see citing works
Srivastava, M. (2023) · doi:10.31219/osf.io/wydct · Cited by 18 · see citing works
AI-powered peer review needs human supervision
Seghier, M. L. (2025) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-09-2024-0132 · Cited by 18 · see citing works
Chen, C., Maqsood, A., Zhang, Z., Wang, X., Duan, L., Wang, H., Chen, T., Liu, S., Li, Q., Luo, J., et al. (2024) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2024.102058 · Cited by 16 · see citing works
Chauhan, C., & Currie, G. (2024) · The American Journal of Pathology · doi:10.1016/j.ajpath.2024.08.002 · Cited by 16 · see citing works
Zhao, M., Li, F., Cai, F., Chen, H., & Li, Z. (2024) · Nankai Business Review International · doi:10.1108/nbri-12-2023-0115 · Cited by 15 · see citing works
MOPRD: A multidisciplinary open peer review dataset
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2022) · arXiv · doi:10.48550/arxiv.2212.04972 · Cited by 11 · cited 1× · see citing works
“…datasets, such as MOPRD, Open Review-Based datasets, and withdrawn paper datasets from arXiv, which provide a broader scope of peer review data across various fields, also exist (Liang et al., 2024b; Lin et al., 2023; Szumega et al., 2023; Tan et al., 2024; Zhang & Abernethy, 2025).”
Generative AI: ensuring transparency and emphasising human intelligence and accountability
Bagenal, J., Biamis, C., Boillot, M., Brierley, R., Chew, M., Dehnel, T., Frankish, H., Grainger, E., Pope, J., Prowse, J., et al. (2024) · The Lancet · doi:10.1016/s0140-6736(24)02615-1 · Cited by 11 · see citing works
Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis
Yu, J., Ding, Z., Tan, J., Luo, K., Weng, Z., Gong, C., Zeng, L., Cui, R., Han, C., Sun, Q., et al. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.595 · Cited by 9 · see citing works
MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning
Das, D., Banerjee, D., Aditya, S., & Kulkarni, A. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.54 · Cited by 8 · see citing works
Use of ChatGPT to Explore Gender and Geographic Disparities in Scientific Peer Review
Sebo, P. (2024) · Journal of Medical Internet Research · doi:10.2196/57667 · Cited by 7 · see citing works
ChatGPT identifies gender disparities in scientific peer review
Verharen, J. P. H. (2023) · eLife · doi:10.7554/elife.90230.3 · Cited by 7 · see citing works
Shopovski, J., Mohdali, R., & Marolov, D. (2025) · doi:10.20944/preprints202502.0058.v1 · Cited by 7 · see citing works
Applications of generative AI in peer review process: friend or foe?
Wu, H., Sun, Z., Guo, Q., & Li, C. (2024) · International Journal of Surgery · doi:10.1097/js9.0000000000001689 · Cited by 6 · see citing works
Self-citationLimGen: Probing the LLMs for Generating Suggestive Limitations of Research Papers
Faizullah, A. R. B. M., Urlana, A., & Mishra, R. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-70344-7_7 · Cited by 6 · see citing works
Leveraging large language models for academic conference organization
Luo, Y., Li, Y., Ogunyemi, O. I., Koski, E., & Himes, B. E. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01492-7 · Cited by 4 · see citing works
ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews
D’Arcy, M., Ross, A., Bransom, E., Kuehl, B., Bragg, J., Hope, T., & Downey, D. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.377 · Cited by 4 · see citing works
Exploring the Limits of Fine-grained LLM-based Physics Inference via Premise Removal Interventions
Meadows, J., James, T. E., & Freitas, A. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.378 · Cited by 3 · see citing works
Tunga, Y., & Çelik, B. (2024) · EDULEARN proceedings · doi:10.21125/edulearn.2024.1561 · Cited by 2 · see citing works
Software searches out reproducibility issues in scientific papers
Chawla, D. S. (2020) · Nature · doi:10.1038/d41586-020-00104-6 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Can large language models assess the quality of peer review? An empirical study
Tang, Y., Kang, Y., Wu, S., Zhang, R., & Sun, Z. (2026) · Scientometrics · doi:10.1007/s11192-026-05622-0 · Cited by 4 · see citing works
Self-citationTorrielli, F., Locci, S., Rapp, A., & Di, L. (2026) · Scientometrics · doi:10.1007/s11192-026-05695-x · Cited by 4 · see citing works
Editorial: artificial intelligence for scientometrics (Part I)
Zhang, Y., Zhang, C., & Kousha, K. (2025) · Scientometrics · doi:10.1007/s11192-025-05470-4 · Cited by 3 · see citing works
Relationship between peer review quality and scientific impact: Insights from LLMs-assessed reviews
Sun, Z. (2026) · Journal of Informetrics · doi:10.1016/j.joi.2026.101801 · Cited by 3 · see citing works
Self-citationAI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality
Fichtl, A. M., Ellinger, L., Kelber, J., Olík, K., & Groh, G. (2026) · arXiv · doi:10.48550/arxiv.2608.03581
Ali, Ș., Oprea, S., & Bâra, A. (2026) · Systems Research and Behavioral Science · doi:10.1002/sres.70163
Unraveling the Ai2 Asta Scholarly Research Assistant Citation System
Orduña‐Malea, E., & Lopezosa, C. (2025) · Revista Panamericana de Comunicación · doi:10.21555/rpc.v7i2.3675
100% AI-Reviewed Preprints are the Future of Open Research
Irfanullah, H. M. (2025) · Trends in Scholarly Publishing · doi:10.21124/tsp.2025.102.108
Lapinsky, S. E., & Frise, C. (2026) · Obstetric Medicine · doi:10.1177/1753495x261446731
AI ethics after generative AI: from consequences to effects
Gianni, R. (2026) · AI & Society · doi:10.1007/s00146-026-03308-y
Ethical evaluation of large language models for scientometrics: a multidimensional perspective
Shen, Z., Liu, Y., Liu, Y., Wang, Z., & Yuan, S. (2026) · Scientometrics · doi:10.1007/s11192-026-05773-0
Comparative performance of large language models for appraising bias in real-world evidence studies
Okeke, C. M., Nechi, R. N., Khalid, J., Karaye, I. M., Thornton, J. D., & Yunusa, I. U. (2026) · Journal of Managed Care & Specialty Pharmacy · doi:10.18553/jmcp.2026.32.9.1062
Evaluating automated refereeing with heterogeneous LLM panels
Douven, I. (2026) · Scientometrics · doi:10.1007/s11192-026-05783-y
A Multi-stage Framework for Detection and Attribution of LLM-Generated Text
Long, V. M. T., Chan, R. C. B., & Truong-Huu, T. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-37274-1_5
Luo, S., Jiang, Y., & Huang, C. (2026) · Scientometrics · doi:10.1007/s11192-026-05806-8
Author metadata affects Large Language Model scores in scientific peer review
Rospocher, M. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02268-y
Artificial Intelligence in Scholarly Peer Review: Applications, Benefits, Risks, and Recommendations
Abdullah, R. Y. (2026) · Health Innovation Reports · doi:10.64048/hir.v2n2.004
- Authors
- Sun, Z.
- Affiliations
- Nanjing University of Posts and Telecommunications (CN)
- Corresponding author
- Sun, Zhuanlan
- ORCID
- Sun, Zhuanlan
- Venue
- Scientometrics
- Published
- 2025-10
- Volume
- 130
- Issue
- 10
- Pages
- 5503-5546
- ISSN
- 0138-9130 (ISSN-L)
- 1588-2861
- OpenAlex
- W4415550727
- Cited by
- 15
- References
- 80
- Citations by year
- 2026: 12 · 2025: 3
- FWCI
- 56.43
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Scientometrics (publishedVersion)
- Funders
- Youth Program of National Natural Science Foundation of China (72404144)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- large language models, peer review, scholarly publishing, automated peer review, reviewer selection, automated feedback, bias detection, LLM agents, prompt engineering, multi-agent systems, editorial system
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
