Artificial intelligence to support publishing and peer review: A summary and review
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Technology is being developed to support the peer review processes of journals, conferences, funders, universities, and national research evaluations. This literature and software summary discusses the partial or complete automation of several publishing‐related tasks: suggesting appropriate journals for an article, providing quality control for submitted papers, finding suitable reviewers for submitted papers or grant proposals, reviewing, and review evaluation. It also discusses attempts to estimate article quality from peer review text and scores as well as from post‐publication scores but not from bibliometric data. The literature and existing examples of working technology show that automation is useful for helping to find reviewers and there is good evidence that it can sometimes help with initial quality control of submitted manuscripts. Much other software supporting publishing and editorial work exists and is being used, but without published academic evaluations of its efficacy. The value of artificial intelligence (AI) to support reviewing has not been clearly demonstrated yet, however. Finally, whilst peer review text and scores can theoretically have value for post‐publication research assessment, it is not yet widely enough available to be a practical evidence source for systematic automation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24750 · Cited by 919
Hosseini, M., & Horbach, S. P. J. M. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00133-5 · Cited by 328 · cited 1×
“…to keep the manuscripts assessed con fi dential; as such, papers under assessment should not be uploaded to LLMs because they may be saved and incorporated into responses to future questions for other users (Hosseini & Horbach, 2023), or may even grant the LLM owner the right to repurpose content.”
BackgroundChecco, A., Bracciale, L., Loreti, P., Pinfield, S., & Bianchi, G. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00703-8 · Cited by 286
Will ChatGPT get you caught? Rethinking of Plagiarism Detection
Khalil, M., & Er, E. (2023) · doi:10.35542/osf.io/fnh48 · Cited by 259
Can We Automate Scientific Reviewing?
Yuan, W., Liu, P., & Neubig, G. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12862 · Cited by 96
The Rigor and Transparency Index Quality Metric for Assessing Biological and Medical Science Methods
Menke, J., Roelandse, M., Ozyurt, B., Martone, M., & Bandrowski, A. (2020) · iScience · doi:10.1016/j.isci.2020.101698 · Cited by 90
AI peer reviewers unleashed to ease publishing grind.
Heaven, D. (2018) · Nature · doi:10.1038/d41586-018-07245-9 · Cited by 75 · cited 1×
“…citations in the manuscript text match the reference list and vice versa (cid:129) Multipurpose manuscript evaluation : Other AI-assisted tools assess multiple quality control aspects of manuscripts such as Frontiers AI Review Assistant 19 or UNSILO Manuscript Evaluation 20 (see also Heaven, 2018).”
BackgroundAspect-based Sentiment Analysis of Scientific Reviews
Chakraborty, S., Goyal, P., & Mukherjee, A. (2020) · Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 · doi:10.1145/3383583.3398541 · Cited by 63 · cited 1×
“There is also evidence that sentiment analysis of review reports could be helpful to predict the fi nal decision (acceptance or rejection) of conference papers (Chakraborty et al., 2020; Ghosal, Verma, et al., 2019; Wang & Wan, 2018) or review scores of funding programmes (Luo et al., 2021).”
BackgroundDeepSentiPeer: Harnessing Sentiment in Review Texts to Recommend Peer Review Decisions
Ghosal, T., Verma, R., Ekbal, A., & Bhattacharyya, P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1106 · Cited by 62 · cited 1×
“There is also evidence that sentiment analysis of review reports could be helpful to predict the fi nal decision (acceptance or rejection) of conference papers (Chakraborty et al., 2020; Ghosal, Verma, et al., 2019; Wang & Wan, 2018) or review scores of funding programmes (Luo et al., 2021).”
BackgroundNuijten, M. B., & Polanin, J. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1408 · Cited by 56 · cited 1×
“(cid:129) Automated statistical checking : StatReviewer 14 checks manuscripts against standardized reporting guidelines and StatCheck 15 detects some statistical errors in the submitted works (Nuijten & Polanin, 2020).”
Background
Research works printed but not linked: 51
- 10.1109/CCAA.2018.8777730 · doi:10.1109/ccaa.2018.8777730
- 10.1371/journal.pone.0005910 · DOI · Paper page
- American Journal Experts. (2018).Peer review: How we found 15 million hours of lost time.www.aje.com/en/arc/peer-review-process-15-million-hours-lost-time/
- 10.1002/asi.23334 · DOI
- 10.1016/j.joi.2012.12.003 · DOI
- 10.1016/j.joi.2015.03.001 · DOI · Paper page
- Cabanac G. Labbé C. &Magazinov A.(2021).Tortured phrases: A dubious writing style emerging in science. Evidence of critical issues affecting established journals.arXiv preprint arXiv:2107.06751. · arXiv
- Charlin L. &Zemel R.(2013).The Toronto paper matching system: An automated paper‐reviewer assignment system.https://openreview.net/pdf?id=caynafZAnBafx
- 10.1038/d41586-019-01517-8 · DOI
- 10.1002/asi.23548 · DOI
- 10.2196/12957 · DOI
- Fiez T. Shah N. &Ratliff L.(2020).A SUPER* algorithm to optimize paper bidding in peer review. InConference on Uncertainty in Artificial Intelligence(pp. 580–589). PMLR.http://proceedings.mlr.press/v124/fiez20a/fiez20a.pdf
- 10.1109/JCDL.2019.00039 · DOI
- 10.1109/JCDL52503.2021.00021 · DOI
- 10.1007/s11192-020-03519-0 · DOI
- 10.3174/ajnr.A4234 · DOI
- 10.6087/kcse.137 · DOI
- 10.1371/journal.pone.0166387 · DOI · Paper page
- Li X. &Thelwall M.(2012).F1000 Mendeley and traditional bibliometric indicators.Proceedings of the 17th International Conference on Science and Technology Indicators(Vol. 2 pp. 451–551).
- 10.1016/j.procs.2013.09.144 · DOI · Paper page
- Lin J. Song J. Zhou Z. &Shi X.(2021).Automated scholarly paper review: Possibility and challenges.arXiv preprint arXiv:2111.07533. · arXiv
- 10.1016/j.compind.2015.11.001 · doi:10.1016/j.compind.2015.11.001
- 10.1007/s11192-022-04389-4 · DOI
- 10.1162/qss_a_00156 · DOI · Paper page
- 10.1007/s11192-013-0993-9 · DOI
- 10.1371/journal.pone.0184711 · DOI · Paper page
- 10.1007/s11192-017-2399-6 · DOI
- 10.1108/AJIM-06-2018-0153 · DOI
- 10.1002/asi.24568 · DOI · Paper page
- 10.1016/j.neucom.2020.11.004 · DOI
- 10.1016/j.knosys.2019.105092 · DOI
- 10.1145/2979672 · DOI · Paper page
- 10.1007/s11948-015-9625-5 · DOI · Paper page
- Roberts J., 2020, pReview: The artificially intelligent conference reviewer. In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), 665
- 10.1136/bmjopen-2021-050270 · DOI
- 10.1109/IAICT50021.2020.9172011 · DOI
- 10.1016/j.gie.2019.03.028 · DOI
- Tedford A., 2015, Rolling out our new editorial system: EVISE®: Find out how the new system will help reviewers streamline their workload
- 10.1177/09610006221079345 · DOI
- Thelwall M. Kousha K. Abdoli M. Stuart E. Makita M. Wilson P. &Cancellieri M.(2022).Predicting article quality scores with machine learning: The UK Research Excellence Framework.arXiv preprint arXiv:2212.05415. · DOI
- 10.1108/OIR-11-2019-0347 · DOI · Paper page
- 10.1002/asi.23040 · DOI · Paper page
- 10.1016/j.knosys.2018.05.001 · DOI · Paper page
- Wang K. &Wan X.(2018).Sentiment analysis of peer review texts for scholarly papers. In the 41st international ACM SIGIR Conference on Research & Development in Information Retrieval (pp. 175–184). · DOI · Paper page
- 10.1016/j.joi.2021.101174 · DOI
- 10.4033/iee.2010.3.3.c · DOI
- 10.1629/uksg.407/ · doi:10.1629/uksg.407/
- 10.1007/s11192-020-03488-4 · DOI · Paper page
- 10.1016/j.emj.2015.12.004 · DOI
- 10.1007/s11192-022-04354-1 · doi:10.1007/s11192-022-04354-1
- 10.1108/OIR-01-2019-0027 · DOI
Cited by
AI and its implications for research in higher education: a critical dialogue
Butson, R., & Spronken-Smith, R. (2024) · Higher Education Research & Development · doi:10.1080/07294360.2023.2280200 · Cited by 130
Exploring the Impact of Generative AI on Peer Review: Insights from Journal Reviewers
Ebadi, S., Nejadghanbar, H., Salman, A. R., & Khosravi, H. (2025) · Journal of Academic Ethics · doi:10.1007/s10805-025-09604-4 · Cited by 44
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · arXiv · doi:10.48550/arxiv.2406.12708 · Cited by 34
“Research has also delved into the operational aspects, such as reviewer assignments [39, 30, 40] and author rebuttals [41], identifying areas for improvement in transparency, fairness, and accountability [2].”
BackgroundAgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.70 · Cited by 218
AI-powered peer review needs human supervision
Seghier, M. L. (2025) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-09-2024-0132 · Cited by 18
How Artificial Intelligence Differs From Humans in Peer Review
Joachim, M. V., Dodson, T. B., & Laviv, A. (2025) · Journal of Oral and Maxillofacial Surgery · doi:10.1016/j.joms.2025.03.015 · Cited by 22
Large language models in peer review: challenges and opportunities
Sun, Z. (2025) · Scientometrics · doi:10.1007/s11192-025-05440-w · Cited by 15
Personal experience with AI-generated peer reviews: a case study
Lo Vecchio, N. (2025) · Research Integrity and Peer Review · doi:10.1186/s41073-025-00161-3 · Cited by 15
Ng, J. Y., Krishnamurthy, M., Deol, G., Al-Khafaji, W. A.-Z., Balaji, V., Abebe, M., Adhvaryu, J., Karrthik, T., Mohanakanthan, P., Vellaparambil, A., et al. (2025) · Research Integrity and Peer Review · doi:10.1186/s41073-025-00178-8 · Cited by 14
“[43] Our study aligns with these findings, where EiCs recognized AICs’ utility in performing routine tasks such as language checks and plagiarism screening.”
MARG: Multi-Agent Review Generation for Scientific Papers
D'Arcy, M., Hope, T., Birnbaum, L., & Downey, D. (2024) · arXiv · doi:10.48550/arxiv.2401.04259 · Cited by 9
Exploring the change in scientific readability following the release of ChatGPT
Alsudais, A. (2025) · Journal of Informetrics · doi:10.1016/j.joi.2025.101679 · Cited by 9
Kousha, K., & Thelwall, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05601-5 · Cited by 10
Self-citationAli, S. I., & Shaban, M. (2025) · International Nursing Review · doi:10.1111/inr.70100 · Cited by 6
The inevitable future of peer review: Human and AI integrated peer review system
Shah, F. A., & Jawaid, S. A. (2025) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.41.4.11941 · Cited by 5
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
Thakkar, N., Yuksekgonul, M., Silberg, J., Garg, A., Peng, N., Sha, F., Yu, R., Vondrick, C., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.09737 · Cited by 149
“Other studies have shown the potential of LLMs to make the entire review pipeline more efficient across various stages [32, 33, 34, 35] such as writing manuscripts [36], initial quality control [37, 38, 27], and even providing AI-generated instructions for how to write reviews [39].”
AI4Research: a survey of artificial intelligence for scientific research
Chen, Q., Yang, M., Qin, L., Liu, J., Yan, Z., Guan, J., Peng, D., Ji, Y., Li, H., Hu, M., et al. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-5000-y · Cited by 4
Science should be machine-readable
Booeshaghi, A. S., Luebbert, L., & Pachter, L. (2026) · doi:10.64898/2026.01.30.702911 · Cited by 3
Large Language Models as Peer Reviewers: Prompt Sensitivity and Model-Dependent Reproducibility
Erturk, S. M., & Durmaz, M. (2026) · Academic Radiology · doi:10.1016/j.acra.2026.04.046 · Cited by 5
Patel, R., Shultz, C., Ollivier, M., & Wascher, D. C. (2026) · The American Journal of Sports Medicine · doi:10.1177/03635465261463006 · Cited by 3
Ali, Ș., Oprea, S., & Bâra, A. (2026) · Systems Research and Behavioral Science · doi:10.1002/sres.70163
Unraveling the Ai2 Asta Scholarly Research Assistant Citation System
Orduña-Malea, E., & Lopezosa, C. (2025) · Revista Panamericana de Comunicación · doi:10.21555/rpc.v7i2.3675
Metadata
- Authors
- Kousha, K., & Thelwall, M.
- Affiliations
- Statistical Cybermetrics and Research Evaluation Group University of Wolverhampton Wolverhampton UK
- Venue
- Learned Publishing
- Publisher
- Wiley
- Published
- 2023-08-08
- Volume
- 37
- Issue
- 1
- Pages
- 4-12
- ISSN
- 0953-1513
- 1741-4857
- Cited by
- 151
- References
- 11
- Language
- en
- Funders
- Research England
- Scottish Funding Council
- Higher Education Funding Council for Wales
- Department for the Economy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.