The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Artificial intelligence to support publishing and peer review: A summary and review
Abstract
Technology is being developed to support the peer review processes of journals, conferences, funders, universities, and national research evaluations. This literature and software summary discusses the partial or complete automation of several publishing‐related tasks: suggesting appropriate journals for an article, providing quality control for submitted papers, finding suitable reviewers for submitted papers or grant proposals, reviewing, and review evaluation. It also discusses attempts to estimate article quality from peer review text and scores as well as from post‐publication scores but not from bibliometric data. The literature and existing examples of working technology show that automation is useful for helping to find reviewers and there is good evidence that it can sometimes help with initial quality control of submitted manuscripts. Much other software supporting publishing and editorial work exists and is being used, but without published academic evaluations of its efficacy. The value of artificial intelligence (AI) to support reviewing has not been clearly demonstrated yet, however. Finally, whilst peer review text and scores can theoretically have value for post‐publication research assessment, it is not yet widely enough available to be a practical evidence source for systematic automation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24750 · Cited by 919
Hosseini, M., & Horbach, S. P. J. M. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00133-5 · Cited by 328 · cited 1×
“…to keep the manuscripts assessed con fi dential; as such, papers under assessment should not be uploaded to LLMs because they may be saved and incorporated into responses to future questions for other users (Hosseini & Horbach, 2023), or may even grant the LLM owner the right to repurpose content.”
BackgroundChecco, A., Bracciale, L., Loreti, P., Pinfield, S., & Bianchi, G. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00703-8 · Cited by 286
Will ChatGPT get you caught? Rethinking of Plagiarism Detection
Khalil, M., & Er, E. (2023) · doi:10.35542/osf.io/fnh48 · Cited by 259
Can We Automate Scientific Reviewing?
Yuan, W., Liu, P., & Neubig, G. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12862 · Cited by 96
The Rigor and Transparency Index Quality Metric for Assessing Biological and Medical Science Methods
Menke, J., Roelandse, M., Ozyurt, B., Martone, M., & Bandrowski, A. (2020) · iScience · doi:10.1016/j.isci.2020.101698 · Cited by 90
AI peer reviewers unleashed to ease publishing grind.
Heaven, D. (2018) · Nature · doi:10.1038/d41586-018-07245-9 · Cited by 75 · cited 1×
“…citations in the manuscript text match the reference list and vice versa (cid:129) Multipurpose manuscript evaluation : Other AI-assisted tools assess multiple quality control aspects of manuscripts such as Frontiers AI Review Assistant 19 or UNSILO Manuscript Evaluation 20 (see also Heaven, 2018).”
BackgroundAspect-based Sentiment Analysis of Scientific Reviews
Chakraborty, S., Goyal, P., & Mukherjee, A. (2020) · Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 · doi:10.1145/3383583.3398541 · Cited by 63
DeepSentiPeer: Harnessing Sentiment in Review Texts to Recommend Peer Review Decisions
Ghosal, T., Verma, R., Ekbal, A., & Bhattacharyya, P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1106 · Cited by 62
Nuijten, M. B., & Polanin, J. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1408 · Cited by 56 · cited 1×
“(cid:129) Automated statistical checking : StatReviewer 14 checks manuscripts against standardized reporting guidelines and StatCheck 15 detects some statistical errors in the submitted works (Nuijten & Polanin, 2020).”
Background
Research works printed but not linked: 51
- 10.1109/CCAA.2018.8777730 · doi:10.1109/ccaa.2018.8777730
- 10.1371/journal.pone.0005910 · DOI · Paper page
- American Journal Experts. (2018).Peer review: How we found 15 million hours of lost time.www.aje.com/en/arc/peer-review-process-15-million-hours-lost-time/
- 10.1002/asi.23334 · DOI
- 10.1016/j.joi.2012.12.003 · DOI
- 10.1016/j.joi.2015.03.001 · DOI
- Cabanac G. Labbé C. &Magazinov A.(2021).Tortured phrases: A dubious writing style emerging in science. Evidence of critical issues affecting established journals.arXiv preprint arXiv:2107.06751. · arXiv
- Charlin L. &Zemel R.(2013).The Toronto paper matching system: An automated paper‐reviewer assignment system.https://openreview.net/pdf?id=caynafZAnBafx
- 10.1038/d41586-019-01517-8 · DOI
- 10.1002/asi.23548 · DOI
- 10.2196/12957 · DOI
- Fiez T. Shah N. &Ratliff L.(2020).A SUPER* algorithm to optimize paper bidding in peer review. InConference on Uncertainty in Artificial Intelligence(pp. 580–589). PMLR.http://proceedings.mlr.press/v124/fiez20a/fiez20a.pdf
- 10.1109/JCDL.2019.00039 · DOI
- 10.1109/JCDL52503.2021.00021 · DOI
- 10.1007/s11192-020-03519-0 · DOI
- 10.3174/ajnr.A4234 · DOI
- 10.6087/kcse.137 · DOI
- 10.1371/journal.pone.0166387 · DOI · Paper page
- Li X. &Thelwall M.(2012).F1000 Mendeley and traditional bibliometric indicators.Proceedings of the 17th International Conference on Science and Technology Indicators(Vol. 2 pp. 451–551).
- 10.1016/j.procs.2013.09.144 · DOI
- Lin J. Song J. Zhou Z. &Shi X.(2021).Automated scholarly paper review: Possibility and challenges.arXiv preprint arXiv:2111.07533. · arXiv
- 10.1016/j.compind.2015.11.001 · doi:10.1016/j.compind.2015.11.001
- 10.1007/s11192-022-04389-4 · DOI
- 10.1162/qss_a_00156 · DOI
- 10.1007/s11192-013-0993-9 · DOI
- 10.1371/journal.pone.0184711 · DOI
- 10.1007/s11192-017-2399-6 · DOI
- 10.1108/AJIM-06-2018-0153 · DOI
- 10.1002/asi.24568 · DOI
- 10.1016/j.neucom.2020.11.004 · DOI
- 10.1016/j.knosys.2019.105092 · DOI
- 10.1145/2979672 · DOI
- 10.1007/s11948-015-9625-5 · DOI · Paper page
- Roberts J., 2020, pReview: The artificially intelligent conference reviewer. In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), 665
- 10.1136/bmjopen-2021-050270 · DOI
- 10.1109/IAICT50021.2020.9172011 · DOI
- 10.1016/j.gie.2019.03.028 · DOI
- Tedford A., 2015, Rolling out our new editorial system: EVISE®: Find out how the new system will help reviewers streamline their workload
- 10.1177/09610006221079345 · DOI
- Thelwall M. Kousha K. Abdoli M. Stuart E. Makita M. Wilson P. &Cancellieri M.(2022).Predicting article quality scores with machine learning: The UK Research Excellence Framework.arXiv preprint arXiv:2212.05415. · DOI
- 10.1108/OIR-11-2019-0347 · DOI
- 10.1002/asi.23040 · DOI
- 10.1016/j.knosys.2018.05.001 · DOI · Paper page
- Wang K. &Wan X.(2018).Sentiment analysis of peer review texts for scholarly papers. In the 41st international ACM SIGIR Conference on Research & Development in Information Retrieval (pp. 175–184). · DOI
- 10.1016/j.joi.2021.101174 · DOI
- 10.4033/iee.2010.3.3.c · DOI
- 10.1629/uksg.407/ · doi:10.1629/uksg.407/
- 10.1007/s11192-020-03488-4 · DOI · Paper page
- 10.1016/j.emj.2015.12.004 · DOI
- 10.1007/s11192-022-04354-1 · doi:10.1007/s11192-022-04354-1
- 10.1108/OIR-01-2019-0027 · DOI
Cited by
AI and its implications for research in higher education: a critical dialogue
Butson, R., & Spronken-Smith, R. (2024) · Higher Education Research & Development · doi:10.1080/07294360.2023.2280200 · Cited by 130
Exploring the Impact of Generative AI on Peer Review: Insights from Journal Reviewers
Ebadi, S., Nejadghanbar, H., Salman, A. R., & Khosravi, H. (2025) · Journal of Academic Ethics · doi:10.1007/s10805-025-09604-4 · Cited by 44
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · arXiv · doi:10.48550/arxiv.2406.12708 · Cited by 34
“Research has also delved into the operational aspects, such as reviewer assignments [39, 30, 40] and author rebuttals [41], identifying areas for improvement in transparency, fairness, and accountability [2].”
BackgroundAgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.70 · Cited by 218
AI-powered peer review needs human supervision
Seghier, M. L. (2025) · Journal of Information, Communication and Ethics in Society · doi:10.1108/jices-09-2024-0132 · Cited by 18
Metadata
- Authors
- Kousha, K., & Thelwall, M.
- Affiliations
- Statistical Cybermetrics and Research Evaluation Group University of Wolverhampton Wolverhampton UK
- Venue
- Learned Publishing
- Publisher
- Wiley
- Published
- 2023-08-08
- Volume
- 37
- Issue
- 1
- Pages
- 4-12
- ISSN
- 0953-1513
- 1741-4857
- Cited by
- 151
- References
- 10
- Language
- en
- Funders
- Research England
- Scottish Funding Council
- Higher Education Funding Council for Wales
- Department for the Economy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Artificial intelligence to support publishing and peer review: A summary and review. https://app.cochat.ai/papers/10.1002/leap.1570
- The paper
- Kousha, K., & Thelwall, M. (2024). Artificial intelligence to support publishing and peer review: A summary and review. Learned Publishing. https://doi.org/10.1002/leap.1570
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.