CoChat

Artificial intelligence to support publishing and peer review: A summary and review

Kousha, Kayvan, Thelwall, Mike · 2024 · Learned Publishing
Cited by 151

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

Abstract

Technology is being developed to support the peer review processes of journals, conferences, funders, universities, and national research evaluations. This literature and software summary discusses the partial or complete automation of several publishing‐related tasks: suggesting appropriate journals for an article, providing quality control for submitted papers, finding suitable reviewers for submitted papers or grant proposals, reviewing, and review evaluation. It also discusses attempts to estimate article quality from peer review text and scores as well as from post‐publication scores but not from bibliometric data. The literature and existing examples of working technology show that automation is useful for helping to find reviewers and there is good evidence that it can sometimes help with initial quality control of submitted manuscripts. Much other software supporting publishing and editorial work exists and is being used, but without published academic evaluations of its efficacy. The value of artificial intelligence (AI) to support reviewing has not been clearly demonstrated yet, however. Finally, whilst peer review text and scores can theoretically have value for post‐publication research assessment, it is not yet widely enough available to be a practical evidence source for systematic automation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/leap.1570

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 11 references

Research works printed but not linked: 51

  1. 10.1109/CCAA.2018.8777730 · doi:10.1109/ccaa.2018.8777730
  2. 10.1371/journal.pone.0005910 · DOI · Paper page
  3. American Journal Experts. (2018).Peer review: How we found 15 million hours of lost time.www.aje.com/en/arc/peer-review-process-15-million-hours-lost-time/
  4. 10.1002/asi.23334 · DOI
  5. 10.1016/j.joi.2012.12.003 · DOI
  6. 10.1016/j.joi.2015.03.001 · DOI · Paper page
  7. Cabanac G. Labbé C. &Magazinov A.(2021).Tortured phrases: A dubious writing style emerging in science. Evidence of critical issues affecting established journals.arXiv preprint arXiv:2107.06751. · arXiv
  8. Charlin L. &Zemel R.(2013).The Toronto paper matching system: An automated paper‐reviewer assignment system.https://openreview.net/pdf?id=caynafZAnBafx
  9. 10.1038/d41586-019-01517-8 · DOI
  10. 10.1002/asi.23548 · DOI
  11. 10.2196/12957 · DOI
  12. Fiez T. Shah N. &Ratliff L.(2020).A SUPER* algorithm to optimize paper bidding in peer review. InConference on Uncertainty in Artificial Intelligence(pp. 580–589). PMLR.http://proceedings.mlr.press/v124/fiez20a/fiez20a.pdf
  13. 10.1109/JCDL.2019.00039 · DOI
  14. 10.1109/JCDL52503.2021.00021 · DOI
  15. 10.1007/s11192-020-03519-0 · DOI
  16. 10.3174/ajnr.A4234 · DOI
  17. 10.6087/kcse.137 · DOI
  18. 10.1371/journal.pone.0166387 · DOI · Paper page
  19. Li X. &Thelwall M.(2012).F1000 Mendeley and traditional bibliometric indicators.Proceedings of the 17th International Conference on Science and Technology Indicators(Vol. 2 pp. 451–551).
  20. 10.1016/j.procs.2013.09.144 · DOI · Paper page
  21. Lin J. Song J. Zhou Z. &Shi X.(2021).Automated scholarly paper review: Possibility and challenges.arXiv preprint arXiv:2111.07533. · arXiv
  22. 10.1016/j.compind.2015.11.001 · doi:10.1016/j.compind.2015.11.001
  23. 10.1007/s11192-022-04389-4 · DOI
  24. 10.1162/qss_a_00156 · DOI · Paper page
  25. 10.1007/s11192-013-0993-9 · DOI
  26. 10.1371/journal.pone.0184711 · DOI · Paper page
  27. 10.1007/s11192-017-2399-6 · DOI
  28. 10.1108/AJIM-06-2018-0153 · DOI
  29. 10.1002/asi.24568 · DOI · Paper page
  30. 10.1016/j.neucom.2020.11.004 · DOI
  31. 10.1016/j.knosys.2019.105092 · DOI
  32. 10.1145/2979672 · DOI · Paper page
  33. 10.1007/s11948-015-9625-5 · DOI · Paper page
  34. Roberts J., 2020, pReview: The artificially intelligent conference reviewer. In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), 665
  35. 10.1136/bmjopen-2021-050270 · DOI
  36. 10.1109/IAICT50021.2020.9172011 · DOI
  37. 10.1016/j.gie.2019.03.028 · DOI
  38. Tedford A., 2015, Rolling out our new editorial system: EVISE®: Find out how the new system will help reviewers streamline their workload
  39. 10.1177/09610006221079345 · DOI
  40. Thelwall M. Kousha K. Abdoli M. Stuart E. Makita M. Wilson P. &Cancellieri M.(2022).Predicting article quality scores with machine learning: The UK Research Excellence Framework.arXiv preprint arXiv:2212.05415. · DOI
  41. 10.1108/OIR-11-2019-0347 · DOI · Paper page
  42. 10.1002/asi.23040 · DOI · Paper page
  43. 10.1016/j.knosys.2018.05.001 · DOI · Paper page
  44. Wang K. &Wan X.(2018).Sentiment analysis of peer review texts for scholarly papers. In the 41st international ACM SIGIR Conference on Research & Development in Information Retrieval (pp. 175–184). · DOI · Paper page
  45. 10.1016/j.joi.2021.101174 · DOI
  46. 10.4033/iee.2010.3.3.c · DOI
  47. 10.1629/uksg.407/ · doi:10.1629/uksg.407/
  48. 10.1007/s11192-020-03488-4 · DOI · Paper page
  49. 10.1016/j.emj.2015.12.004 · DOI
  50. 10.1007/s11192-022-04354-1 · doi:10.1007/s11192-022-04354-1
  51. 10.1108/OIR-01-2019-0027 · DOI

Cited by

Metadata

Authors
Kousha, K., & Thelwall, M.
Affiliations
Statistical Cybermetrics and Research Evaluation Group University of Wolverhampton Wolverhampton UK
Venue
Learned Publishing
Publisher
Wiley
Published
2023-08-08
Volume
37
Issue
1
Pages
4-12
ISSN
  • 0953-1513
  • 1741-4857
Cited by
151
References
11
Language
en
Funders
  • Research England
  • Scottish Funding Council
  • Higher Education Funding Council for Wales
  • Department for the Economy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.