The state of the art in peer review
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Scholarly communication is in a perpetual state of disruption. Within this, peer review of research articles remains an essential part of the formal publication process, distinguishing it from virtually all other modes of communication. In the last several years, there has been an explosive wave of innovation in peer review research, platforms, discussions, tools and services. This is largely coupled with the ongoing and parallel evolution of scholarly communication as it adapts to rapidly changing environments, within what is widely considered as the 'open research' or 'open science' movement. Here, we summarise the current ebb and flow around changes to peer review and consider its role in a modern digital research and communications infrastructure and suggest why uptake of new models of peer review appears to have been so low compared to what is often viewed as the 'traditional' method of peer review. Finally, we offer some insight into the potential futures of scholarly peer review and consider what impacts this might have on the broader scholarly research ecosystem. In particular, we focus on the key traits of certification and reputation, moderation and quality control and engagement incentives, and discuss how these interact with socio-technical aspects of peer review and academic culture.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
Peer Review: A Flawed Process at the Heart of Science and Journals
Smith, R. (2006) · Journal of the Royal Society of Medicine · doi:10.1177/014107680609900414 · Cited by 880
Reviewer bias in single- versus double-blind peer review
Tomkins, A., Zhang, M., & Heavlin, W. D. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1707323114 · Cited by 605
Gender bias in scholarly peer review
Helmer, M., Schottdorf, M., Neef, A., & Battaglia, D. (2017) · eLife · doi:10.7554/elife.21718 · Cited by 380 · cited 1×
“2017), despite substantial evidence for a wide range of inherent biases (Lee et al. 2013; Helmer et al. 2017; Kuehn 2017; Tennant 2017; Tomkins, Zhang and Heavlin 2017; Iezzoni 2018).”
BackgroundManuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine
Goodman, S. N., Berlin, J., Fletcher, S. W., & Fletcher, R. H. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-121-1-199407010-00003 · Cited by 324
Peer review: still king in the digital age
NICHOLAS, D., WATKINSON, A., JAMALI, H. R., HERMAN, E., TENOPIR, C., VOLENTINE, R., ALLARD, S., & LEVINE, K. (2015) · Learned Publishing · doi:10.1087/20150104 · Cited by 242
What is open peer review? A systematic review
Ross-Hellauer, T. (2017) · F1000Research · doi:10.12688/f1000research.11369.1 · Cited by 232
Bruce, R., Chauvin, A., Trinquart, L., Ravaud, P., & Boutron, I. (2016) · BMC Medicine · doi:10.1186/s12916-016-0631-5 · Cited by 218 · cited 1×
“2004); The length of time taken for the peer review process (Bornmann and Daniel 2010; Lyman 2013); That valuable contextual information is often lost as review reports remain unpublished (Walker and Rocha da Silva 2015; Ross-Hellauer 2017a); What the best operational processes should be for different research communi…”
BackgroundPrestigious Science Journals Struggle to Reach Even Average Reliability
Brembs, B. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00037 · Cited by 213 · cited 2×
“It also has the potential to help shape amore rigorous method of scholarly evaluation and assessment that could help to simultaneously resolve issues to do with traditional journal-based methods of communication and ranking, something that is critically required for the modern academy (Brembs, Button, and Munafò 2013;…”
MethodLet’s make peer review scientific
Rennie, D. (2016) · Nature · doi:10.1038/535031a · Cited by 111 · cited 1×
“The key to maintaining this momentum will be sustained engagement with the different stakeholders to develop a more holistic framework of peer review, in which risk perception is minimised while the advantages are made much more explicit and evidence-based (Rennie 2016).”
BackgroundFox, C. W., Albert, A. Y. K., & Vines, T. H. (2017) · Research Integrity and Peer Review · doi:10.1186/s41073-017-0027-x · Cited by 109
Reviewer Fatigue? Why Scholars Decline to Review their Peers’ Work
Breuning, M., Backstrom, J., Brannon, J., Gross, B. I., & Widmeier, M. (2015) · Political Science and Politics · doi:10.1017/s1049096515000827 · Cited by 89
Can editors save peer review from peer reviewers?
D’Andrea, R., & O’Dwyer, J. P. (2017) · PLOS ONE · doi:10.1371/journal.pone.0186111 · Cited by 80
Cited by
The limitations to our understanding of peer review
Tennant, J. P., & Ross-Hellauer, T. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-020-00092-1 · Cited by 384
“There is no doubt that editors will continue to play a prominent role here in terms of arbitration, quality control, and encouraging engagement while fostering a community environment [76].”
BackgroundSelf-citationThe evaluation of scholarship in academic promotion and tenure processes: Past, present, and future
Schimanski, L. A., & Alperin, J. P. (2018) · F1000Research · doi:10.12688/f1000research.16493.1 · Cited by 342
Checco, A., Bracciale, L., Loreti, P., Pinfield, S., & Bianchi, G. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00703-8 · Cited by 286
A review on blockchain technology and blockchain projects fostering open science
Leible, S., Schlager, S., Schubotz, M., & Gipp, B. (2019) · Frontiers in Blockchain · doi:10.3389/fbloc.2019.00016 · Cited by 134
Overburdening of peer reviewers: a multi-stakeholder perspective on causes and effects
Severin, A., & Chataway, J. (2021) · Learned Publishing · doi:10.1002/leap.1392 · Cited by 73
“The traditional model of peer review has been recognized to put increasing strain on the academic community (Alberts et al., 2008; Tennant, 2018).”
Background
Metadata
- Authors
- Tennant, J. P.
- Venue
- FEMS Microbiology Letters
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-08-23
- Volume
- 365
- Issue
- 19
- ISSN
- 0378-1097 (ISSN-L)
- 1574-6968
- OpenAlex
- W2888173901
- Cited by
- 164
- References
- 13
- FWCI
- 82.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.