A multi-disciplinary perspective on emergent and future innovations in peer review
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
8 Social Hierarchy: The Self‐Reinforcing Nature of Power and Status
Magee, J. C., & Galinsky, A. D. (2008) · Academy of Management Annals · doi:10.5465/19416520802211628 · Cited by 2,385
Where Is Current Research on Blockchain Technology?—A Systematic Review
Yli-Huumo, J., Ko, D., Choi, S., Park, S., & Smolander, K. (2016) · PLOS ONE · doi:10.1371/journal.pone.0163477 · Cited by 2,317
Bornmann, L., & Mutz, R. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23329 · Cited by 1,576
Peer-review practices of psychological journals: The fate of published articles, submitted again
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011183 · Cited by 1,079
Publication prejudices: An experimental study of confirmatory bias in the peer review system
Mahoney, M. J. (1977) · Cognitive Therapy and Research · doi:10.1007/bf01173636 · Cited by 1,013
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
Larsen, P. O., & von Ins, M. (2010) · Scientometrics · doi:10.1007/s11192-010-0202-z · Cited by 964
Peer Review: A Flawed Process at the Heart of Science and Journals
Smith, R. (2006) · Journal of the Royal Society of Medicine · doi:10.1177/014107680609900414 · Cited by 880
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716
Reviewer bias in single- versus double-blind peer review
Tomkins, A., Zhang, M., & Heavlin, W. D. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1707323114 · Cited by 605
The Academic, Economic and Societal Impacts of Open Access: An Evidence-Based Review
Tennant, J. P., Waldner, F., Jacques, D. C., Masuzzo, P., Collister, L. B., & Hartgerink, C. H. J. (2016) · F1000Research · doi:10.12688/f1000research.8460.3 · Cited by 560
Self-citationDouble-blind review favours increased representation of female authors
BUDDEN, A., TREGENZA, T., AARSSEN, L., KORICHEVA, J., LEIMU, R., & LORTIE, C. (2008) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2007.07.008 · Cited by 552
Deep impact: unintended consequences of journal rank
Brembs, B., Button, K., & Munafò, M. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00291 · Cited by 441
Why has the number of scientific retractions increased?
Steen, R. G., Casadevall, A., & Fang, F. C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0068397 · Cited by 415
van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999) · BMJ · doi:10.1136/bmj.318.7175.23 · Cited by 415
Peer review in a changing world: An international study measuring the attitudes of researchers
Mulligan, A., Hall, L., & Raphael, E. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22798 · Cited by 393
Gender bias in scholarly peer review
Helmer, M., Schottdorf, M., Neef, A., & Battaglia, D. (2017) · eLife · doi:10.7554/elife.21718 · Cited by 380
Gøtzsche, P. C. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90017-2 · Cited by 339
Godlee, F., Gale, C. R., & Martyn, C. N. (1998) · JAMA · doi:10.1001/jama.280.3.237 · Cited by 333
Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine
Goodman, S. N., Berlin, J., Fletcher, S. W., & Fletcher, R. H. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-121-1-199407010-00003 · Cited by 324
Editorial peer review for improving the quality of reports of biomedical studies
Jefferson, T., Rudin, M., Brodney Folse, S., & Davidoff, F. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000016.pub3 · Cited by 312
Does Masking Author Identity Improve Peer Review Quality?: A Randomized Controlled Trial
Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., Rennie, D., & and the PEER Investigators (1998) · JAMA · doi:10.1001/jama.280.3.240 · Cited by 278
The Effects of Blinding on the Quality of Peer Review: A Randomized Trial
McNutt, R. A. (1990) · JAMA · doi:10.1001/jama.1990.03440100079012 · Cited by 261
Open peer review: a randomised controlled trial
Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000) · British Journal of Psychiatry · doi:10.1192/bjp.176.1.47 · Cited by 261
Effect of Blinded Peer Review on Abstract Acceptance
Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., Hachinski, V. C., Gibbons, R. J., Gardner, T. J., & Krumholz, H. M. (2006) · JAMA · doi:10.1001/jama.295.14.1675 · Cited by 250
Measuring the quality of editorial peer review.
Jefferson, T. (2002) · JAMA · doi:10.1001/jama.287.21.2786 · Cited by 230
Bruce, R., Chauvin, A., Trinquart, L., Ravaud, P., & Boutron, I. (2016) · BMC Medicine · doi:10.1186/s12916-016-0631-5 · Cited by 218
D. Chambers, C., 1 Cardiff University Brain Research Imaging Centre, S. O. P., Feredoes, E., D. Muthukumaraswamy, S., & J. Etchells, P. (2014) · AIMS Neuroscience · doi:10.3934/neuroscience.2014.1.4 · Cited by 213
Single-blind vs Double-blind Peer Review in the Setting of Author Prestige
Okike, K., Hug, K. T., Kocher, M. S., & Leopold, S. S. (2016) · JAMA · doi:10.1001/jama.2016.11014 · Cited by 192
Making Reviewers Visible: Openness, Accountability, and Credit
Godlee, F. (2002) · JAMA · doi:10.1001/jama.287.21.2762 · Cited by 167
Baxt, W. G., Waeckerle, J. F., Berlin, J. A., & Callaham, M. L. (1998) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(98)70006-x · Cited by 141
Let’s make peer review scientific
Rennie, D. (2016) · Nature · doi:10.1038/535031a · Cited by 111
Fox, C. W., Albert, A. Y. K., & Vines, T. H. (2017) · Research Integrity and Peer Review · doi:10.1186/s41073-017-0027-x · Cited by 109
Reviewer Fatigue? Why Scholars Decline to Review their Peers’ Work
Breuning, M., Backstrom, J., Brannon, J., Gross, B. I., & Widmeier, M. (2015) · Political Science and Politics · doi:10.1017/s1049096515000827 · Cited by 89
Pierie, J.-P. E., Walvoort, H. C., & Overbeke, A. J. P. (1996) · The Lancet · doi:10.1016/s0140-6736(96)05016-7 · Cited by 84
Can editors save peer review from peer reviewers?
D’Andrea, R., & O’Dwyer, J. P. (2017) · PLOS ONE · doi:10.1371/journal.pone.0186111 · Cited by 80
Collaborative problem solving: a study of MathOverflow
Tausczik, Y. R., Kittur, A., & Kraut, R. E. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531690 · Cited by 78
Patel, J. (2014) · BMC Medicine · doi:10.1186/s12916-014-0128-z · Cited by 75
Closed versus open reviewing of journal manuscripts: how far do comments differ in language use?
Bornmann, L., Wolf, M., & Daniel, H.-D. (2012) · Scientometrics · doi:10.1007/s11192-011-0569-5 · Cited by 49
Peerage of Science: will it work?
Hettyey, A., Griggio, M., Mann, M., Raveh, S., Schaedelin, F. C., Thonhauser, K. E., Thoß, M., van Dongen, W. F. D., White, J., Zala, S. M., et al. (2012) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2012.01.005 · Cited by 20
Morrison, J. (2006) · Medical Education · doi:10.1111/j.1365-2929.2006.02573.x · Cited by 14
Cited by
Improving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2019) · Journal of Neuroscience Research · doi:10.1002/jnr.24340 · Cited by 83
Overburdening of peer reviewers: a multi-stakeholder perspective on causes and effects
Severin, A., & Chataway, J. (2021) · Learned Publishing · doi:10.1002/leap.1392 · Cited by 73
Pranić, S. M., Malički, M., Marušić, S. L., Mehmani, B., & Marušić, A. (2021) · Learned Publishing · doi:10.1002/leap.1344 · Cited by 19
PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality
Chen, Z., Yang, Z., Ma, Y., Backes, M., & Zhang, Y. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1170
Metadata
- Authors
- Tennant, J. P., Dugan, J. M., Graziotin, D., Jacques, D. C., Waldner, F., Mietchen, D., Elkhatib, Y., B. Collister, L., Pikas, C. K., Crick, T., Masuzzo, P., Caravaggi, A., Berg, D. R., Niemeyer, K. E., Ross-Hellauer, T., Mannheimer, S., Rigling, L., Katz, D. S., Greshake Tzovaras, B., Pacheco-Mendoza, J., Fatima, N., Poblet, M., Isaakidis, M., Irawan, D. E., Renaut, S., Madan, C. R., Matthias, L., Nørgaard Kjær, J., O'Donnell, D. P., Neylon, C., Kearns, S., Selvaraju, M., & Colomb, J.
- Venue
- F1000Research
- Publisher
- F1000 Research Ltd
- Published
- 2017-11-01
- Volume
- 6
- Pages
- 1151
- ISSN
- 2046-1402 (ISSN-L)
- OpenAlex
- W4238684574
- Cited by
- 38
- References
- 42
- FWCI
- 19.43
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.