The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
What is open peer review? A systematic review
Open accessAbstract
Background : “Open peer review” (OPR), despite being a major pillar of Open Science, has neither a standardized definition nor an agreed schema of its features and implementations. The literature reflects this, with numerous overlapping and contradictory definitions. While for some the term refers to peer review where the identities of both author and reviewer are disclosed to each other, for others it signifies systems where reviewer reports are published alongside articles. For others it signifies both of these conditions, and for yet others it describes systems where not only “invited experts” are able to comment. For still others, it includes a variety of combinations of these and other novel methods. Methods : Recognising the absence of a consensus view on what open peer review is, this article undertakes a systematic review of definitions of “open peer review” or “open review”, to create a corpus of 122 definitions. These definitions are systematically analysed to build a coherent typology of the various innovations in peer review signified by the term, and hence provide the precise technical definition currently lacking. Results : This quantifiable data yields rich information on the range and extent of differing definitions over time and by broad subject area. Quantifying definitions in this way allows us to accurately portray exactly how ambiguously the phrase “open peer review” has been used thus far, for the literature offers 22 distinct configurations of seven traits, effectively meaning that there are 22 different definitions of OPR in the literature reviewed. Conclusions : I propose a pragmatic definition of open peer review as an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the aims of Open Science, including making reviewer and author identities open, publishing review reports and enabling greater participation in the peer review process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Retracted Science and the Retraction Index
Fang, F. C., & Casadevall, A. (2011) · Infection and Immunity · doi:10.1128/iai.05661-11 · Cited by 405
RetractedMisconduct accounts for the majority of retracted scientific publications
Fang, F. C., Steen, R. G., & Casadevall, A. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212247109 · Cited by 1,232
A method for taxonomy development and its application in information systems
Nickerson, R. C., Varshney, U., & Muntermann, J. (2013) · European Journal of Information Systems · doi:10.1057/ejis.2012.26 · Cited by 1,143
Peer-review practices of psychological journals: The fate of published articles, submitted again
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011183 · Cited by 1,079
Publication prejudices: An experimental study of confirmatory bias in the peer review system
Mahoney, M. J. (1977) · Cognitive Therapy and Research · doi:10.1007/bf01173636 · Cited by 1,013
Peer Review: A Flawed Process at the Heart of Science and Journals
Smith, R. (2006) · Journal of the Royal Society of Medicine · doi:10.1177/014107680609900414 · Cited by 880
Dickersin, K. (1992) · JAMA · doi:10.1001/jama.1992.03480030052036 · Cited by 854
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
Open Science: One Term, Five Schools of Thought
Fecher, B., & Friesike, S. (2014) · Opening Science · doi:10.1007/978-3-319-00026-8_2 · Cited by 654
Ioannidis, J. P. A. (1998) · JAMA · doi:10.1001/jama.279.4.281 · Cited by 632
Double-blind review favours increased representation of female authors
BUDDEN, A., TREGENZA, T., AARSSEN, L., KORICHEVA, J., LEIMU, R., & LORTIE, C. (2008) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2007.07.008 · Cited by 552
The history of the peer-review process
Spier, R. (2002) · Trends in Biotechnology · doi:10.1016/s0167-7799(02)01985-6 · Cited by 445
Why has the number of scientific retractions increased?
Steen, R. G., Casadevall, A., & Fang, F. C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0068397 · Cited by 415
van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999) · BMJ · doi:10.1136/bmj.318.7175.23 · Cited by 415
Peer review in a changing world: An international study measuring the attitudes of researchers
Mulligan, A., Hall, L., & Raphael, E. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22798 · Cited by 393
Godlee, F., Gale, C. R., & Martyn, C. N. (1998) · JAMA · doi:10.1001/jama.280.3.237 · Cited by 333
US and Non-US Submissions: An Analysis of Reviewer Bias
Link, A. M. (1998) · JAMA · doi:10.1001/jama.280.3.246 · Cited by 318
A multi-disciplinary perspective on emergent and future innovations in peer review
Tennant, J. P., Dugan, J. M., Graziotin, D., Jacques, D. C., Waldner, F., Mietchen, D., Elkhatib, Y., B. Collister, L., Pikas, C. K., Crick, T., et al. (2017) · F1000Research · doi:10.12688/f1000research.12037.1 · Cited by 289
Self-citationDoes Masking Author Identity Improve Peer Review Quality?: A Randomized Controlled Trial
Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., Rennie, D., & and the PEER Investigators (1998) · JAMA · doi:10.1001/jama.280.3.240 · Cited by 278
Effects of training on quality of peer review: randomised controlled trial
Schroter, S., Black, N., Evans, S., Carpenter, J., Godlee, F., & Smith, R. (2004) · BMJ · doi:10.1136/bmj.38023.700775.ae · Cited by 268
“Excellence R Us”: university research and the fetishisation of excellence
Moore, S., Neylon, C., Paul Eve, M., Paul O’Donnell, D., & Pattinson, D. (2017) · Palgrave Communications · doi:10.1057/palcomms.2016.105 · Cited by 264
Peer review for journals: Evidence on quality control, fairness, and innovation
Armstrong, J. S. (1997) · Science and Engineering Ethics · doi:10.1007/s11948-997-0017-3 · Cited by 262
The Effects of Blinding on the Quality of Peer Review: A Randomized Trial
McNutt, R. A. (1990) · JAMA · doi:10.1001/jama.1990.03440100079012 · Cited by 261
Effect of Blinded Peer Review on Abstract Acceptance
Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., Hachinski, V. C., Gibbons, R. J., Gardner, T. J., & Krumholz, H. M. (2006) · JAMA · doi:10.1001/jama.295.14.1675 · Cited by 250
New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System
Travis, G. D. L., & Collins, H. M. (1991) · Science, Technology, & Human Values · doi:10.1177/016224399101600303 · Cited by 245
Manuscript Characteristics Which Influence Acceptance for Management and Social Science Journals.
Kerr, S., Tolliver, J., & Petree, D. (1977) · Academy of Management Journal · doi:10.2307/255467 · Cited by 245
The effects of blinding on the quality of peer review. A randomized trial
Mcnutt, R. (1990) · JAMA · doi:10.1001/jama.263.10.1371 · Cited by 216
Peer Review for Journals as it Stands Today—Part 1
CAMPANARIO, J. M. (1998) · Science Communication · doi:10.1177/1075547098019003002 · Cited by 195
Emerging trends in peer review—a survey
Walker, R., & Rocha da Silva, P. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00169 · Cited by 190
Kravitz, R. L., Franks, P., Feldman, M. D., Gerrity, M., Byrne, C., & Tierney, W. M. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010072 · Cited by 178
Peerless Science: Peer Review and U.S. Science Policy.
Higgins, A. C., Chubin, D. E., & Hackett, E. J. (1990) · Social Forces · doi:10.2307/2579681 · Cited by 175
The Effects of Blinding on Acceptance of Research Papers by Peer Review
Fisher, M. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520020069019 · Cited by 164
van Rooyen, S., Delamothe, T., & Evans, S. J. W. (2010) · BMJ · doi:10.1136/bmj.c5729 · Cited by 162
Defining and Characterizing Open Peer Review: A Review of the Literature
Ford, E. (2013) · Journal of Scholarly Publishing · doi:10.3138/jsp.44-4-001 · Cited by 159
Barriers to scientific contributions: The author's formula
Armstrong, J. S. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011201 · Cited by 158
Gender bias in the refereeing process
Tregenza, T. (2002) · Trends in Ecology & Evolution · doi:10.1016/s0169-5347(02)02545-4 · Cited by 147
The invisible hand of peer review
Harnad, S. R. (1998) · Nature · doi:10.1038/nature28029 · Cited by 144
Fitzpatrick, K. (2011) · New York University Press eBooks · doi:10.18574/nyu/9780814728970.001.0001 · Cited by 133
Fostering open science to research using a taxonomy and an eLearning portal
Pontika, N., Knoth, P., Cancellieri, M., & Pearce, S. (2015) · International Conference on Knowledge Management and Knowledge Technologies · doi:10.1145/2809563.2809571 · Cited by 118
Guardians of Science: Fairness and Reliability of Peer Review
Daniel, H. (1993) · Wiley eBooks · doi:10.1002/3527602208 · Cited by 116
The effects of blinding on the quality of peer review. A randomized trial.
McNutt, R. A., Evans, A. T., Fletcher, R. H., & Fletcher, S. W. (1990) · PubMed · Cited by 113
Kriegeskorte, N. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00079 · Cited by 110
Research: Successful Approaches
Horn, L. V. (2007) · Cited by 110
Casadevall, A., & Fang, F. C. (2014) · Infection and Immunity · doi:10.1128/iai.01530-13 · Cited by 104
Peer Review: Recent Experience and Future Directions
Ware, M. (2011) · New Review of Information Networking · doi:10.1080/13614576.2011.566812 · Cited by 101
Pöschl, U. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00033 · Cited by 96
Gender factors in reviewer recommendations for manuscript publication.
Lloyd, M. E. (1990) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.1990.23-539 · Cited by 90
Peerless science peer review and U.S. science policy
Chubin, D. E., & Hackett, E. J. (1990) · Cited by 86
Peer review: The current landscape and future trends
Jubb, M. (2016) · Learned Publishing · doi:10.1002/leap.1008 · Cited by 75
Bias in reporting clinical trials
Bardy, A. H. (1998) · British Journal of Clinical Pharmacology · doi:10.1046/j.1365-2125.1998.00759.x · Cited by 70
Herron, D. M. (2012) · Surgical Endoscopy · doi:10.1007/s00464-012-2171-1 · Cited by 70
Experience with NIH Peer Review: Researchers' Cynicism and Desire for Change
Gillespie, G. W., Chubin, D., & Kurzon, G. M. (1985) · Science Technology & Human Values · doi:10.1177/016224398501000306 · Cited by 40
New journal models and publishing perspectives in the evolving digital environment
Cassella, M., & Calvi, L. (2010) · IFLA Journal · doi:10.1177/0340035209359559 · Cited by 38
Cooperation between Referees and Authors Increases Peer Review Accuracy
Leek, J. T., Taub, M. A., & Pineda, F. J. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026895 · Cited by 35
Extending ArXiv.org to Achieve Open Peer Review and Publishing
Boldt, A. (2010) · Journal of Scholarly Publishing · doi:10.1353/scp.2011.0001 · Cited by 33
Use of politeness strategies in signed open peer review
Nobarany, S., & Booth, K. S. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23229 · Cited by 31
Bornmann, L., Herich, H., Joos, H., & Daniel, H. (2012) · Scientometrics · doi:10.1007/s11192-012-0731-8 · Cited by 31
Open peer review at four STEM journals: an observational overview
Ford, E. (2015) · F1000Research · doi:10.12688/f1000research.6005.1 · Cited by 28
Ernst, E., & Kienbacher, T. (1991) · Nature · doi:10.1038/352560b0 · Cited by 26
The eLife approach to peer review
Schekman, R., Watt, F., & Weigel, D. (2013) · eLife · doi:10.7554/elife.00799 · Cited by 25
The changing face of peer review
Hames, I. (2014) · Science Editing · doi:10.6087/kcse.2014.1.9 · Cited by 25
Open and transparent: the review process of the Semantic Web journal
Janowicz, K., & Hitzler, P. (2012) · Learned Publishing · doi:10.1087/20120107 · Cited by 24
Natural selection of academic papers
Perakakis, P., Taylor, M., Mazza, M., & Trachana, V. (2010) · Scientometrics · doi:10.1007/s11192-010-0253-1 · Cited by 22
Maintaining Live Discussion in Two-Stage Open Peer Review
Sandewall, E. (2011) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00009 · Cited by 22
Extending ArXiv.org to Achieve Open Peer Review and Publishing
Boldt, A. (2010) · Journal of Scholarly Publishing · doi:10.3138/jsp.42.2.238 · Cited by 22
Authors and reviewers who suffer from confirmatory bias
García, J. A., Sánchez, R. M. R., & Fdez-Valdivia, J. (2016) · Scientometrics · doi:10.1007/s11192-016-2079-y · Cited by 18
Peer review and journal modelsPreprint
Dall'Aglio, P. (2006) · arXiv (Cornell University) · doi:10.48550/arxiv.physics/0608307 · Cited by 17
Vernacular and vehicular language
Cronin, B. (2009) · J. Assoc. Inf. Sci. Technol. · doi:10.1002/asi.21010 · Cited by 13
Early adopters of ORCID functionality enabling recognition of peer review: Two brief case studies
Hanson, B., Lawrence, R., Meadows, A., & Paglione, L. (2016) · Learned Publishing · doi:10.1002/leap.1004 · Cited by 12
A brief survey on peer review in scholarly communication
Nicholson, J. M., & Alperín, J. P. (2016) · The Winnower · Cited by 11
Day, C. (2015) · Physics Today · doi:10.1063/pt.5.010330 · Cited by 3
Review Of Definitions Of Open Peer Review In The Scholarly Literature 2016
Ross‐Hellauer, T. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.438024 · Cited by 2
Self-citationOpenaire'S Experiments In Open Peer Review / Report
OpenAIRE (2016) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.154647 · Cited by 2
Research works printed but not linked: 7
- T Bloom, 2017, Referee Report For: What is open peer review? A systematic review [version 1; referees: 1 approved, 3 approved with reservations]., F1000Res., 6, 588, 10.5256/f1000research.12273.r22301 · doi:10.5256/f1000research.12273.r22301
- D Chubin, 1990, Peerless Science: Peer Review and US Science Policy.
- 2016, About | The EMBO Journal [WWW Document]
- K Fitzpatrick, 2012, Open Review, A Study of Contexts and Practices
- S Kaplan, 2015, Major Publisher Retracts 64 Scientific Papers in Fake Peer Review Outbreak.
- M Ware, Peer Review: Benefits, Perceptions and Alternatives, 2008
- M Ware, 2016, Peer Review Survey 2015
Cited by
The limitations to our understanding of peer review
Tennant, J. P., & Ross-Hellauer, T. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-020-00092-1 · Cited by 384
Self-citationThe evaluation of scholarship in academic promotion and tenure processes: Past, present, and future
Schimanski, L. A., & Alperin, J. P. (2018) · F1000Research · doi:10.12688/f1000research.16493.1 · Cited by 342
Checco, A., Bracciale, L., Loreti, P., Pinfield, S., & Bianchi, G. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00703-8 · Cited by 286
The effect of publishing peer review reports on referee behavior in five scholarly journals
Bravo, G., Grimaldo, F., López-Iñesta, E., Mehmani, B., & Squazzoni, F. (2019) · Nature Communications · doi:10.1038/s41467-018-08250-2 · Cited by 273
The changing forms and expectations of peer review
Horbach, S. P. J. M., & Halffman, W. (. W. (2018) · Research Integrity and Peer Review · doi:10.1186/s41073-018-0051-5 · Cited by 179
Metadata
- Authors
- Ross-Hellauer, T.
- Venue
- F1000Research
- Publisher
- F1000 Research Ltd
- Published
- 2017-08-31
- Volume
- 6
- Pages
- 588
- ISSN
- 2046-1402 (ISSN-L)
- OpenAlex
- W2609499779
- Cited by
- 449
- References
- 73
- FWCI
- 9.62
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- European Commission (643410)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.