Effect of Blinding and Unmasking on the Quality of Peer Review: A Randomized Trial
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
CONTEXT: Little research has been conducted into the quality of peer review and, in particular, the effects of blinding peer reviewers to authors' identities or masking peer reviewers' identities. OBJECTIVE: To determine whether concealing authors' identities from reviewers (blinding) and/or revealing the reviewer's identity to a coreviewer (unmasking) affects the quality of reviews, the time taken to carry out reviews, and the recommendation regarding publication. DESIGN AND SETTING: Randomized trial of 527 consecutive manuscripts submitted to BMJ, which were randomized and each sent to 2 peer reviewers. INTERVENTIONS: Manuscripts were randomized as to whether the reviewers were unmasked, masked, or uninformed that a study was taking place. Two reviewers for each manuscript were randomized to receive either a blinded or an unblinded version. MAIN OUTCOME MEASURES: Mean total quality score, time taken to carry out the review, and recommendation regarding publication. RESULTS: Of the 527 manuscripts entered into the study, 467 (89%) were successfully randomized and followed up. The mean total quality score was 2.87. There was little or no difference in review quality between the masked and unmasked groups (scores of 2.82 and 2.96, respectively) and between the blinded and unblinded groups (scores of 2.87 and 2.90, respectively). There was no apparent Hawthorne effect. There was also no significant difference between groups in the recommendations regarding publication or time taken to review. CONCLUSIONS: Blinding and unmasking made no editorially significant difference to review quality, reviewers' recommendations, or time taken to review. Other considerations should guide decisions as to the form of peer review adopted by a journal, and improvements in the quality of peer review should be sought via other means.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
What Makes a Good Reviewer and a Good Review for a General Medical Journal?
Black, N. (1998) · JAMA · doi:10.1001/jama.280.3.231 · Cited by 266
Self-citationThe Effects of Blinding on the Quality of Peer Review: A Randomized Trial
McNutt, R. A. (1990) · JAMA · doi:10.1001/jama.1990.03440100079012 · Cited by 261
Peer review in biomedical publication
Ingelfinger, F. J. (1974) · The American Journal of Medicine · doi:10.1016/0002-9343(74)90635-4 · Cited by 241
Peer review. Crude and understudied, but indispensable
Kassirer, J. P. (1994) · JAMA · doi:10.1001/jama.272.2.96 · Cited by 235
A Difficult Balance: Editorial Peer Review in Medicine
(1992) · Annals of Internal Medicine · doi:10.7326/0003-4819-116-11-960_3 · Cited by 212
A Difficult Balance: Editorial Peer Review in Medicine: Review of a Book
Carney, M. J. (1986) · JAMA · doi:10.1001/jama.1986.03380020117036 · Cited by 189
The Effects of Blinding on Acceptance of Research Papers by Peer Review
Fisher, M. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520020069019 · Cited by 164
A Citation Analysis of the Impact of Blinded Peer Review
Laband, D. N. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520020073020 · Cited by 120
Peer Review in Medical Journals
Robin, E. D., & Burke, C. M. (1987) · CHEST Journal · doi:10.1378/chest.91.2.252 · Cited by 27
The decision to publish. Ethical dilemmas
Shapiro, S. (1985) · Journal of Chronic Diseases · doi:10.1016/0021-9681(85)90082-7 · Cited by 25
Cited by
McCambridge, J., Witton, J., & Elbourne, D. R. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.08.015 · Cited by 2,748
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
Peer Review: A Flawed Process at the Heart of Science and Journals
Smith, R. (2006) · Journal of the Royal Society of Medicine · doi:10.1177/014107680609900414 · Cited by 880
Self-citationKroeze, W., Werkman, A., & Brug, J. (2006) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm3103_2 · Cited by 570
Double-blind review favours increased representation of female authors
BUDDEN, A., TREGENZA, T., AARSSEN, L., KORICHEVA, J., LEIMU, R., & LORTIE, C. (2008) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2007.07.008 · Cited by 552
Testing and adjusting for publication bias
Møller, A. P., & Jennions, M. D. (2001) · Trends in Ecology & Evolution · doi:10.1016/s0169-5347(01)02235-2 · Cited by 418
van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999) · BMJ · doi:10.1136/bmj.318.7175.23 · Cited by 415
Self-citationEffect of Patient Reminder/Recall Interventions on Immunization Rates: A Review
Szilagyi, P. G., Bordley, C., Vann, J. C., Chelminski, A., Kraus, R. M., Margolis, P. A., & Rodewald, L. E. (2000) · JAMA · doi:10.1001/jama.284.14.1820 · Cited by 377
Effects of Editorial Peer Review: A Systematic Review
Jefferson, T., Alderson, P., Wager, E., & Davidoff, F. (2002) · JAMA · doi:10.1001/jama.287.21.2784 · Cited by 347
Godlee, F., Gale, C. R., & Martyn, C. N. (1998) · JAMA · doi:10.1001/jama.280.3.237 · Cited by 333
Self-citationEditorial peer review for improving the quality of reports of biomedical studies
Jefferson, T., Rudin, M., Brodney Folse, S., & Davidoff, F. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000016.pub3 · Cited by 312
Boutron, I., Tubach, F., Giraudeau, B., & Ravaud, P. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.12.010 · Cited by 300
The ups and downs of peer review
Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M., Mans, R., Mayhew, D., McGowan, S., Polter, A., et al. (2007) · Advances in Physiology Education · doi:10.1152/advan.00104.2006 · Cited by 294
A multi-disciplinary perspective on emergent and future innovations in peer review
Tennant, J. P., Dugan, J. M., Graziotin, D., Jacques, D. C., Waldner, F., Mietchen, D., Elkhatib, Y., B. Collister, L., Pikas, C. K., Crick, T., et al. (2017) · F1000Research · doi:10.12688/f1000research.12037.1 · Cited by 289
Effects of training on quality of peer review: randomised controlled trial
Schroter, S., Black, N., Evans, S., Carpenter, J., Godlee, F., & Smith, R. (2004) · BMJ · doi:10.1136/bmj.38023.700775.ae · Cited by 268
Self-citationWhat Makes a Good Reviewer and a Good Review for a General Medical Journal?
Black, N. (1998) · JAMA · doi:10.1001/jama.280.3.231 · Cited by 266
Self-citationMeasuring the quality of editorial peer review.
Jefferson, T. (2002) · JAMA · doi:10.1001/jama.287.21.2786 · Cited by 230
Bruce, R., Chauvin, A., Trinquart, L., Ravaud, P., & Boutron, I. (2016) · BMC Medicine · doi:10.1186/s12916-016-0631-5 · Cited by 218
Single-blind vs Double-blind Peer Review in the Setting of Author Prestige
Okike, K., Hug, K. T., Kocher, M. S., & Leopold, S. S. (2016) · JAMA · doi:10.1001/jama.2016.11014 · Cited by 192
Development of the Review Quality Instrument (RQI) for Assessing Peer Reviews of Manuscripts
van Rooyen, S., Black, N., & Godlee, F. (1999) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00047-5 · Cited by 183
Self-citationStatistical Reviewers Improve Reporting in Biomedical Articles: A Randomized Trial
Cobo, E., Selva-O'Callagham, A., Ribera, J.-M., Cardellach, F., Dominguez, R., & Vilardell, M. (2007) · PLoS ONE · doi:10.1371/journal.pone.0000332 · Cited by 112
Single- versus double-blind reviewing: an analysis of the literature
Snodgrass, R. (2006) · ACM SIGMOD Record · doi:10.1145/1168092.1168094 · Cited by 118
The sociology of scientific validity: How professional networks shape judgement in peer review
Teplitskiy, M., Acuna, D., Elamrani-Raoult, A., Körding, K., & Evans, J. (2018) · Research Policy · doi:10.1016/j.respol.2018.06.014 · Cited by 113
Patel, J. (2014) · BMC Medicine · doi:10.1186/s12916-014-0128-z · Cited by 75
Alam, M., Kim, N. A., Havey, J., Rademaker, A., Ratner, D., Tregre, B., West, D. P., & Coleman III, W. P. (2011) · British Journal of Dermatology · doi:10.1111/j.1365-2133.2011.10432.x · Cited by 58
Quality Assessment of Reviewers’ Reports Using a Simple Instrument
Landkroon, A. P., Euser, A. M., Veeken, H., Hart, W., & Overbeke, A. J. P. M. (2006) · Obstetrics & Gynecology · doi:10.1097/01.aog.0000231675.74957.48 · Cited by 31
Ucci, M. A., D'Antonio, F., & Berghella, V. (2022) · American Journal of Obstetrics & Gynecology MFM · doi:10.1016/j.ajogmf.2022.100645 · Cited by 25
O’Brien, B. C., Artino, A. R., Costello, J. A., Driessen, E., & Maggio, L. A. (2021) · PLOS ONE · doi:10.1371/journal.pone.0260558 · Cited by 22
Thompson, S. R., Agel, J., & Losina, E. (2016) · Learned Publishing · doi:10.1002/leap.1009 · Cited by 13
MOPRD: A multidisciplinary open peer review dataset
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2022) · arXiv · doi:10.48550/arxiv.2212.04972 · Cited by 11
“These data, especially structured data, are of great importance to arti-ficial intelligence (AI) research [24].”
BackgroundJones, C. W., Adams, A., Misemer, B. S., Weaver, M. A., Schroter, S., Khan, H., Margolis, B., Schriger, D. L., & Platts-Mills, T. F. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-066624 · Cited by 10
Metadata
- Authors
- van Rooyen, S., Godlee, F., Evans, S., Smith, R., & Black, N.
- Venue
- JAMA
- Publisher
- American Medical Association (AMA)
- Published
- 1998-07-15
- Volume
- 280
- Issue
- 3
- Pages
- 234
- ISSN
- 0002-9955 (ISSN-L)
- 0098-7484
- OpenAlex
- W1972376036
- Cited by
- 280
- References
- 13
- FWCI
- 76.30
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.