The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Report of the Ad Hoc Committee on Double-Blind Refereeing
Open accessAbstract
This report of the Ad Hoc Committee on Double-Blind Refereeing was presented to the IMS Council in August 1992. It summarizes arguments for and against blinding referees to the identity of the authors and suggests an experiment to evaluate the process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Double-anonymizing the peer-review process would promote diverse points of view in journal output.
“Double-anonymizing the peer-review process could also help level the playing field for academics from other marginalized groups, giving them a fairer chance to succeed, which, in turn, would promote diverse points of view in journal output and eventually expand the journal peer-reviewer pool to authors coming from such backgrounds (3–5, 19–23).”
From Nobel and novice: Author prominence affects peer review (2022) · cites this paper as [3]
The Advantages section (p. 1) argues that double-blind refereeing lets papers be judged without regard to an author’s rank, gender, institution, or seniority, supporting the proposed fairness mechanism. The paper does not say that this would promote diverse points of view in published journal output; the committee also says the benefits remain uncertain (p. 2). (LLM, conf=0.84)
Experimental — accuracy not yet measuredDouble-anonymizing the peer-review process would eventually expand the journal peer-reviewer pool to authors from marginalized backgrounds.
“Double-anonymizing the peer-review process could also help level the playing field for academics from other marginalized groups, giving them a fairer chance to succeed, which, in turn, would promote diverse points of view in journal output and eventually expand the journal peer-reviewer pool to authors coming from such backgrounds (3–5, 19–23).”
From Nobel and novice: Author prominence affects peer review (2022) · cites this paper as [3]
The Advantages section (p. 1) says double-blind refereeing allows papers to be judged without regard to author characteristics such as gender or rank, supporting the proposed fairer chance. The Gender Bias section (p. 6) also reports a higher acceptance rate for women after one association adopted it. Neither passage supports the further claim that this would diversify journal output or eventually expand the peer-reviewer pool to authors from marginalized backgrounds. (LLM, conf=0.86)
Experimental — accuracy not yet measuredDouble-anonymizing the peer-review process could help level the playing field for academics from marginalized groups, giving them a fairer chance to succeed.
“Double-anonymizing the peer-review process could also help level the playing field for academics from other marginalized groups, giving them a fairer chance to succeed, which, in turn, would promote diverse points of view in journal output and eventually expand the journal peer-reviewer pool to authors coming from such backgrounds (3–5, 19–23).”
From Nobel and novice: Author prominence affects peer review (2022) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1968) · PubMed · Cited by 4,803
Publication prejudices: An experimental study of confirmatory bias in the peer review system
Mahoney, M. J. (1977) · Cognitive Therapy and Research · doi:10.1007/bf01173636 · Cited by 1,013
Zuckerman, H., & Merton, R. (1971) · Minerva · doi:10.1007/bf01553188 · Cited by 789
Are women prejudiced against women?
Goldberg, P. A. (1968) · Society · doi:10.1007/bf03180445 · Cited by 643
Cole, J. R., & Cole, S. (1972) · Science · doi:10.1126/science.178.4059.368 · Cited by 389
The effects of blinding on the quality of peer review. A randomized trial
Mcnutt, R. (1990) · JAMA · doi:10.1001/jama.263.10.1371 · Cited by 271
Gottfredson, S. D. (1978) · American Psychologist · doi:10.1037/0003-066x.33.10.920 · Cited by 258
Peer review in biomedical publication
Ingelfinger, F. J. (1974) · The American Journal of Medicine · doi:10.1016/0002-9343(74)90635-4 · Cited by 241
Editorial Policies and Practices Among Leading Journals in Four Scientific Fields
Beyer, J. M. (1978) · Sociological Quarterly · doi:10.1111/j.1533-8525.1978.tb02172.x · Cited by 226
Stratification in American Science*
Zuckerman, H. (1970) · Sociological Inquiry · doi:10.1111/j.1475-682x.1970.tb01010.x · Cited by 210
Paradigm Development and Particularism: Journal Publication in Three Scientific Disciplines
Pfeffer, J., Leong, A. L., & Strehl, K. (1977) · Social Forces · doi:10.1093/sf/55.4.938 · Cited by 194
Manuscript characteristics influencing reviewers' decisions.
Rowney, J. A., & Zenisek, T. J. (1980) · Canadian Psychology/Psychologie canadienne · doi:10.1037/h0081144 · Cited by 182
(1974) · American Psychologist · doi:10.1037//0003-066x.29.9.698 · Cited by 169
Reliability of reviews for the American Psychologist: A biostatistical assessment of the data.
Cicchetti, D. V. (1980) · American Psychologist · doi:10.1037/0003-066x.35.3.300 · Cited by 161
Improving manuscript evaluation procedures.
Bowen, D. D., Perloff, R., & Jacoby, J. (1972) · American Psychologist · doi:10.1037/h0033849 · Cited by 159
Psychology of Scientist: XXX. Credibility of Psychologists: An Empirical Study
Goodstein, L. D., & Brazis, K. L. (1970) · Psychological Reports · doi:10.2466/pr0.1970.27.3.835 · Cited by 142
Yoels, W. C. (1974) · Sociological Quarterly · doi:10.1111/j.1533-8525.1974.tb00893.x · Cited by 130
Are Women Still Prejudiced against Women? A Replication and Extension of Goldberg's Study
Levenson, H., Burford, B., Bonno, B., & Davis, L. (1975) · The Journal of Psychology · doi:10.1080/00223980.1975.9923907 · Cited by 127
Chance and interrater agreement on manuscripts.
Watkins, M. W. (1979) · American Psychologist · doi:10.1037/0003-066x.34.9.796 · Cited by 116
Reliability of ratings of research papers.
McReynolds, P. (1971) · American Psychologist · doi:10.1037/h0037937 · Cited by 109
Oromaner, M. J. (1977) · Social Studies of Science · doi:10.1177/030631277700700307 · Cited by 103
Discrimination or favoritism? Sex bias in book reviews.
Moore, M. (1978) · American Psychologist · doi:10.1037/0003-066x.33.10.936 · Cited by 86
Recognition of Authors in Blind Review of Manuscripts
Rosenblatt, A., & Kirk, S. A. (1980) · Journal of Social Service Research · doi:10.1300/j079v03n04_04 · Cited by 86
Goldberg revisited: What's in an author's name
Paludi, M. A., & Bauer, W. D. (1983) · Sex Roles · doi:10.1007/bf00289673 · Cited by 85
Patterns of evaluation in science
Moravcsik, M. J. (1971) · Minerva · doi:10.1007/bf01553368 · Cited by 85
Problems Identified by Secondary Review of Accepted Manuscripts
Garfunkel, J. M. (1990) · JAMA · doi:10.1001/jama.1990.03440100077011 · Cited by 72
Interrater agreement on manuscript is not so badp.
Crandall, R. (1978) · American Psychologist · doi:10.1037//0003-066x.33.6.623 · Cited by 71
A physics editor comments on Peters and Ceci's peer-review study
Adair, R. K. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011195 · Cited by 57
Peer review: A philosophically faulty concept which is proving disastrous for science
Horrobin, D. F. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011444 · Cited by 51
Interreferee agreement and acceptance rates in physics
Lazarus, D. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011468 · Cited by 50
Anosmic peer review: A rose by another name is evidently not a rose
Scarr, S. W. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0001164x · Cited by 50
Perhaps it was right to reject the resubmitted manuscripts
Thomas, G. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011675 · Cited by 42
Effect of Acceptance or Rejection on the Author's Evaluation of Peer Review of Medical Manuscripts
Garfunkel, J. M. (1990) · JAMA · doi:10.1001/jama.1990.03440100084013 · Cited by 31
Competency testing for reviewers and editors
Yalow, R. S. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011729 · Cited by 23
Rejecting published work: Similar fate for fiction
Ross, C. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011626 · Cited by 9
Personality and Social Psychology Bulletin: Editorial Comment
Hendrick, C. (1975) · Personality and Social Psychology Bulletin · doi:10.1177/014616727500100301 · Cited by 6
Deception in the study of the peer-review process
Fleiss, J. L. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011341 · Cited by 6
Flagged first, then by citations
Research works printed but not linked: 8
- ADAIR (1982). Discussion of "Peer-review practices of psychological journals," by D. P. Peters and S. J. Ceci. The Behavioral and Brain Sciences 5.
- CicCHETTI, D. V. and ERoN, L. D. (1979). The reliability of manuscript reviewing for the Journal of Abnormal Psychology. In Proceedings of the Social Statistics Section 596-600. Amer. Statist. Assoc., Washington, D.C.
- GorDoN, M. D. (1980). The role of referees in scientific communication. In The Psychology of Written Communication (J. Hartley, ed.) 263–275. Kogan Page, London.
- HENDRick, C. (1977). Editorial comment. Personality and Social Psychology Bulletin 3 1-2.
- McCARTNEY, J. L. (1973). Manuscript reviewing. Sociological Quarterly 14 290, 444-446.
- PETERs, D. P. and CECI, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again (with discussion). The Behavioral and Brain Sciences 5 187-255.
- ToBAcí, E. (1980). ". . . that ye be judged." In An Evaluation of the Peer Review System in Psychological Research (J. Demarest, Chairman). Open Forum at the American Psychological Convention, Montreal.
- YoToPoULos, P. A. (1961). Institutional affiliation of the contributors to three professional journals. American Economic Review 51 665-670.
Cited by
Sorted by citations
Gender differences in grant peer review: A meta-analysis
BORNMANN, L., MUTZ, R., & DANIEL, H. (2007) · Journal of Informetrics · doi:10.1016/j.joi.2007.03.001 · Cited by 337
“Apart from assessment of the grant applicant’s proposed research, decisions on applications are also based on an assessment of the applicant’s track records ( Cox, Gleser, Perlman, Reid, & Roeder, 1993 ).”
MethodEditorial peer review for improving the quality of reports of biomedical studies
Jefferson, T., Rudin, M., Brodney Folse, S., & Davidoff, F. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000016.pub3 · Cited by 312
Nobel and novice: Author prominence affects peer review
Huber, J., Inoua, S., Kerschbamer, R., König-Kersting, C., Palan, S., & Smith, V. L. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2205779119 · Cited by 148
“” Alluding to the Gospel according to Matthew 25:29, Merton termed this pattern of the misallocation of credit for scientific contributions “the Matthew effect in science,” while others called it “status bias” (3).”
BackgroundResultPeer Review for Journals as it Stands Today—Part 2
CAMPANARIO, J. M. (1998) · Science Communication · doi:10.1177/1075547098019004002 · Cited by 132
Single- versus double-blind reviewing: an analysis of the literature
Snodgrass, R. (2006) · ACM SIGMOD Record · doi:10.1145/1168092.1168094 · Cited by 118
“This committee issued a report that contained a three-page literature review [Cox et al. 1993].”
BackgroundInfluentialEditorial peer review for improving the quality of reports of biomedical studies
Jefferson, T., Rudin, M., Folse, S. B., & Davidoff, F. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000016.pub2 · Cited by 79
Snodgrass, R. T. (2007) · ACM Transactions on Database Systems · doi:10.1145/1206049.1206050 · Cited by 72
The Reliability of Peer Reviews of Papers on Information Systems
Wood, M. C., Roberts, M., & Howell, B. (2004) · Journal of Information Science · doi:10.1177/0165551504041673 · Cited by 31
“[1] described the general level of agreement between reviewers of the same paper, as reported by the studies they discuss, as ‘low’ in both the behavioural and the medical sciences.”
BackgroundNobel and Novice: Author Prominence Affects Peer Review
Huber, J., M. Inoua, S., Kerschbamer, R., König-Kersting, C., Palan, S., & Smith, V. L. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4190976 · Cited by 146
Radžvilas, M., Pretis, F. D., Peden, W., Tortoli, D., & Osimani, B. (2022) · Computational Economics · doi:10.1007/s10614-022-10250-w · Cited by 19
“There are already debates about how these systems can affect reviewer bias (Armstrong, 1997; Bernard, 2018; Budden et al., 2008; Cox et al., 1993; Darling, 2014; Engqvist and Frommen, 2008; Ferber and Teiman, 1980; Fish, 1989; Garvalov, 2015; Largent & Snodgrass, 2016; Lotfi & Mahian, 2014; Tomkins…”
BackgroundBlind Peer Review by Academic Journals
Largent, E. A., & Snodgrass, R. T. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-802460-7.00005-x · Cited by 14
“A potential limitation to the study is that “the referees receiving the blinded copy of the manuscript would have been aware that they were part of an experiment and may in consequence have been more careful with their reports” (Cox et al., 1993, p. 316). Jadad et al. (1996) sought to determine whether blinding would…”
BackgroundResultInfluentialThe false ‘Ortega Hypothesis’: a literature science case study
Száva‐Kováts, E. (2004) · Journal of Information Science · doi:10.1177/0165551504047823 · Cited by 13
“, the ‘halo effect’ [34], or the ‘peer review’ [42]; • the falsifying misinterpretation of the original doctrine of Ortega continued and progressed: ‘prolificacy’ appeared as a criterion within the ‘Ortega Hypothesis’ [43], an idea which is rooted only in the Coles’ definition of the ‘elite’; • there was an increase i…”
BackgroundMethodEvaluating the Pros and Cons of Different Peer Review Policies via Simulation
Zhu, J., Fung, G. P. C., Wong, W. H., Li, Z., & Xu, C. (2015) · Science and Engineering Ethics · doi:10.1007/s11948-015-9683-8 · Cited by 6
“An endless debate surrounds the subject of the double-blind review policy (Billard 1993; Blank 1991; Cox et al. 1993; Evans et al. 1990; Fisher et al. 1994; Hill and Provost 2003; Justice et al. 1998; Snodgrass 2006).”
BackgroundEditorial: Double-Blind Refereeing
Raab, G., & Moorthy, U. (1998) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/1467-985x.00093 · Cited by 3
“The IMS committee report sets out the arguments for and against double-blind refereeing, as follows.”
BackgroundInfluentialEditorial: Refereeing—How Blind Should We Be?
Rothkopf, M. H. (1994) · INFORMS Journal on Applied Analytics · doi:10.1287/inte.24.6.14 · Cited by 2
Evaluation bias against work in progress: evidence from a Spanish marketing conference
Calderón-Martínez, A., Ruiz-Conde, E., Parreño-Selva, J., & Ruiz, F. M. (2010) · Higher Education · doi:10.1007/s10734-009-9306-x · Cited by 1
“Among the first characteristics, we can highlight bias against studies with results that are not statistically significant or that are contrary to expectations and bias against studies that replicate previous studies (Campanario 1998a, b). The latter type of bias (bias against certain institutions/individuals, termed…”
ResultTansella, M., Pancheri, P., & Cassano, G. B. (1997) · Epidemiologia e Psichiatria Sociale · doi:10.1017/s1121189x00004851 · Cited by 1
Science and Technology Peer Review: GPRA
Kostoff, R. N. (2003) · doi:10.21236/ada418868 · Cited by 1
Radžvilas, M., Pretis, F. D., Peden, W., Tortoli, D., & Osimani, B. (2020) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2011.07797
“One issue is that there are other means by which editors can accomplish these objectives [Peters and Ceci, 1982, Cox et al., 1993].”
Influential(2014) · Chapman & Hall/CRC eBooks · doi:10.1201/b16720-26
Author, I., & Author, I. I. (2014) · Journal of Science Teacher Education · doi:10.1007/s10972-014-9408-x
Is the Lack of Inter-referee Reliability a Threat to the Effectiveness of Peer Review in Science?
Wray, K. B. (2025) · Perspectives on Science · doi:10.1162/posc_a_00639
“Cox et al. report that “ similar low levels of agreement have been found in the medical sciences ” (Cox et al. 1993, p. 313).”
Influential
Metadata
- Authors
- Cox, D., Gleser, L., Perlman, M., Reid, N., & Roeder, K.
- Affiliations
- University of Oxford (GB)
- University of Pittsburgh (US)
- University of Washington (US)
- University of Toronto (CA)
- Yale University (US)
- Venue
- Statistical Science
- Publisher
- Institute of Mathematical Statistics
- Published
- 1993-08-01
- Volume
- 8
- Issue
- 3
- Pages
- 310-317
- ISSN
- 0883-4237 (ISSN-L)
- 2168-8745
- OpenAlex
- W2016359003
- Cited by
- 21
- References
- 37
- Citations by year
- 2025: 1 · 2022: 2 · 2020: 1 · 2016: 1 · 2015: 1 · 2014: 2
- FWCI
- 2.89
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Statistical Science (publishedVersion)
- Project Euclid (Cornell University) (submittedVersion)
- Topics
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- peer review, author identity disclosure, journal editorial policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review