The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Chance and Consensus in Peer Review
Abstract
An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to a significant extent on chance. The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it. No evidence of systematic bias in the selection of NSF reviewers was found.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Early quantitative studies already showed that agreement between reviewers is often low, with correlations between independent scores commonly ranging from weak to moderate (e.g., Cole et al, 1981; Cicchetti, 1991).”
From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Cole et al, 1981
Citations
Cites
Toward a Metric of Science: The Advent of Science Indicators.
Wilken, P. H., Elkana, Y., Lederberg, J., Merton, R. Κ., Thackray, A. W., & Zuckerman, H. (1979) · Social Forces · doi:10.2307/2577293 · Cited by 280
Peer Review and the Support of Science
Cole, S., Rubin, L., & Cole, J. R. (1977) · Scientific American · doi:10.1038/scientificamerican1077-34 · Cited by 140
Self-citationWhich researcher will get the grant?
Cole, J. R., & Cole, S. (1979) · Nature · doi:10.1038/279575a0 · Cited by 21
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- COLE, S, METRIC SCI ADVENT SC: 162 (1978).
Cited by
Sorted by citations
Items 601–623 of 623
Abramo, G., D’Angelo, C. A., & Grilli, L. (2025) · Journal of Data and Information Science · doi:10.2478/jdis-2025-0010
Tracing the Trajectory of Good Corporate Governance: A Bibliometric Analysis
Mustafa, Z., Ramakrishnan, S., Ali, A., Raza, H., & Faisal, F. (2025) · Journal of the Knowledge Economy · doi:10.1007/s13132-025-02706-9 · Cited by 1
“Generally, experts’ opinions are commonly accepted, but again it can take a long time and is contingent on factors such as expert selection (Cole et al., 1981).”
Martinovich, V. (2025) · Salud Colectiva · doi:10.18294/sc.2025.5658
“…o para reconocer textos generados por inteli - gencia artificial (76) y, por otro, en una especie de males - tar hacia ciertas cualidades intrínsecamente humanas que comenzaron a ser consideradas negativas, como la divergencia de criterios, la subjetividad, y la diversidad de perspectivas (77,78) .”
Yan, Z. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009381338.005
Yan, Z. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009381338.009
Gallo, S. A., Schmaling, K. B., Thompson, L. A., & Glisson, S. R. (2020) · Figshare · doi:10.6084/m9.figshare.12315863
Gallo, S. A., Schmaling, K. B., Thompson, L. A., & Glisson, S. R. (2020) · Figshare · doi:10.6084/m9.figshare.12315866
The Mean-Variance Innovation Tradeoff in AI-Augmented EvaluationsPreprint
Grumbach, C., Lane, J. N., & von Krogh, G. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5933495
Boudreau, K. J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6025114 · Cited by 1
Rubrics, Not Vibes: Structured AI Integration as Quality Control for Peer ReviewPreprint
Turner, S. D., Swiatecka‐Urban, A., & Krishnan, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6314421
Two decades of academic invention in France : scope and driving forces
Carpentier, E. (2022) · theses.fr (ABES) · doi:10.70675/64e32e3ez2e96z43a9zb86bz3e2bfc3e0ec5
Reviewing the reviewers: how social and traditional media repurpose open promotion reviews
Remisiewicz, Ł. (2026) · Science and Public Policy · doi:10.1093/scipol/scag023
van den Besselaar, P., & Mom, C. (2026) · Publications · doi:10.3390/publications14020033
“At the latest since the early 1980s, studies critical of peer review were published (J. R. Cole & Cole, 1981; S. Cole et al., 1981; Chubin & Hackett, 1990) and the debate continues (e.g., Marsh et al., 2008; for a recent overview: Aczel et al., 2025).”
Distributed Peer Review at ALMA: An Empirical Comparison with Panel-Based Review
Carpenter, J. M., & Corvillón, A. (2026) · Publications of the Astronomical Society of the Pacific · doi:10.1088/1538-3873/ae7f51
“Independent studies in which the same proposals are evaluated by separate expert panels have shown that final outcomes can differ substantially between panels, even when each achieves internal convergence (e.g., S. Cole et al. 1981; M. Obrecht et al. 2007; E. L. Pier et al. 2018).”
Schönwolff, M., & Müller, R. (2026) · Research Evaluation · doi:10.1093/reseval/rvag032 · Cited by 1
Puljak, L., & Pacheco, R. L. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112430
Can AI Replace the Peer Reviewer (Yet)?Preprint
Huang, D., Kalay, A., & Schallheim, J. S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6872380
Xie, Z., Du, P., & Bai, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7467668
Generative AI in peer review: evolving policies and regulatory pathways for scientific rigour
Jozaghi, E. (2026) · AI and Ethics · doi:10.1007/s43681-026-01394-8
“Studies have reported differences in assessments among reviewers evaluating the same manuscript [3, 6, 7 ].”
Information Technology, Cognition, and Communication
Coleman, A. S. (2005) · UA Campus Repository (The University of Arizona)
Synergies Sparked: A Research Agenda for Practicing Professionals
Coleman, A. S. (2005) · UA Campus Repository (The University of Arizona)
Three Ways To Become An Academic SuperstarPreprint
Heesen, R. (2014) · PhilSci-Archive (University of Pittsburgh)
Metadata
- Authors
- Cole, S., Cole, J. R., & Simon, G. A.
- Affiliations
- Stony Brook University (US)
- Columbia University (US)
- ORCID
- Cole, Stephen
- Venue
- Science
- Published
- 1981-11-20
- Volume
- 214
- Issue
- 4523
- Pages
- 881-886
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2091375917
- Semantic Scholar
- 20d3ad461253ead52398dcc0b194c5f5af1f6e08
- Cited by
- 623
- References
- 3
- Citations by year
- 2026: 15 · 2025: 16 · 2024: 16 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 27 · 2019: 24 · 2018: 18 · 2017: 17 · 2016: 21 · 2015: 23 · 2014: 22 · 2013: 15 · 2012: 18
- FWCI
- 279.85
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- DigitalGeorgetown (Georgetown University Library) (submittedVersion)
- Topics
- Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- peer review, research funding, National Science Foundation, reviewer disagreement, reviewer selection, scientific consensus, research evaluation
- MeSH terms
- Peer Review, Research Support as Topic, United States
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
