The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Chance and Consensus in Peer Review
Abstract
An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to a significant extent on chance. The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it. No evidence of systematic bias in the selection of NSF reviewers was found.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Early quantitative studies already showed that agreement between reviewers is often low, with correlations between independent scores commonly ranging from weak to moderate (e.g., Cole et al, 1981; Cicchetti, 1991).”
From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Cole et al, 1981
Citations
Cites
Toward a Metric of Science: The Advent of Science Indicators.
Wilken, P. H., Elkana, Y., Lederberg, J., Merton, R. Κ., Thackray, A. W., & Zuckerman, H. (1979) · Social Forces · doi:10.2307/2577293 · Cited by 280
Peer Review and the Support of Science
Cole, S., Rubin, L., & Cole, J. R. (1977) · Scientific American · doi:10.1038/scientificamerican1077-34 · Cited by 140
Self-citationWhich researcher will get the grant?
Cole, J. R., & Cole, S. (1979) · Nature · doi:10.1038/279575a0 · Cited by 21
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- COLE, S, METRIC SCI ADVENT SC: 162 (1978).
Cited by
Sorted by citations
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447
DiPrete, T. A., & Eirich, G. M. (2006) · Annual Review of Sociology · doi:10.1146/annurev.soc.32.061604.123127 · Cited by 2,069
“A second consequence was that chance events (e.g., unequal luck in the draw of reviewers upon submitting a grant proposal for funding (Cole, Cole, and Simon 1981) would produce a relative advantage for one of two individuals of identical talent, and this relative advantage could persist and…”
BackgroundFrom translations to problematic networks: An introduction to co-word analysis
Callon, M., Courtial, J.-P., Turner, W. A., & Bauin, S. (1983) · Social Science Information · doi:10.1177/053901883022002003 · Cited by 1,951
Nepotism and sexism in peer-review
Wennerås, C., & Wold, A. E. (1997) · Nature · doi:10.1038/387341a0 · Cited by 1,346
Peer-review practices of psychological journals: The fate of published articles, submitted again
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011183 · Cited by 1,079
Metadata
- Authors
- Cole, S., Cole, J. R., & Simon, G. A.
- Affiliations
- Stony Brook University (US)
- Columbia University (US)
- ORCID
- Cole, Stephen
- Venue
- Science
- Published
- 1981-11-20
- Volume
- 214
- Issue
- 4523
- Pages
- 881-886
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2091375917
- Semantic Scholar
- 20d3ad461253ead52398dcc0b194c5f5af1f6e08
- Cited by
- 623
- References
- 3
- Citations by year
- 2026: 15 · 2025: 16 · 2024: 16 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 27 · 2019: 24 · 2018: 18 · 2017: 17 · 2016: 21 · 2015: 23 · 2014: 22 · 2013: 15 · 2012: 18
- FWCI
- 279.85
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- DigitalGeorgetown (Georgetown University Library) (submittedVersion)
- Topics
- Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- peer review, research funding, National Science Foundation, reviewer disagreement, reviewer selection, scientific consensus, research evaluation
- MeSH terms
- Peer Review, Research Support as Topic, United States
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
