The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Chance and Consensus in Peer Review
Abstract
An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to a significant extent on chance. The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it. No evidence of systematic bias in the selection of NSF reviewers was found.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Early quantitative studies already showed that agreement between reviewers is often low, with correlations between independent scores commonly ranging from weak to moderate (e.g., Cole et al, 1981; Cicchetti, 1991).”
From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Cole et al, 1981
Citations
Cites
Toward a Metric of Science: The Advent of Science Indicators.
Wilken, P. H., Elkana, Y., Lederberg, J., Merton, R. Κ., Thackray, A. W., & Zuckerman, H. (1979) · Social Forces · doi:10.2307/2577293 · Cited by 280
Peer Review and the Support of Science
Cole, S., Rubin, L., & Cole, J. R. (1977) · Scientific American · doi:10.1038/scientificamerican1077-34 · Cited by 140
Self-citationWhich researcher will get the grant?
Cole, J. R., & Cole, S. (1979) · Nature · doi:10.1038/279575a0 · Cited by 21
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- COLE, S, METRIC SCI ADVENT SC: 162 (1978).
Cited by
Sorted by citations
Items 501–600 of 623
Some procedural obscurities in Peters and Ceci's peer-review study
White, M. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011687 · Cited by 1
Quality Assessment of Medical Research and Education
Eriksson, H. (1992) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300013763 · Cited by 1
Science evaluation: A glossary of possible contents
Shadish, W. R. (1989) · Social Epistemology · doi:10.1080/02691728908578531 · Cited by 1
Designing peer review for the subjective as well as the objective side of science
Mitroff, I. I. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011523 · Cited by 1
Simulating a nomination procedure
Kochen, M., Blaivas, A. S., Brumbaugh, R. C., et al. (1984) · Mathematical Modelling · doi:10.1016/0270-0255(84)90034-4 · Cited by 1
Walker, E. F., & Mittal, V. A. (2007) · Applied and Preventive Psychology · doi:10.1016/j.appsy.2007.07.012 · Cited by 1
Refereeing Mortus Est, Vivat Refereeing
Sala, S. D., & Grafman, J. H. (2002) · Cortex · doi:10.1016/s0010-9452(08)70657-9 · Cited by 1
Sinclair, J. D. (1993) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00030788 · Cited by 1
Article Commentary: Social Constructivism, Positivism, and Facilitated Communication
Whitehurst, G. J. “., & Crone, D. A. (1994) · Research and Practice for Persons with Severe Disabilities · doi:10.1177/154079699401900306 · Cited by 1
Referee report on an earlier draft of Peters and Ceci's target article
Scott, W. A. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011651 · Cited by 1
Reliability and validity of peer review
Zeaman, D. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011730 · Cited by 1
New NIE Peer Review Procedures
Justiz, M. J., & Moorman, H. (1985) · Educational Researcher · doi:10.2307/1174474 · Cited by 1
Criterion problems in journal review practices
Cone, J. D. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011304 · Cited by 1
Wasserman, G. S. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00066000 · Cited by 1
Peer Review of Impact: Could It Work?
Derrick, G. (2018) · Springer eBooks · doi:10.1007/978-3-319-63627-6_2 · Cited by 1
Adl, S. M. (2017) · Rhizosphere · doi:10.1016/j.rhisph.2017.07.003 · Cited by 1
Avaliação por pares e processo decisorio nas agencias de fomento a pesquisa. O CNPq e a FAPESP
Davyt, A. (2001) · doi:10.47749/t/unicamp.2001.222387 · Cited by 1
WEIZSÄCKER, E. U. V. (1986) · Elsevier eBooks · doi:10.1016/b978-0-08-031281-1.50019-9 · Cited by 1
(2017) · Elsevier eBooks · doi:10.1016/b978-0-12-811306-6.02002-9 · Cited by 1
(2012) · Elsevier eBooks · doi:10.1016/b978-1-84334-589-3.50014-8 · Cited by 1
Cole, S. (1995) · Sociological Forum · doi:10.1007/bf02098561 · Cited by 1
Self-citationUncertainty in peer reviewPreprint
Barnett, A. G., McCreanor, V., Allen, L., Lash, T. L., & Aldcroft, A. (2024) · doi:10.31219/osf.io/fm6st · Cited by 1
The Matthew effect and early-career setbacks in research funding—a replication studyPreprint
Traag, V., Brady, E., Vincent‐Lamarre, P., Bidel, F., Bento, C. L., Andersen, J. P., & Bloch, C. W. (2025) · eLife · doi:10.7554/elife.109042.1 · Cited by 1
The Matthew effect and early-career setbacks in research funding—a replication studyPreprint
Traag, V., Brady, E., Vincent‐Lamarre, P., Bidel, F., Bento, C. L., Andersen, J. P., & Bloch, C. W. (2025) · eLife · doi:10.7554/elife.109042 · Cited by 2
The evaluation of competitive research funding : .an application to French programs
Lanoë, M. (2018) · theses.fr (ABES) · doi:10.70675/fec8e8a0zb536z48b7zb067z92c6b59563fc · Cited by 1
Machine Learning in Scientometrics
Martín, A. I. (2015) · doi:10.20868/upm.thesis.36488 · Cited by 1
von Weizsäcker, E. U. (2014) · SpringerBriefs on pioneers in science and practice · doi:10.1007/978-3-319-03662-5_4
Simonton, D. K. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165358.004
Social Constructivism, Positivism, and Facilitated Communication.
Whitehurst, G. J. “., & Crone, D. A. (1994)
(2004) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165358.010
Roumbanis, L. (2015) · KTH Publication Database DiVA (KTH Royal Institute of Technology)
Mondor, P. (2013) · Commposite
Public Policy Survey Method Research Based on Multi-Fit Perspective
Fan, W., Cai, X., Pan, L., et al. (2011) · Cross-cultural communication · doi:10.3968/j.ccc.1923670020110704.400
Aiming for Transdisciplinary Science: Reflections and Guidelines
Gómez-Gómez, J. M., & Hochberg, D. (2014) · Interdisciplinary Science Reviews · doi:10.1179/0308018814z.00000000081
“Nothing so extreme exists today, but there are echoes in the way that certain peer review panels stake out a collective vision of scientific truth and therefore resist what might be innovative proposals for funding (Cole et al. 1981, 881; Lamont 2009).”
BackgroundIgnatyev, A. A., & Mirskaja, E. Z. (1989) · Social Epistemology · doi:10.1080/02691728908578533
Mathematical Modeling of Cumulative Advantage for Tenure Tracking Young Scientists
Taypanhyavong, M. (2013) · Journal of Information and Computational Science · doi:10.12733/jics20101807
“in [6], where they find that peer review outcome is greatly affected by random elements due to a high degree of disagreement among reviewers.”
BackgroundExperimenter and reviewer bias
Witt, J. C., & Hannafin, M. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011717
A Cumulative Advantage Model for Tenure Tracking Young Scientists
Taypanhyavong, M., & Zhang, H. (2012) · Global Congress on Intelligent Systems · doi:10.1109/gcis.2012.59
Bias in awarding scientific grants: A modest proposal for an audit system
McGarity, T. O. (1992) · Accountability in Research · doi:10.1080/08989629208573817
A Debate on Assessment System: Responsibility of Scientist
Chaturvedi, A. K. (2015)
The editor-referee system and publication an editor's view of the process
Shore, S. N. (2011) · EAS Publications Series · doi:10.1051/eas/1149003
Raumplanung: Rückzonungen sollen Einzonungen ermöglichen
Frey, R. L. (2014)
Negotiating Epistemological Differences in Peer Review
Lamont, M., & Guetzkow, J. (2009)
Maia, M. D. F. S., & Caregnato, S. E. (2015) · Em Questão · doi:10.19132/1808-5245213.368-389
Simonton, D. K. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165358.006
Stolker, C. J. J. M. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781139696418.015
Does group discussion contribute reliability of complex judgments?
Cohen, P. R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065729
Now that we know how low the reliability is, what shall we do?
Salzinger, K. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065961
Is there an alternative to peer review?
Greene, R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065845
Replication, reliability and peer review: A case study
Gorman, M. E. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065833
Different rates of agreement on acceptance and rejection: A statistical artifact?
Demorest, M. E. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0006578x
L'activité littéraire des comités de politique scientifique
Teil, G. (1992) · Politiques et management public · doi:10.3406/pomap.1992.3066
Strategyproof Peer Selection.Preprint
Aziz, H., Lev, O., Mattei, N., et al. (2016) · arXiv (Cornell University)
Whose Article Is It Anyway? Student Editors and Law ReviewsPreprint
Potuto, J. R. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2477228
Consolidation: Creativity in Science
Simonton, D. K. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165358.009
Journal Management Issues Based on Citation Metrics
Sarasvady, S. (2013)
Tahir, S. (2010) · Journal of Liaquat University of Medical & Health Sciences
Olsson, G. (2016) · Socialvetenskaplig tidskrift · doi:10.3384/svt.1999.6.2.2885
CHAPTER 5 PEER REVIEW AND EV ALUATION OF R&D IMPACTS
Bozeman, B. L. (1993)
Topics in Statistical Physics: Protein Stability, Non-Equilibrium Thermodynamics and Bibliometrics
Hazoglou, M. J. (2016) · PhDT
Evaluation practices in the selection of ground-breaking research proposals, Working Paper
WWTF, Luukkonen, Stampfer, & Strassnig (2015) · doi:10.22163/fteval.2018.292
Publication and evaluation challenges in games & interactive media
Lawley, E. L. (2019) · International Conference on the Foundations of Digital Games (FDG) · doi:10.1145/3337722.3341873
“Cole et al ’s 1981 article “Chance and consensus in peer review” [10] independently evaluated 150 proposal submitted to the National Science Foundation (NSF), and performed an analysis comparing their independent reviews to the actual funding outcomes.”
MethodFunding Panels as Declarative Bodies: Meritocracy and Paradoxes of Decision-Making in Modern Science
Stark, L. (2019) · International Journal for History Culture and Modernity · doi:10.18352/hcm.593
Reviewing, Reviewers and the Scientific Enterprise
Amanor‐Boadu, V., & Amanor-Boadu, V. (2010) · RePEc: Research Papers in Economics · doi:10.22004/ag.econ.93560
Significance of scientific peer-review system: A case study of the Journal of High Energy Physics.
Sikdar, S. (2018) · Bull. IEEE Tech. Comm. Digit. Libr.
Why We Do Not Have More Pluralism
Roos, M. W. M., & Hoffart, F. M. (2020) · Palgrave studies in sustainability, environment and macroeconomics · doi:10.1007/978-3-030-48423-1_5
Bieri, M., Roser, K., Heyard, R., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.11.26.400028
IJEE Soapbox: Cooperation, Competition, and the Social Organization of the Scientific Enterprise
Holt, R. D. (2010) · Israel Journal of Ecology and Evolution · doi:10.1560/ijee.56.1.1
A New Way to Distribute Research Seed Funding: Peer Review Without Formal Review
Burns, M. A., Gallimore, A. D., Zurbuchen, T. H., Johnson, V. N., & Lattuca, L. R. (2021) · Change The Magazine of Higher Learning · doi:10.1080/00091383.2021.1883976
“Although some studies support the validity of traditional peer review (Li & Agha, 2015), questions have long centered on reviewers’ capacity for accurately and consistently selecting the worthiest proposals at federal agencies such as the National Science Foundation (Cole et al., 1981).”
ResultScientific Discoveries: Real and Imagined
Franzen, S. (2021) · Springer eBooks · doi:10.1007/978-3-030-68063-3_3
Ho, D. E. (2018) · Observational Studies · doi:10.1353/obs.2018.0005
“Peers can of course have dramatically divergent standards of tenability (Simon et al., 1981), but Bross went further and charged that agencies manipulated peer review in low-level radiation “to suppress, vilify, or cut o↵ the funding of the little scientists” (Walker, 2000, p. 95).”
BackgroundDynamic patterns of open review process
Zhao, Z.-D., Chen, J., Lu, Y.-C., Zhao, N., Jiang, D., & Wang, B.-H. (2021) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2021.126245
Argilés-Bosch, J. M., García-Blandón, J., & Ravenda, D. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3966194
Simonton, D. K. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165358.008
Simonton, D. K. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165358.002
(1988) · ASHE-ERIC Higher Education Report · doi:10.1002/aehe.3640170811
Simonton, D. K. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165358.005
Simonton, D. K. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165358.007
Simonton, D. K. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165358.001
Atkinson, R. C. (1981) · Science · doi:10.1126/science.214.4527.1292.a
Westley, J. (1982) · Science · doi:10.1126/science.215.4531.344.c
Hower, J. (1982) · Science · doi:10.1126/science.215.4531.344-b
Academic Evaluation in Higher EducationPreprint
Hamann, J., & Beljean, S. (2019) · doi:10.31235/osf.io/z47ft
Cronbach, L. J. (1981) · Science · doi:10.1126/science.214.4527.1294.a
Westley, J. (1982) · Science · doi:10.1126/science.215.4531.344-c
Schenker‐Wicki, A. (1996) · Deutscher Universitätsverlag eBooks · doi:10.1007/978-3-663-08485-3_7
Die Gründung der Soziologischen Revue
Hartmann, H. (2015) · Springer eBooks · doi:10.1007/978-3-658-07998-7_50-1
Hower, J. (1982) · Science · doi:10.1126/science.215.4531.344.b
Humphreys, L. G. (1982) · Science · doi:10.1126/science.215.4531.344.a
Cronbach, L. J. (1981) · Science · doi:10.1126/science.214.4527.1294-a
Singer, I. M. (1981) · Science · doi:10.1126/science.214.4527.1292.b
Expert Evaluation in the Flow of Information Science
Smrekar, D. D., & Vilar, P. (2014) · Knjižnica revija za področje bibliotekarstva in informacijske znanosti · doi:10.55741/knj.58.1-2.13932
“Ugotavljanje in raziskovanje pomanjkljivosti in slabosti ekspertnega ocenjevanja se pogosteje pojavlja zadnjih trideset let, z začetno objavo »chance and consensus in Peer review« (Priložnost in soglasje v ekspertnem ocenjevanju) (cole, cole in Simon, 1981).”
BackgroundResearch Management: Trends and IssuesPreprint
Volyanskaya, V., & Nazaykinskaya, O. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4646780
Peer Evaluations of Corporate ExternalitiesPreprint
Bernard, D., Juliani, E. M., & Lawrence, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4935123
Impartial Peer Selection: An Annotated Reading List
Lev, O., Lyon, H., & Mattei, N. (2024) · ACM SIGecom Exchanges · doi:10.1145/3699824.3699834
“Academic reviews and grant boards have been the subject of many empirical studies that, among other issues, have focused on the effectiveness and limits of the system, often substantiating anecdotal complaints of bias and inaccuracy [Cole et al. 1981; McNutt et al. 1990; Wenneras and Wold 1997].”
Background
Metadata
- Authors
- Cole, S., Cole, J. R., & Simon, G. A.
- Affiliations
- Stony Brook University (US)
- Columbia University (US)
- ORCID
- Cole, Stephen
- Venue
- Science
- Published
- 1981-11-20
- Volume
- 214
- Issue
- 4523
- Pages
- 881-886
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2091375917
- Semantic Scholar
- 20d3ad461253ead52398dcc0b194c5f5af1f6e08
- Cited by
- 623
- References
- 3
- Citations by year
- 2026: 15 · 2025: 16 · 2024: 16 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 27 · 2019: 24 · 2018: 18 · 2017: 17 · 2016: 21 · 2015: 23 · 2014: 22 · 2013: 15 · 2012: 18
- FWCI
- 279.85
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- DigitalGeorgetown (Georgetown University Library) (submittedVersion)
- Topics
- Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- peer review, research funding, National Science Foundation, reviewer disagreement, reviewer selection, scientific consensus, research evaluation
- MeSH terms
- Peer Review, Research Support as Topic, United States
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
