Chance and Consensus in Peer Review
Abstract
An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to a significant extent on chance. The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it. No evidence of systematic bias in the selection of NSF reviewers was found.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Early quantitative studies showed that agreement between reviewers is often low, with correlations between independent scores commonly ranging from weak to moderate.
“Early quantitative studies already showed that agreement between reviewers is often low, with correlations between independent scores commonly ranging from weak to moderate (e.g., Cole et al, 1981; Cicchetti, 1991).”
From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Cole et al, 1981
Cites
Toward a Metric of Science: The Advent of Science Indicators.
Wilken, P. H., Elkana, Y., Lederberg, J., Merton, R. Κ., Thackray, A. W., & Zuckerman, H. (1979) · Social Forces · doi:10.2307/2577293 · Cited by 280 · see citing works
Peer Review and the Support of Science
Cole, S., Rubin, L., & Cole, J. R. (1977) · Scientific American · doi:10.1038/scientificamerican1077-34 · Cited by 140 · see citing works
Self-citationWhich researcher will get the grant?
Cole, J. R., & Cole, S. (1979) · Nature · doi:10.1038/279575a0 · Cited by 21 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- COLE, S, METRIC SCI ADVENT SC: 162 (1978).
Cited by
Sorted by citations
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447 · see citing works
DiPrete, T. A., & Eirich, G. M. (2006) · Annual Review of Sociology · doi:10.1146/annurev.soc.32.061604.123127 · Cited by 2,069 · see citing works
“A second consequence was that chance events (e.g., unequal luck in the draw of reviewers upon submitting a grant proposal for funding (Cole, Cole, and Simon 1981) would produce a relative advantage for one of two individuals of identical talent, and this relative advantage could persist and…”
BackgroundFrom translations to problematic networks: An introduction to co-word analysis
Callon, M., Courtial, J.-P., Turner, W. A., & Bauin, S. (1983) · Social Science Information · doi:10.1177/053901883022002003 · Cited by 1,951 · see citing works
Nepotism and sexism in peer-review
Wennerås, C., & Wold, A. E. (1997) · Nature · doi:10.1038/387341a0 · Cited by 1,346 · see citing works
Peer-review practices of psychological journals: The fate of published articles, submitted again
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011183 · Cited by 1,079 · see citing works
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000 · see citing works
Steinpreis, R. E., Anders, K. A., & Ritzke, D. M. (1999) · Sex Roles · doi:10.1023/a:1018839203698 · Cited by 736 · see citing works
Simonton, D. K. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.4.475 · Cited by 716 · see citing works
The Matthew effect in science funding
Bol, T., de Vaan, M., & van de Rijt, A. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1719557115 · Cited by 638 · see citing works
“To the extent that luck plays a role in early academic success (11), the Matthew effect may undermine meritocracy by allowing an initially fortunate scientist’s recognition to self-perpetuate, while an equally talented but initially less fortunate counterpart remains underappreciated (1, 5–7).”
BackgroundImpact of bibliometrics upon the science system: Inadvertent consequences?
Weingart, P. (2005) · Scientometrics · doi:10.1007/s11192-005-0007-7 · Cited by 603 · see citing works
The Hierarchy of the Sciences?
Cole, S. (1983) · American Journal of Sociology · doi:10.1086/227835 · Cited by 485 · see citing works
Self-citationBoudreau, K. J., Guinan, E. C., Lakhani, K. R., & Riedl, C. (2014) · Management Science · doi:10.1287/mnsc.2015.2285 · Cited by 489 · see citing works
“Variance among reviewers is sometimes greater than variance between submissions (Cole et al. 1981).”
BackgroundCreativity in science: chance, logic, genius, and zeitgeist
(2005) · Choice Reviews Online · doi:10.5860/choice.42-2773 · Cited by 447 · see citing works
Simonton, D. K. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165358 · Cited by 428 · see citing works
Bornmann, L. (2011) · Annual Review of Information Science and Technology · doi:10.1002/aris.2011.1440450112 · Cited by 567 · see citing works
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496 · see citing works
Measuring the effectiveness of scientific gatekeeping
Siler, K., Lee, K. P., & Bero, L. A. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1418218112 · Cited by 338 · see citing works
“(56) argued that roughly one-half of a competitive National Science Foundation granting process was a matter of luck.”
BackgroundA review of bibliometric and other science indicators and their role in research evaluation
King, J. (1987) · Journal of Information Science · doi:10.1177/016555158701300501 · Cited by 334 · see citing works
Ericsson, K. A., Roring, R. W., & Nandagopal, K. (2007) · High Ability Studies · doi:10.1080/13598130701350593 · Cited by 282 · see citing works
“For instance, studies investigating the peer-review process reveal considerable inconsistency among referees, illustrating a large role played by chance in determining the success of a grant proposal or journal article (Cole et al., 1981; Fiske & Fogg, 1990; Cicchetti, 1991).”
BackgroundCitations and journal impact factors: questionable indicators of research quality
Seglen, P. O. (1997) · Allergy · doi:10.1111/j.1398-9995.1997.tb00175.x · Cited by 268 · see citing works
“, giving the whole process the character of a lottery (1-4).”
BackgroundVeletsianos, G., & Kimmons, R. M. (2011) · Computers & Education · doi:10.1016/j.compedu.2011.10.001 · Cited by 260 · see citing works
Rothwell, P. M. (2000) · Brain · doi:10.1093/brain/123.9.1964 · Cited by 254 · see citing works
New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System
Travis, G., & Collins, H. M. (1991) · Science Technology & Human Values · doi:10.1177/016224399101600303 · Cited by 247 · see citing works
“The results are presented in monographs on the two phases of the study (Cole, Rubin, and Cole 1978; Cole and Cole 1981) and two articles (Cole, Rubin, and Cole 1977; Cole, Cole, and Simon 1981).”
BackgroundCitation rates and journal impact factors are not suitable for evaluation of research
Seglen, P. O. (1998) · Acta Orthopaedica Scandinavica · doi:10.3109/17453679809000920 · Cited by 240 · see citing works
Peer Review and the Social Construction of Knowledge in the Management Discipline
Bedeian, A. G. (2004) · Academy of Management Learning and Education · doi:10.5465/amle.2004.13500489 · Cited by 234 · see citing works
“Recognizing that the likelihood of refusing to make any recommendations for revision and still have their work published is slim or none, these authors give ground in some areas and hold fast in others. This jockeying, of course, introduces a further element of risk into the review process, as editors and referees may…”
BackgroundLow agreement among reviewers evaluating the same NIH grant applications
Pier, E. L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M. J., Ford, C. E., & Carnes, M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1714379115 · Cited by 210 · see citing works
The Manuscript Review and Decision-Making Process
Bakanic, V., McPhail, C., & Simon, R. J. (1987) · American Sociological Review · doi:10.2307/2095599 · Cited by 200 · see citing works
Interrater agreement for journal manuscript reviews.
Whitehurst, G. J. “. (1984) · American Psychologist · doi:10.1037/0003-066x.39.1.22 · Cited by 189 · see citing works
Cialdini, R. B. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01091.x · Cited by 189 · see citing works
Gläser, J., & Laudel, G. (2016) · European Journal of Sociology · doi:10.1017/s0003975616000047 · Cited by 186 · see citing works
Conservatism and risk-taking in peer review: Emerging ERC practices
Luukkonen, T. (2012) · Research Evaluation · doi:10.1093/reseval/rvs001 · Cited by 185 · see citing works
“The most debated area is the ‘grey’ boundary area around the line demarcating the proposals that will be funded and those that will not (Cole et al., 1981).”
BackgroundBig names or big ideas: Do peer-review panels select the best science proposals?
Li, D., & Agha, L. (2015) · Science · doi:10.1126/science.aaa0185 · Cited by 227 · see citing works
Collaboration, interdisciplinarity, and the epistemology of contemporary science
Andersen, H. M. (2015) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2015.10.006 · Cited by 181 · see citing works
Multidimensional Assessment of Scholarly Research Impact
Moed, H. F., & Halevi, G. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23314 · Cited by 168 · see citing works
PFEFFER'S Barriers to The Advance Of Organizational Science: A Rejoinder
Cannella, A. A., & Paetzold, R. L. (1994) · Academy of Management Review · doi:10.5465/amr.1994.9410210755 · Cited by 208 · see citing works
Pampel, H., & Dallmeier-Tiessen, S. (2013) · Springer eBooks · doi:10.1007/978-3-319-00026-8 · Cited by 186 · see citing works
Research Funding: the Case for a Modified Lottery
Fang, F. C., & Casadevall, A. (2016) · mBio · doi:10.1128/mbio.00422-16 · Cited by 162 · see citing works
“found that the odds of a proposal submitted to the National Science Foundation (NSF) getting funded were largely based on chance—the chance that specific reviewers would be chosen (26).”
Barriers to scientific contributions: The author's formula
Armstrong, J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011201 · Cited by 151 · see citing works
The psychometrics of everyday life
Kunda, Z., & Nisbett, R. E. (1986) · Cognitive Psychology · doi:10.1016/0010-0285(86)90012-5 · Cited by 150 · see citing works
“Actual correlations for NSF proposals were estimated from data for solid state physics and economics panels obtained by Cole, Cole, and Simon (1981).”
MethodPeer review of grant applications: what do we know?
Wessely, S. C. (1998) · The Lancet · doi:10.1016/s0140-6736(97)11129-1 · Cited by 146 · see citing works
Human subjects review, personal values, and the regulation of social science research.
Ceci, S. J., Peters, D. P., & Plotkin, J. (1985) · American Psychologist · doi:10.1037/0003-066x.40.9.994 · Cited by 142 · see citing works
The Social Construction of Two Biologists' Proposals
Myers, G. (1985) · Written Communication · doi:10.1177/0741088385002003001 · Cited by 142 · see citing works
Funding Science: The Real Defects of Peer Review and An Alternative To It
Roy, R. (1985) · Science Technology & Human Values · doi:10.1177/016224398501000309 · Cited by 140 · see citing works
Graves, N., Barnett, A. G., & Clarke, P. (2011) · BMJ · doi:10.1136/bmj.d4797 · Cited by 134 · see citing works
Overcoming the Illegitimacy Discount: Cultural Entrepreneurship in the US Feature Film Industry
Zhao, E. Y., Ishihara, M., & Lounsbury, M. D. (2013) · Organization Studies · doi:10.1177/0170840613485844 · Cited by 126 · see citing works
Mallard, G., Lamont, M., & Guetzkow, J. (2009) · Science Technology & Human Values · doi:10.1177/0162243908329381 · Cited by 124 · see citing works
The Ratings Game: Asymmetry in Classification
Waguespack, D. M., & Sorenson, O. (2010) · Organization Science · doi:10.1287/orsc.1100.0533 · Cited by 120 · see citing works
“…have time and again confirmed that, when evaluators know the identities of the individuals being evaluated, biases creep into the process with high-status participants (and usually men) receiving more favorable evaluations (e.g., Cole et al. 1981, Blank 1991, Goldin and Rouse 2000).”
BackgroundOpen exchange and epistemic progress.
Mahoney, M. J. (1985) · American Psychologist · doi:10.1037/0003-066x.40.1.29 · Cited by 121 · see citing works
MacRae, R. J., Hill, S. B., Henning, J., & Mehuys, G. R. (1989) · Biological Agriculture & Horticulture · doi:10.1080/01448765.1989.9754518 · Cited by 116 · see citing works
Research and teaching: symbiosis or conflict
Elton, L. (1986) · Higher Education · doi:10.1007/bf00129218 · Cited by 115 · see citing works
Quality, Quantity, and Age: The Careers of Ten Distinguished Psychologists
Simonton, D. K. (1986) · The International Journal of Aging and Human Development · doi:10.2190/kb7e-a45m-x8x7-dqj4 · Cited by 115 · see citing works
Bornmann, L., Wallon, G., & Ledin, A. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003480 · Cited by 113 · see citing works
“Cole and his colleagues [8] concluded in their highly influential study on grant peer review at the National Science Foundation (NSF, Arlington, VA, USA) that ‘‘the fate of a particular application is roughly half determined by the characteristics of the proposal and the principal investigator, and about half by appar…”
BackgroundTracking Replicability as a Method of Post-Publication Open Evaluation
Hartshorne, J. K., & Schachner, A. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00008 · Cited by 102 · see citing works
“Peer review, the primary vetting procedure for publication, is often slow, contentious, and uneven (Mahoney, 1977; Cole et al., 1981; Peters and Ceci, 1982; Eysenck and Eysenck, 1992; Newton, 2010).”
BackgroundWhat do we know about grant peer review in the health sciences?
Guthrie, S., Ghiga, I., & Wooding, S. (2018) · F1000Research · doi:10.12688/f1000research.11917.2 · Cited by 102 · see citing works
How to avoid borrowed plumes in academia
Osterloh, M., & Frey, B. S. (2019) · Research Policy · doi:10.1016/j.respol.2019.103831 · Cited by 108 · see citing works
“It is also indicated by empirical findings on the “luck of the reviewer draw” (Cole et al., 1981; Bornmann and Daniel, 2009), which in many cases is decisive for the acceptance or rejection of a grant proposal or paper.”
BackgroundPeer review for improving the quality of grant applications
Demicheli, V., & Pietrantonj, C. D. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000003.pub2 · Cited by 99 · see citing works
Written Argumentation for Real Audiences: Suggestions for Teacher Research and Classroom Practice
Johns, A. M. (1993) · TESOL Quarterly · doi:10.2307/3586952 · Cited by 96 · see citing works
Halfaker, A., Kittur, A., Kraut, R. E., & Riedl, J. (2009) · International Symposium on Wikis and Open Collaboration (WikiSym) · doi:10.1145/1641309.1641332 · Cited by 95 · see citing works
“found that the previous funding rate of an NSF applicant was not highly correlated with the probability of the current application being funded [8].”
BackgroundThe sociology of scientific validity: How professional networks shape judgement in peer review
Teplitskiy, M., Acuna, D., Elamrani-Raoult, A., Körding, K., & Evans, J. (2018) · Research Policy · doi:10.1016/j.respol.2018.06.014 · Cited by 113 · see citing works
Panel discussion does not improve reliability of peer review for medical research grant proposals.
Fogelholm, M., Leppinen, S., Auvinen, A., Raitanen, J., Nuutinen, A., & Väänänen, K. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.05.001 · Cited by 83 · see citing works
“These differences mean that the fate of a particular manuscript or proposal will partly depend on the reviewers selected [7,8].”
BackgroundResultInfluentialNo time for that now! Qualitative changes in manuscript peer review during the Covid-19 pandemic
Horbach, S. P. J. M. (2020) · Research Evaluation · doi:10.1093/reseval/rvaa037 · Cited by 82 · see citing works
The Air We Breathe: A Critical Look at Practices and Alternatives in the Peer-Review Process
Suls, J. M., & Martin, R. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01105.x · Cited by 85 · see citing works
Peer Review Verfahren auf dem Prüfstand / Peer Review Research – Reviewed
Hirschauer, S. (2004) · Zeitschrift für Soziologie · doi:10.1515/zfsoz-2004-0104 · Cited by 84 · see citing works
Neglected considerations in the analysis of agreement among journal referees
Hargens, L. L., & Herting, J. R. (1990) · Scientometrics · doi:10.1007/bf02130467 · Cited by 82 · see citing works
Is Peer Review a Game of Chance?
Neff, B. D., & Olden, J. D. (2006) · BioScience · doi:10.1641/0006-3568(2006)56[333:iprago]2.0.co;2 · Cited by 81 · see citing works
Roumbanis, L. (2019) · Science Technology & Human Values · doi:10.1177/0162243918822744 · Cited by 79 · see citing works
Norton, R. A. (1999) · Journal of Agricultural and Food Chemistry · doi:10.1021/jf980995t · Cited by 80 · see citing works
From Manuscript Evaluation to Article Valuation: The Changing Technologies of Journal Peer Review
Pontille, D., & Torny, D. (2014) · Human Studies · doi:10.1007/s10746-014-9335-z · Cited by 78 · see citing works
The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications
Gallo, S. A., Sullivan, J., & Glisson, S. R. (2016) · PLoS ONE · doi:10.1371/journal.pone.0165147 · Cited by 76 · see citing works
“Cole et al. (1981) has suggested that reviewer disagreement can, in part, be attributed to the process of translating opinion to score as well as epistemological differences in the quality of science (among individuals and among fields), while many studies have suggested a variety of individual…”
BackgroundCommunicating and compromising on disciplinary expertise in the peer review of research proposals
Huutoniemi, K. I. (2012) · Social Studies of Science · doi:10.1177/0306312712458478 · Cited by 73 · see citing works
“This hardly means that the outcome of evaluation depends mainly on chance, as some have suggested (Cole et al., 1981).”
BackgroundProblems Identified by Secondary Review of Accepted Manuscripts
Garfunkel, J. M. (1990) · JAMA · doi:10.1001/jama.1990.03440100077011 · Cited by 72 · see citing works
Peering at peer review revealed high degree of chance associated with funding of grant applications
Mayo, N. E., Brophy, J. M., GOLDBERG, M., et al. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.12.007 · Cited by 71 · see citing works
The crisis of peer review: Part of the evolution of science
Horta, H., & Jung, J. (2024) · Higher Education Quarterly · doi:10.1111/hequ.12511 · Cited by 77 · see citing works
MAPPING OF SCIENCE: POSSIBILITIES AND LIMITATIONS
Rip, A. (1988) · Elsevier eBooks · doi:10.1016/b978-0-444-70537-2.50014-3 · Cited by 69 · see citing works
An evaluation of the quality of therapeutic studies in perinatal medicine
Tyson, J. E., Furzán, J. A., REISCH, J. S., & Mize, S. G. (1983) · The Journal of Pediatrics · doi:10.1016/s0022-3476(83)80277-7 · Cited by 67 · see citing works
What do we know about grant peer review in the health sciences?
Guthrie, S., Ghiga, I., & Wooding, S. (2017) · F1000Research · doi:10.12688/f1000research.11917.1 · Cited by 69 · see citing works
The knowledge pool: Measurement challenges in evaluating fundamental research programs
Cozzens, S. E. (1997) · Evaluation and Program Planning · doi:10.1016/s0149-7189(96)00038-9 · Cited by 68 · see citing works
Do Journal Rejection Rates Index Consensus?
Cole, S., Simon, G., & Cole, J. R. (1988) · American Sociological Review · doi:10.2307/2095740 · Cited by 66 · see citing works
Self-citationHerron, D. (2012) · Surgical Endoscopy · doi:10.1007/s00464-012-2171-1 · Cited by 67 · see citing works
“A 1981 study of 150 proposals submitted to the National Science Foundation found so much disagreement among reviewers that grant selection ‘‘depend[ed] in a large proportion of cases upon which reviewers happened to be selected for it’’ [3].”
BackgroundThe Academic Mind at the Top: The Political Behavior and Values of Faculty Elites
Lipset, S. M. (1982) · Public Opinion Quarterly · doi:10.1086/268710 · Cited by 65 · see citing works
van Arensbergen, P., van der Weijden, I., & van den Besselaar, P. (2014) · Research Evaluation · doi:10.1093/reseval/rvu017 · Cited by 64 · see citing works
“However, already 30 years ago the study of Cole, Cole, and Simon (1981) on funding decisions within the National Science Foundation could not confirm this.”
BackgroundAssessing non-standard article impact using F1000 labels
Mohammadi, E., & Thelwall, M. A. (2013) · Scientometrics · doi:10.1007/s11192-013-0993-9 · Cited by 63 · see citing works
Sauder, M. (2020) · Sociological Theory · doi:10.1177/0735275120941178 · Cited by 64 · see citing works
How reliable is peer review of scientific abstracts?
Rubin, H. R., Redelmeier, D. A., Wu, A. W., & Steinberg, E. P. (1993) · Journal of General Internal Medicine · doi:10.1007/bf02600092 · Cited by 62 · see citing works
Scientific Grant FundingPreprint
Azoulay, P., & Li, D. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3563957 · Cited by 62 · see citing works
How Professors Think: Inside the Curious World of Academic Judgment
Langfeldt, L. (2011) · International Studies in the Philosophy of Science · doi:10.1080/02698595.2011.552423 · Cited by 59 · see citing works
Roumbanis, L. (2016) · Social Studies of Science · doi:10.1177/0306312716659789 · Cited by 59 · see citing works
“Cole et al. (1981) conducted an experimental study of differences in assessments of 150 grant proposals, reviewed by the National Science Foundation (NSF) panel group as well as by a control group, and found a remarkable amount of variation: The fate of a particular grant application is roughly…”
BackgroundA avaliação da ciência e a revisão por pares: passado e presente. Como será o futuro?
Davyt, A., & Velho, L. (2000) · História Ciências Saúde-Manguinhos · doi:10.1590/s0104-59702000000200005 · Cited by 58 · see citing works
Diamond, A. M. (1996) · Knowledge, technology & policy/Knowledge in society/Knowledge, technology, & policy · doi:10.1007/bf02696298 · Cited by 58 · see citing works
“accept the school's position, still find a need for what Cozzens and Gieryn (1990) call a "post-relativist" (12) attitude that explores theoretical and empirical "roads not taken" (13).”
BackgroundThe Nonuse of Psychological Research at Two Federal Agencies
Arkes, H. R. (2003) · Psychological Science · doi:10.1111/1467-9280.01410 · Cited by 58 · see citing works
“For example, Cole, Cole, and Simon (1981) found that the variance in NSF proposal ratings attributable to the reviewer was approximately twice the size of the variance attributable to the proposal.”
BackgroundAn Examination of Sources of Peer-Review Bias
Blackburn, J. L., & Hakel, M. D. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01715.x · Cited by 57 · see citing works
“In a study of grant applications, Cole et al. (1981) found that 43 to 60% of the variance in ratings was attributable to reviewers rather than the applications themselves.”
BackgroundPerformance Indicators in Education
Hattie, J. (1990) · Australian Journal of Education · doi:10.1177/000494419003400304 · Cited by 57 · see citing works
A physics editor comments on Peters and Ceci's peer-review study
Adair, R. K. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011195 · Cited by 57 · see citing works
Open science and modified funding lotteries can impede the natural selection of bad science
Smaldino, P. E., Turner, M. A., & Kallens, P. C. (2019) · Royal Society Open Science · doi:10.1098/rsos.190194 · Cited by 53 · see citing works
Creativity as a Constrained Stochastic Process.
Simonton, D. K. (2006) · American Psychological Association eBooks · doi:10.1037/10692-006 · Cited by 56 · see citing works
The insufficiencies of methodological inadequacy
Hogan, R. T. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011420 · Cited by 56 · see citing works
Gordon, R., & Poulin, B. J. (2009) · Accountability in Research · doi:10.1080/08989620802689821 · Cited by 55 · see citing works
“…decisions on who to fund are made with unacceptable D o w n l o a d e d B y : [ C a n a d i a n R e s e a r c h K n o w l e d g e N e t w o r k ] A t : 1 8 : 3 1 1 5 J u n e 2 0 0 9 frequency (Cole and Stephen, 1981; Cole et al., 1981; Humphreys et al., 1982; Hodgson, 1997; Mayo et al., 2006).”
BackgroundOrtega, J. L. (2017) · Scientometrics · doi:10.1007/s11192-017-2399-6 · Cited by 55 · see citing works
“Cole and Simon (1981) concluded that there was a high degree of disagreement among reviewers. Rothwell and Martyn (2000) also found that the agreement between reviewers in clinical neuroscience was little greater than would be expected by chance alone.”
BackgroundReliability, fairness, objectivity and other inappropriate goals in peer review
Bailar, J. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065705 · Cited by 53 · see citing works
Bornmann, L. (2008) · Human architecture · Cited by 53 · see citing works
- Authors
- Cole, S., Cole, J. R., & Simon, G. A.
- Affiliations
- Stony Brook University (US)
- Columbia University (US)
- ORCID
- Cole, Stephen
- Venue
- Science
- Published
- 1981-11-20
- Volume
- 214
- Issue
- 4523
- Pages
- 881-886
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2091375917
- Semantic Scholar
- 20d3ad461253ead52398dcc0b194c5f5af1f6e08
- Cited by
- 623
- References
- 3
- Citations by year
- 2026: 15 · 2025: 16 · 2024: 16 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 27 · 2019: 24 · 2018: 18 · 2017: 17 · 2016: 21 · 2015: 23 · 2014: 22 · 2013: 15 · 2012: 18
- FWCI
- 279.85
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- DigitalGeorgetown (Georgetown University Library) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Research, Science, and Academia
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- peer review, research funding, National Science Foundation, reviewer disagreement, reviewer selection, scientific consensus, research evaluation
- MeSH terms
- Peer Review, Research Support as Topic, United States
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
