Cited by 623Research, Science, and Academiascientometrics and bibliometrics researchPsychology Research and Bibliometrics

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Chance and Consensus in Peer Review

Cole, Stephen, Cole, Jonathan R., Simon, Gary A.·1981·Science

DOI 10.1126/science.7302566

Open this paper in CoChat — free account

Abstract

An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to a significant extent on chance. The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it. No evidence of systematic bias in the selection of NSF reviewers was found.

Integrity signals

Registry

Crossref · 10.1126/science.7302566

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Toward a Metric of Science: The Advent of Science Indicators.
Wilken, P. H., Elkana, Y., Lederberg, J., Merton, R. Κ., Thackray, A. W., & Zuckerman, H. (1979) · Social Forces · doi:10.2307/2577293 · Cited by 280
Peer Review and the Support of Science
Cole, S., Rubin, L., & Cole, J. R. (1977) · Scientific American · doi:10.1038/scientificamerican1077-34 · Cited by 140
Self-citation
Which researcher will get the grant?
Cole, J. R., & Cole, S. (1979) · Nature · doi:10.1038/279575a0 · Cited by 21
Self-citation

Show all 3 references

Research works printed but not linked: 1

COLE, S, METRIC SCI ADVENT SC: 162 (1978).

Sorted by citations

Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447
Cumulative Advantage as a Mechanism for Inequality: A Review of Theoretical and Empirical Developments
DiPrete, T. A., & Eirich, G. M. (2006) · Annual Review of Sociology · doi:10.1146/annurev.soc.32.061604.123127 · Cited by 2,069
“A second consequence was that chance events (e.g., unequal luck in the draw of reviewers upon submitting a grant proposal for funding (Cole, Cole, and Simon 1981) would produce a relative advantage for one of two individuals of identical talent, and this relative advantage could persist and…”
Background
From translations to problematic networks: An introduction to co-word analysis
Callon, M., Courtial, J.-P., Turner, W. A., & Bauin, S. (1983) · Social Science Information · doi:10.1177/053901883022002003 · Cited by 1,951
Nepotism and sexism in peer-review
Wennerås, C., & Wold, A. E. (1997) · Nature · doi:10.1038/387341a0 · Cited by 1,346
Peer-review practices of psychological journals: The fate of published articles, submitted again
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011183 · Cited by 1,079
Bias in peer review
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
The Impact of Gender on the Review of the Curricula Vitae of Job Applicants and Tenure Candidates: A National Empirical Study
Steinpreis, R. E., Anders, K. A., & Ritzke, D. M. (1999) · Sex Roles · doi:10.1023/a:1018839203698 · Cited by 736
Scientific creativity as constrained stochastic behavior: The integration of product, person, and process perspectives.
Simonton, D. K. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.4.475 · Cited by 744
The Matthew effect in science funding
Bol, T., de Vaan, M., & van de Rijt, A. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1719557115 · Cited by 638
“To the extent that luck plays a role in early academic success (11), the Matthew effect may undermine meritocracy by allowing an initially fortunate scientist’s recognition to self-perpetuate, while an equally talented but initially less fortunate counterpart remains underappreciated (1, 5–7).”
Background
Impact of bibliometrics upon the science system: Inadvertent consequences?
Weingart, P. (2005) · Scientometrics · doi:10.1007/s11192-005-0007-7 · Cited by 603
The Hierarchy of the Sciences?
Cole, S. (1983) · American Journal of Sociology · doi:10.1086/227835 · Cited by 485
Self-citation
Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance and Resource Allocation in Science
Boudreau, K. J., Guinan, E. C., Lakhani, K. R., & Riedl, C. (2014) · Management Science · doi:10.1287/mnsc.2015.2285 · Cited by 518
“Variance among reviewers is sometimes greater than variance between submissions (Cole et al. 1981).”
Background
Creativity in science: chance, logic, genius, and zeitgeist
(2005) · Choice Reviews Online · doi:10.5860/choice.42-2773 · Cited by 447
Creativity in Science
Simonton, D. K. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165358 · Cited by 445
Scientific peer review
Bornmann, L. (2011) · Annual Review of Information Science and Technology · doi:10.1002/aris.2011.1440450112 · Cited by 567
The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496
Measuring the effectiveness of scientific gatekeeping
Siler, K., Lee, K. P., & Bero, L. A. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1418218112 · Cited by 356
“(56) argued that roughly one-half of a competitive National Science Foundation granting process was a matter of luck.”
Background
A review of bibliometric and other science indicators and their role in research evaluation
King, J. (1987) · Journal of Information Science · doi:10.1177/016555158701300501 · Cited by 334
Giftedness and evidence for reproducibly superior performance: an account based on the expert performance framework
Ericsson, K. A., Roring, R. W., & Nandagopal, K. (2007) · High Ability Studies · doi:10.1080/13598130701350593 · Cited by 304
“For instance, studies investigating the peer-review process reveal considerable inconsistency among referees, illustrating a large role played by chance in determining the success of a grant proposal or journal article (Cole et al., 1981; Fiske & Fogg, 1990; Cicchetti, 1991).”
Background
Citations and journal impact factors: questionable indicators of research quality
Seglen, P. O. (1997) · Allergy · doi:10.1111/j.1398-9995.1997.tb00175.x · Cited by 268
“, giving the whole process the character of a lottery (1-4).”
Background
Networked Participatory Scholarship: Emergent techno-cultural pressures toward open and digital scholarship in online networks
Veletsianos, G., & Kimmons, R. M. (2011) · Computers & Education · doi:10.1016/j.compedu.2011.10.001 · Cited by 260
Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone?
Rothwell, P. M. (2000) · Brain · doi:10.1093/brain/123.9.1964 · Cited by 254
New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System
Travis, G., & Collins, H. M. (1991) · Science Technology & Human Values · doi:10.1177/016224399101600303 · Cited by 247
“The results are presented in monographs on the two phases of the study (Cole, Rubin, and Cole 1978; Cole and Cole 1981) and two articles (Cole, Rubin, and Cole 1977; Cole, Cole, and Simon 1981).”
Background
Citation rates and journal impact factors are not suitable for evaluation of research
Seglen, P. O. (1998) · Acta Orthopaedica Scandinavica · doi:10.3109/17453679809000920 · Cited by 240
Peer Review and the Social Construction of Knowledge in the Management Discipline
Bedeian, A. G. (2004) · Academy of Management Learning and Education · doi:10.5465/amle.2004.13500489 · Cited by 234
“Recognizing that the likelihood of refusing to make any recommendations for revision and still have their work published is slim or none, these authors give ground in some areas and hold fast in others. This jockeying, of course, introduces a further element of risk into the review process, as editors and referees may…”
Background
Big names or big ideas: Do peer-review panels select the best science proposals?
Li, D., & Agha, L. (2015) · Science · doi:10.1126/science.aaa0185 · Cited by 227
Low agreement among reviewers evaluating the same NIH grant applications
Pier, E. L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M. J., Ford, C. E., & Carnes, M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1714379115 · Cited by 210
The Manuscript Review and Decision-Making Process
Bakanic, V., McPhail, C., & Simon, R. J. (1987) · American Sociological Review · doi:10.2307/2095599 · Cited by 200
Interrater agreement for journal manuscript reviews.
Whitehurst, G. J. “. (1984) · American Psychologist · doi:10.1037/0003-066x.39.1.22 · Cited by 189
We Have to Break Up
Cialdini, R. B. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01091.x · Cited by 189
Governing Science
Gläser, J., & Laudel, G. (2016) · European Journal of Sociology · doi:10.1017/s0003975616000047 · Cited by 201
Conservatism and risk-taking in peer review: Emerging ERC practices
Luukkonen, T. (2012) · Research Evaluation · doi:10.1093/reseval/rvs001 · Cited by 185
“The most debated area is the ‘grey’ boundary area around the line demarcating the proposals that will be funded and those that will not (Cole et al., 1981).”
Background
Collaboration, interdisciplinarity, and the epistemology of contemporary science
Andersen, H. M. (2015) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2015.10.006 · Cited by 181
Multidimensional Assessment of Scholarly Research Impact
Moed, H. F., & Halevi, G. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23314 · Cited by 168
PFEFFER'S Barriers to The Advance Of Organizational Science: A Rejoinder
Cannella, A. A., & Paetzold, R. L. (1994) · Academy of Management Review · doi:10.5465/amr.1994.9410210755 · Cited by 208
Opening Science
Pampel, H., & Dallmeier-Tiessen, S. (2013) · Springer eBooks · doi:10.1007/978-3-319-00026-8 · Cited by 241
Research Funding: the Case for a Modified Lottery
Fang, F. C., & Casadevall, A. (2016) · mBio · doi:10.1128/mbio.00422-16 · Cited by 186
“found that the odds of a proposal submitted to the National Science Foundation (NSF) getting funded were largely based on chance—the chance that specific reviewers would be chosen (26).”
Barriers to scientific contributions: The author's formula
Armstrong, J. S. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011201 · Cited by 151
The psychometrics of everyday life
Kunda, Z., & Nisbett, R. E. (1986) · Cognitive Psychology · doi:10.1016/0010-0285(86)90012-5 · Cited by 157
“Actual correlations for NSF proposals were estimated from data for solid state physics and economics panels obtained by Cole, Cole, and Simon (1981).”
Method
Peer review of grant applications: what do we know?
Wessely, S. C. (1998) · The Lancet · doi:10.1016/s0140-6736(97)11129-1 · Cited by 146
Human subjects review, personal values, and the regulation of social science research.
Ceci, S. J., Peters, D. P., & Plotkin, J. (1985) · American Psychologist · doi:10.1037/0003-066x.40.9.994 · Cited by 146
The Social Construction of Two Biologists' Proposals
Myers, G. (1985) · Written Communication · doi:10.1177/0741088385002003001 · Cited by 142
Funding Science: The Real Defects of Peer Review and An Alternative To It
Roy, R. (1985) · Science Technology & Human Values · doi:10.1177/016224398501000309 · Cited by 140
Funding grant proposals for scientific research: retrospective analysis of scores by members of grant review panel
Graves, N., Barnett, A. G., & Clarke, P. (2011) · BMJ · doi:10.1136/bmj.d4797 · Cited by 134
Overcoming the Illegitimacy Discount: Cultural Entrepreneurship in the US Feature Film Industry
Zhao, E. Y., Ishihara, M., & Lounsbury, M. D. (2013) · Organization Studies · doi:10.1177/0170840613485844 · Cited by 126
Fairness as Appropriateness
Mallard, G., Lamont, M., & Guetzkow, J. (2009) · Science Technology & Human Values · doi:10.1177/0162243908329381 · Cited by 124
The Ratings Game: Asymmetry in Classification
Waguespack, D. M., & Sorenson, O. (2010) · Organization Science · doi:10.1287/orsc.1100.0533 · Cited by 124
“…have time and again confirmed that, when evaluators know the identities of the individuals being evaluated, biases creep into the process with high-status participants (and usually men) receiving more favorable evaluations (e.g., Cole et al. 1981, Blank 1991, Goldin and Rouse 2000).”
Background
Open exchange and epistemic progress.
Mahoney, M. J. (1985) · American Psychologist · doi:10.1037/0003-066x.40.1.29 · Cited by 121
Agricultural Science and Sustainable Agriculture: a Review of the Existing Scientific Barriers to Sustainable Food Production and Potential Solutions
MacRae, R. J., Hill, S. B., Henning, J., & Mehuys, G. R. (1989) · Biological Agriculture & Horticulture · doi:10.1080/01448765.1989.9754518 · Cited by 125
Research and teaching: symbiosis or conflict
Elton, L. (1986) · Higher Education · doi:10.1007/bf00129218 · Cited by 115
Quality, Quantity, and Age: The Careers of Ten Distinguished Psychologists
Simonton, D. K. (1986) · The International Journal of Aging and Human Development · doi:10.2190/kb7e-a45m-x8x7-dqj4 · Cited by 113
Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes
Bornmann, L., Wallon, G., & Ledin, A. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003480 · Cited by 113
“Cole and his colleagues [8] concluded in their highly influential study on grant peer review at the National Science Foundation (NSF, Arlington, VA, USA) that ‘‘the fate of a particular application is roughly half determined by the characteristics of the proposal and the principal investigator, and about half by appar…”
Background
Tracking Replicability as a Method of Post-Publication Open Evaluation
Hartshorne, J. K., & Schachner, A. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00008 · Cited by 107
“Peer review, the primary vetting procedure for publication, is often slow, contentious, and uneven (Mahoney, 1977; Cole et al., 1981; Peters and Ceci, 1982; Eysenck and Eysenck, 1992; Newton, 2010).”
Background
What do we know about grant peer review in the health sciences?
Guthrie, S., Ghiga, I., & Wooding, S. (2018) · F1000Research · doi:10.12688/f1000research.11917.2 · Cited by 104
How to avoid borrowed plumes in academia
Osterloh, M., & Frey, B. S. (2019) · Research Policy · doi:10.1016/j.respol.2019.103831 · Cited by 113
“It is also indicated by empirical findings on the “luck of the reviewer draw” (Cole et al., 1981; Bornmann and Daniel, 2009), which in many cases is decisive for the acceptance or rejection of a grant proposal or paper.”
Background
Peer review for improving the quality of grant applications
Demicheli, V., & Pietrantonj, C. D. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000003.pub2 · Cited by 99
Written Argumentation for Real Audiences: Suggestions for Teacher Research and Classroom Practice
Johns, A. M. (1993) · TESOL Quarterly · doi:10.2307/3586952 · Cited by 96
A jury of your peers
Halfaker, A., Kittur, A., Kraut, R. E., & Riedl, J. (2009) · International Symposium on Wikis and Open Collaboration (WikiSym) · doi:10.1145/1641309.1641332 · Cited by 95
“found that the previous funding rate of an NSF applicant was not highly correlated with the probability of the current application being funded [8].”
Background
The sociology of scientific validity: How professional networks shape judgement in peer review
Teplitskiy, M., Acuna, D., Elamrani-Raoult, A., Körding, K., & Evans, J. (2018) · Research Policy · doi:10.1016/j.respol.2018.06.014 · Cited by 113
Panel discussion does not improve reliability of peer review for medical research grant proposals.
Fogelholm, M., Leppinen, S., Auvinen, A., Raitanen, J., Nuutinen, A., & Väänänen, K. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.05.001 · Cited by 83
“These differences mean that the fate of a particular manuscript or proposal will partly depend on the reviewers selected [7,8].”
BackgroundResultInfluential
No time for that now! Qualitative changes in manuscript peer review during the Covid-19 pandemic
Horbach, S. P. J. M. (2020) · Research Evaluation · doi:10.1093/reseval/rvaa037 · Cited by 85
The Air We Breathe: A Critical Look at Practices and Alternatives in the Peer-Review Process
Suls, J. M., & Martin, R. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01105.x · Cited by 88
Peer Review Verfahren auf dem Prüfstand / Peer Review Research – Reviewed
Hirschauer, S. (2004) · Zeitschrift für Soziologie · doi:10.1515/zfsoz-2004-0104 · Cited by 85
Neglected considerations in the analysis of agreement among journal referees
Hargens, L. L., & Herting, J. R. (1990) · Scientometrics · doi:10.1007/bf02130467 · Cited by 82
Is Peer Review a Game of Chance?
Neff, B. D., & Olden, J. D. (2006) · BioScience · doi:10.1641/0006-3568(2006)56[333:iprago]2.0.co;2 · Cited by 81
Peer Review or Lottery? A Critical Analysis of Two Different Forms of Decision-making Mechanisms for Allocation of Research Grants
Roumbanis, L. (2019) · Science Technology & Human Values · doi:10.1177/0162243918822744 · Cited by 79
Inhibition of Aflatoxin B1 Biosynthesis in Aspergillus flavus by Anthocyanidins and Related Flavonoids
Norton, R. A. (1999) · Journal of Agricultural and Food Chemistry · doi:10.1021/jf980995t · Cited by 80
From Manuscript Evaluation to Article Valuation: The Changing Technologies of Journal Peer Review
Pontille, D., & Torny, D. (2014) · Human Studies · doi:10.1007/s10746-014-9335-z · Cited by 83
The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications
Gallo, S. A., Sullivan, J., & Glisson, S. R. (2016) · PLoS ONE · doi:10.1371/journal.pone.0165147 · Cited by 76
“Cole et al. (1981) has suggested that reviewer disagreement can, in part, be attributed to the process of translating opinion to score as well as epistemological differences in the quality of science (among individuals and among fields), while many studies have suggested a variety of individual…”
Background
Communicating and compromising on disciplinary expertise in the peer review of research proposals
Huutoniemi, K. I. (2012) · Social Studies of Science · doi:10.1177/0306312712458478 · Cited by 73
“This hardly means that the outcome of evaluation depends mainly on chance, as some have suggested (Cole et al., 1981).”
Background
Problems Identified by Secondary Review of Accepted Manuscripts
Garfunkel, J. M. (1990) · JAMA · doi:10.1001/jama.1990.03440100077011 · Cited by 72
Peering at peer review revealed high degree of chance associated with funding of grant applications
Mayo, N. E., Brophy, J. M., GOLDBERG, M., et al. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.12.007 · Cited by 71
The crisis of peer review: Part of the evolution of science
Horta, H., & Jung, J. (2024) · Higher Education Quarterly · doi:10.1111/hequ.12511 · Cited by 77
MAPPING OF SCIENCE: POSSIBILITIES AND LIMITATIONS
Rip, A. (1988) · Elsevier eBooks · doi:10.1016/b978-0-444-70537-2.50014-3 · Cited by 69
An evaluation of the quality of therapeutic studies in perinatal medicine
Tyson, J. E., Furzán, J. A., REISCH, J. S., & Mize, S. G. (1983) · The Journal of Pediatrics · doi:10.1016/s0022-3476(83)80277-7 · Cited by 67
What do we know about grant peer review in the health sciences?
Guthrie, S., Ghiga, I., & Wooding, S. (2017) · F1000Research · doi:10.12688/f1000research.11917.1 · Cited by 70
The knowledge pool: Measurement challenges in evaluating fundamental research programs
Cozzens, S. E. (1997) · Evaluation and Program Planning · doi:10.1016/s0149-7189(96)00038-9 · Cited by 68
Do Journal Rejection Rates Index Consensus?
Cole, S., Simon, G., & Cole, J. R. (1988) · American Sociological Review · doi:10.2307/2095740 · Cited by 66
Self-citation
Is expert peer review obsolete? A model suggests that post-publication reader review may exceed the accuracy of traditional peer review
Herron, D. (2012) · Surgical Endoscopy · doi:10.1007/s00464-012-2171-1 · Cited by 67
“A 1981 study of 150 proposals submitted to the National Science Foundation found so much disagreement among reviewers that grant selection ‘‘depend[ed] in a large proportion of cases upon which reviewers happened to be selected for it’’ [3].”
Background
The Academic Mind at the Top: The Political Behavior and Values of Faculty Elites
Lipset, S. M. (1982) · Public Opinion Quarterly · doi:10.1086/268710 · Cited by 65
The selection of talent as a group process. A literature review on the social dynamics of decision making in grant panels
van Arensbergen, P., van der Weijden, I., & van den Besselaar, P. (2014) · Research Evaluation · doi:10.1093/reseval/rvu017 · Cited by 64
“However, already 30 years ago the study of Cole, Cole, and Simon (1981) on funding decisions within the National Science Foundation could not confirm this.”
Background
Assessing non-standard article impact using F1000 labels
Mohammadi, E., & Thelwall, M. A. (2013) · Scientometrics · doi:10.1007/s11192-013-0993-9 · Cited by 63
A Sociology of Luck
Sauder, M. (2020) · Sociological Theory · doi:10.1177/0735275120941178 · Cited by 49
How reliable is peer review of scientific abstracts?
Rubin, H. R., Redelmeier, D. A., Wu, A. W., & Steinberg, E. P. (1993) · Journal of General Internal Medicine · doi:10.1007/bf02600092 · Cited by 62
Scientific Grant FundingPreprint
Azoulay, P., & Li, D. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3563957 · Cited by 61
How Professors Think: Inside the Curious World of Academic Judgment
Langfeldt, L. (2011) · International Studies in the Philosophy of Science · doi:10.1080/02698595.2011.552423 · Cited by 59
Academic judgments under uncertainty: A study of collective anchoring effects in Swedish Research Council panel groups
Roumbanis, L. (2016) · Social Studies of Science · doi:10.1177/0306312716659789 · Cited by 59
“Cole et al. (1981) conducted an experimental study of differences in assessments of 150 grant proposals, reviewed by the National Science Foundation (NSF) panel group as well as by a control group, and found a remarkable amount of variation: The fate of a particular grant application is roughly…”
Background
A avaliação da ciência e a revisão por pares: passado e presente. Como será o futuro?
Davyt, A., & Velho, L. (2000) · História Ciências Saúde-Manguinhos · doi:10.1590/s0104-59702000000200005 · Cited by 58
The economics of science
Diamond, A. M. (1996) · Knowledge, technology & policy/Knowledge in society/Knowledge, technology, & policy · doi:10.1007/bf02696298 · Cited by 58
“accept the school's position, still find a need for what Cozzens and Gieryn (1990) call a "post-relativist" (12) attitude that explores theoretical and empirical "roads not taken" (13).”
Background
The Nonuse of Psychological Research at Two Federal Agencies
Arkes, H. R. (2003) · Psychological Science · doi:10.1111/1467-9280.01410 · Cited by 59
“For example, Cole, Cole, and Simon (1981) found that the variance in NSF proposal ratings attributable to the reviewer was approximately twice the size of the variance attributable to the proposal.”
Background
An Examination of Sources of Peer-Review Bias
Blackburn, J. L., & Hakel, M. D. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01715.x · Cited by 57
“In a study of grant applications, Cole et al. (1981) found that 43 to 60% of the variance in ratings was attributable to reviewers rather than the applications themselves.”
Background
Performance Indicators in Education
Hattie, J. (1990) · Australian Journal of Education · doi:10.1177/000494419003400304 · Cited by 57
A physics editor comments on Peters and Ceci's peer-review study
Adair, R. K. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011195 · Cited by 57
Open science and modified funding lotteries can impede the natural selection of bad science
Smaldino, P. E., Turner, M. A., & Kallens, P. C. (2019) · Royal Society Open Science · doi:10.1098/rsos.190194 · Cited by 54
Creativity as a Constrained Stochastic Process.
Simonton, D. K. (2006) · American Psychological Association eBooks · doi:10.1037/10692-006 · Cited by 57
The insufficiencies of methodological inadequacy
Hogan, R. T. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011420 · Cited by 56
Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant
Gordon, R., & Poulin, B. J. (2009) · Accountability in Research · doi:10.1080/08989620802689821 · Cited by 55
“…decisions on who to fund are made with unacceptable D o w n l o a d e d B y : [ C a n a d i a n R e s e a r c h K n o w l e d g e N e t w o r k ] A t : 1 8 : 3 1 1 5 J u n e 2 0 0 9 frequency (Cole and Stephen, 1981; Cole et al., 1981; Humphreys et al., 1982; Hodgson, 1997; Mayo et al., 2006).”
Background
Are peer-review activities related to reviewer bibliometric performance? A scientometric analysis of Publons
Ortega, J. L. (2017) · Scientometrics · doi:10.1007/s11192-017-2399-6 · Cited by 59
“Cole and Simon (1981) concluded that there was a high degree of disagreement among reviewers. Rothwell and Martyn (2000) also found that the agreement between reviewers in clinical neuroscience was little greater than would be expected by chance alone.”
Background
Reliability, fairness, objectivity and other inappropriate goals in peer review
Bailar, J. C. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065705 · Cited by 56
Scientiﬁc Peer Review: An Analysis of the Peer Review Process from the Perspective of Sociology of Science Theories
Bornmann, L. (2008) · Human architecture · Cited by 53

Authors: Cole, S., Cole, J. R., & Simon, G. A.
Affiliations: Stony Brook University (US)
Columbia University (US)
ORCID: Cole, Stephen
Venue: Science
Publisher: American Association for the Advancement of Science (AAAS)
Published: 1981-11-20
Volume: 214
Issue: 4523
Pages: 881-886
ISSN: 0036-8075 (ISSN-L)
1095-9203
DOI: 10.1126/science.7302566
OpenAlex: W2091375917
Semantic Scholar: 20d3ad461253ead52398dcc0b194c5f5af1f6e08
Cited by: 623
References: 3
Citations by year: 2026: 15 · 2025: 16 · 2024: 16 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 27 · 2019: 24 · 2018: 18 · 2017: 17 · 2016: 21 · 2015: 23 · 2014: 22 · 2013: 15 · 2012: 18
FWCI: 279.85
Type: journal article
Language: en
Open access: closed
Locations: Science (publishedVersion)
PubMed (publishedVersion)
DigitalGeorgetown (Georgetown University Library) (submittedVersion)
Topics: Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords: peer review, research funding, National Science Foundation, reviewer disagreement, reviewer selection, scientific consensus, research evaluation
MeSH terms: Peer Review, Research Support as Topic, United States

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Chance and Consensus in Peer Review

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks