Cited by 623Research, Science, and Academiascientometrics and bibliometrics researchPsychology Research and Bibliometrics

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Chance and Consensus in Peer Review

Cole, Stephen, Cole, Jonathan R., Simon, Gary A.·1981·Science

DOI 10.1126/science.7302566

Open this paper in CoChat — free account

Abstract

An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to a significant extent on chance. The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it. No evidence of systematic bias in the selection of NSF reviewers was found.

Integrity signals

Registry

Crossref · 10.1126/science.7302566

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Toward a Metric of Science: The Advent of Science Indicators.
Wilken, P. H., Elkana, Y., Lederberg, J., Merton, R. Κ., Thackray, A. W., & Zuckerman, H. (1979) · Social Forces · doi:10.2307/2577293 · Cited by 280
Peer Review and the Support of Science
Cole, S., Rubin, L., & Cole, J. R. (1977) · Scientific American · doi:10.1038/scientificamerican1077-34 · Cited by 140
Self-citation
Which researcher will get the grant?
Cole, J. R., & Cole, S. (1979) · Nature · doi:10.1038/279575a0 · Cited by 21
Self-citation

Show all 3 references

Research works printed but not linked: 1

COLE, S, METRIC SCI ADVENT SC: 162 (1978).

Sorted by citations

Items 201–300 of 623

Does Peer Review Identify the Best Papers? A Simulation Study of Editors, Reviewers, and the Scientific Publication Process
Esarey, J. (2017) · PS Political Science & Politics · doi:10.1017/s1049096517001081 · Cited by 27
“…major errors in submitted papers Riis, and Karlsson 1994; Schroter et al. 2004; 2008), allows chance to strongly influence whether a paper will be published (Baxt et al. 1998; Cole, Cole, and Simon 1981; Mayo et al. 2006), and is subject to confirmatory biases by peer reviewers (Mahoney 1977).”
Background
Publication rejection among ecologists
Cassey, P., & Blackburn, T. M. (2003) · Trends in Ecology & Evolution · doi:10.1016/s0169-5347(03)00160-5 · Cited by 23
Use of bibliometric information to assist research policy making. A comparison of publication and citation profiles of Full and Associate Professors at a School of Chemistry in Uruguay
Ventura, O. N., & Mombrú, Á. W. (2006) · Scientometrics · doi:10.1007/s11192-006-0154-5 · Cited by 23
Evaluation of performance of health research in the Netherlands
Rigter, H. (1986) · Research Policy · doi:10.1016/0048-7333(86)90021-1 · Cited by 23
Competency testing for reviewers and editors
Yalow, R. S. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011729 · Cited by 23
Perspectives from early career researchers on the publication process in ecology - a response to Statzner & Resh (2010)
Schäfer, R. B., Cooke, S. J., Arlinghaus, R., Bonada‬‬‬‬‬‬‬‬‬‬‬, N., Brischoux, F., Casper, A. F., Catford, J. A., & Rolland, V. (2011) · Freshwater Biology · doi:10.1111/j.1365-2427.2011.02673.x · Cited by 22
“This is especially important as several studies have highlighted the subjectivity of reviewers judgments (Cole et al., 1981; Rothwell & Martyn, 2000; Rasmussen, Langer & Alroe, 2006; Neff & Olden, 2006), which may inhibit the spread of innovative ideas.”
Background
Peer review and journal impact factor: the two pillars of contemporary medical publishing.
Triaridis, S., & Kyrgidis, Α. (2010) · Europe PMC (PubMed Central) · Cited by 22
Advantages and dangers of ‘remote’ peer evaluation
Gläser, J., & Laudel, G. (2005) · Research Evaluation · doi:10.3152/147154405781776085 · Cited by 22
Prescriptions for medical research II--Is medical research well served by peer review?
Gillett, R. (1993) · BMJ · doi:10.1136/bmj.306.6893.1672 · Cited by 22
Disagreement and Agonistic Chance in Peer Review
Roumbanis, L. (2022) · Science, Technology, & Human Values · doi:10.1177/01622439211026016 · Cited by 29
“…of grant proposals ’ quality is influenced by a seemingly unavoidable measure of chance, which is tied to the variation and interplay between various expert judgments (Boudreau et al. 2016; Cole, Cole, and Simon 1981; Chubin and Hackett 1990; Graves, Barnett, and Clarke 2011; Roumbanis 2017).”
Should we fund research randomly? An epistemological criticism of the lottery model as an alternative to peer review for the funding of science
Bedessem, B. (2019) · Research Evaluation · doi:10.1093/reseval/rvz034 · Cited by 21
Critical essay: Blinding faith – Paradoxes and pathologies of opacity in peer review
Willmott, H. C. (2021) · Human Relations · doi:10.1177/00187267211016752 · Cited by 22
“As an author, I doubt that I am alone in responding to pathologies of manuscript evaluation by treating editorial decision making as something of a lottery (Cole et al., 1981) – with the obvious difference that drawing lots is more trans-parently random.”
When nonreliability of reviews indicates solid science
Eckberg, D. L. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065791 · Cited by 21
Improving Participation and Interrater Agreement in Scoring Ambulatory Pediatric Association Abstracts
Kemper, K. J. (1996) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/archpedi.1996.02170290046007 · Cited by 21
Peer Review in Prague
Rennie, D. (1998) · JAMA · doi:10.1001/jama.280.3.214 · Cited by 21
Predictors of applying for and winning an ERC Proof-of-Concept grant: An automated machine learning model
Seeber, M., Alon, I., Pina, D. G., Piro, F. N., & Seeber, M. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122009 · Cited by 20
Preentry Contacts and the Generation of Nascent Networks in Organizations
Sterling, A. D. (2015) · Organ. Sci. · doi:10.1287/orsc.2014.0921 · Cited by 17
Artificial selection versus natural selection: Which causes the Matthew effect of science funding allocation in China?
Zhang, G., Xiong, L., Wang, X., Dong, J., & Duan, H. (2020) · Science and Public Policy · doi:10.1093/scipol/scaa024 · Cited by 20
Distrust in grant peer review—reasons and remedies
Langfeldt, L., Reymert, I., & Svartefoss, S. M. (2023) · Science and Public Policy · doi:10.1093/scipol/scad051 · Cited by 19
Does reviewing experience reduce disagreement in proposals evaluation? Insights from Marie Skłodowska-Curie and COST Actions
Seeber, M., Vlegels, J., Reimink, E., Marušić, A., & Pina, D. G. (2021) · Research Evaluation · doi:10.1093/reseval/rvab011 · Cited by 18
La Psicología Social de la Ciencia: Revisión y discusión de una nueva área de investigación
Íñiguez-Rueda, L., & Pallí, C. (2002) · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · Cited by 18
The case against
Klein, R. (1997) · BMJ · doi:10.1136/bmj.314.7079.503 · Cited by 19
Peer Review and Evaluation of R&D Impacts
Bozeman, B. L. (1993) · Springer eBooks · doi:10.1007/978-1-4757-5182-6_5 · Cited by 19
Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion
Gallo, S. A., Schmaling, K. B., THOMPSON, L. A., & Glisson, S. R. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-020-00093-0 · Cited by 20
Practical Suggestions for Improving Scholarly Peer Review Quality and Reducing Cycle Times
Ralph, P. (2016) · Communications of the Association for Information Systems · doi:10.17705/1cais.03813 · Cited by 17
The lottery in Babylon—On the role of chance in scientific success
Reinhart, M., & Schendzielorz, C. (2020) · Journal of Responsible Innovation · doi:10.1080/23299460.2020.1806429 · Cited by 18
Research evaluation per discipline: a peer-review method and its outcomes
Rons, N. (2008) · VUBIR (Vrije Universiteit Brussel) · Cited by 18
Reflections on the peer review process
Marsh, H. W., & Ball, S. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065912 · Cited by 18
Scientific Misconduct as a Dilemma for Nursing
Hawley, D. J., & Jeffers, J. M. (1992) · Journal of Nursing Scholarship · doi:10.1111/j.1547-5069.1992.tb00699.x · Cited by 18
A randomized trial of fellowships for early career researchers finds a high reliability in funding decisions
Clarke, P., Herbert, D., Graves, N., & Barnett, A. G. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.010 · Cited by 21
“In 1977 the 4 US National Science Foundation re-reviewed 150 proposals using a second independent peer 5 review panel and found a 24–30% disagreement in funding outcomes.(1) A Canadian study of 248 6 proposals submitted to two major funding agencies with similar peer review processes found a 7 27% disagreement in fund…”
Background
Navigating the impact-innovation double hurdle: The case of a climate change research fund
Lettice, F., Smart, P., Baruch, Y., & Johnson, M. (2012) · Research Policy · doi:10.1016/j.respol.2012.03.003 · Cited by 17
Bias in Peer Review of Organic Farming Grant Applications
Rasmussen, J., Langer, V., & Alrøe, H. F. (2006) · Agriculture and Human Values · doi:10.1007/s10460-005-6105-6 · Cited by 15
“In general, peer review studies show substantial reviewer variance and thereby poor agreement between different reviewers of grant applications and manuscripts (Cole et al., 1981; Cicchetti, 1991a; Rothwell and Martyn, 2000).”
BackgroundResultInfluential
Analysis of bibliometric indicators to determine citation bias
Šimko, I. (2015) · Humanities and Social Sciences Communications · doi:10.1057/palcomms.2015.11 · Cited by 17
Science‐Policy‐Practice Interfaces: Emergent knowledge and monarch butterfly conservation
Gustafsson, K. M., Wolf, S. A., & Agrawal, A. A. (2017) · Environmental Policy and Governance · doi:10.1002/eet.1792 · Cited by 17
The evaluation of the quality of research
Rudd, E. (1988) · Studies in Higher Education · doi:10.1080/03075078812331377955 · Cited by 17
The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome
Langfeldt, L. (2022) · Springer eBooks · doi:10.1007/978-3-030-75263-7_13 · Cited by 22
Economic Explanations of the Behaviour of Universities and Scholars
Diamond, A. M. (1993) · Journal of Economic Studies · doi:10.1108/eum0000000000174 · Cited by 17
What do we know about peer review?
Wessely, S. C. (1996) · Psychological Medicine · doi:10.1017/s0033291700035224 · Cited by 17
The Quality of Research Articles in the Journal of Educational Research, 1970 and 1980
Stallings, W. M., West, C. K., & Carmody, C. (1983) · The Journal of Educational Research · doi:10.1080/00220671.1983.10885500 · Cited by 17
Reviewer “bias”: Do Peters and Ceci protest too much?
Perlman, D. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011572 · Cited by 17
The Novelty Paradox & Bias for Normal Science: Evidence from Randomized Medical Grant Proposal EvaluationsPreprint
Boudreau, K. J., Guinan, E. C., Lakhani, K. R., & Riedl, C. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2184791 · Cited by 17
“For example, in their seminal study of NSF grants, Cole et al. (1981) found that the variance among evaluators for a given proposal was higher than the variance between proposals.”
Journal ratings as predictors of articles quality in Arts, Humanities and Social Sciences: an analysis based on the Italian Research Evaluation Exercise
Bonaccorsi, A. J., Cicero, T., Ferrara, A., & Malgarini, M. (2015) · F1000Research · doi:10.12688/f1000research.6478.1 · Cited by 16
A SIMULATION OF DISAGREEMENT FOR CONTROL OF RATIONAL CHEATING IN PEER REVIEW
Grimaldo, F., & Paolucci, M. (2013) · Advances in Complex Systems · doi:10.1142/s0219525913500045 · Cited by 16
Quantitative/qualitative federal research impact evaluation practices
Kostoff, R. N. (1994) · Technological Forecasting and Social Change · doi:10.1016/0040-1625(94)90093-0 · Cited by 16
Under repair: A publication ethics and research record in the making
Jacob, M. (2019) · Social Studies of Science · doi:10.1177/0306312718824663 · Cited by 16
“Peer-review and editorial decisions are distinct, but both offer helpful analogies to approach laterally the practices of the Forum members as a form of collective and ‘live’ peer evaluation (e.g. Cole et al., 1981; Hirschauer, 2009; Zuckerman and Merton, 1971).”
Background
Deviant scientists and scientific deviance
Bechtel, H. K., & Pearson, W. (1985) · Deviant Behavior · doi:10.1080/01639625.1985.9967676 · Cited by 15
Confusion between reviewer reliability and wise editorial and funding decisions
Kiesler, C. A. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065869 · Cited by 16
Editorial responsibilities in manuscript review
Crandall, R. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011316 · Cited by 16
Reflections from the peer review mirror
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00066024 · Cited by 16
Decision-making in expert panels evaluating research: Constraints, processes and bias
Langfeldt, L. (2002) · NORA - Norwegian Open Research Archives · Cited by 15
Supplementary Report I to the Independent Review of the Role of Metrics in Research Assessment and Management
Wouters, P. F., Thelwall, M. A., Kousha, K., Waltman, L., de Rijcke, S., & Franssen, T. (2015) · Cited by 16
The urgency for investment on local data for advancing food assessments in Africa: A review case study for APSIM crop modeling
Carcedo, A. J. P., Vieira, N. A., Marziotte, L., Correndo, A. A., Araya, A., Prasad, P. V. V., Min, D., Stewart, Z., Faye, A., & Ciampitti, I. A. (2023) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2023.105633 · Cited by 16
The impact of early debut on scientists: Evidence from the Young Scientists Fund of the NSFC
Zhang, W., Wang, X., Chen, H., & Liu, J. (2023) · Research Policy · doi:10.1016/j.respol.2023.104935 · Cited by 16
Peer review in scientifically small countries
Pouris, A. (1988) · R and D Management · doi:10.1111/j.1467-9310.1988.tb00608.x · Cited by 15
Impact vitality: an indicator based on citing publications in search of excellent scientists
Rons, N., & Amez, L. M. (2009) · Research Evaluation · doi:10.3152/095820209x470563 · Cited by 15
Ejes de discusión en la evaluación de la ciencia: revisión por pares, bibliometría y pertinencia
Sarthou, N. F. (2016) · Revista de Estudios Sociales · doi:10.7440/res58.2016.06 · Cited by 15
Bibliometrically Disciplined Peer Review: on Using Indicators in Research Evaluation
van den Besselaar, P., & Sandström, U. (2020) · Scholarly Assessment Reports · doi:10.29024/sar.16 · Cited by 15
Toward an Epistemologically-Relevant Sociology of Science
Campbell, D. T. (1985) · Science Technology & Human Values · doi:10.1177/016224398501000106 · Cited by 15
Reform peer review: The Peters and Ceci study in the context of other current studies of scientific evaluation
Manwell, C., & Baker, C. M. A. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011493 · Cited by 15
Do grant proposal texts matter for funding decisions? A field experiment
Şimşek, M., de Vaan, M., & van de Rijt, A. (2024) · Scientometrics · doi:10.1007/s11192-024-04968-7 · Cited by 15
Evaluation and Decision Making in Higher Education
Posselt, J. R., Hernandez, T. E., Villarreal, C. D., Rodgers, A. J., & Irwin, L. N. (2020) · Higher education · doi:10.1007/978-3-030-31365-4_8 · Cited by 14
The Bias of Sighted Reviewers in Research Proposal Evaluation: A Comparative Analysis of Blind and Open Review in Korea
Lee, M., Om, K., & Koh, J. (2000) · Scientometrics · doi:10.1023/a:1005636503358 · Cited by 14
Reliability and bias in peer-review practices
Rosenthal, R. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011614 · Cited by 15
Academic Evaluation in Higher Education
Hamann, J., & Beljean, S. (2017) · Springer eBooks · doi:10.1007/978-94-017-9553-1_295-1 · Cited by 14
Partial lottery can make grant allocation more fair, more efficient, and more diverse
Horbach, S. P. J. M., Tijdink, J. K., & Bouter, L. M. (2022) · Science and Public Policy · doi:10.1093/scipol/scac009 · Cited by 13
Gender bias in funding evaluation: A randomized experiment
Castro, L. C., & Menéndez, L. S. (2023) · Quantitative Science Studies · doi:10.1162/qss_a_00263 · Cited by 13
“Since the classic study at the NSF by the Coles (Cole & Cole, 1979, 1981; Cole, Cole, & Simon, 1981; Cole, Rubin, & Cole, 1977, 1978), which found small differences in funding by gender but more related to rank or past performance, the controversy has continued.”
Background
Research Funding in Open Science
Eisfeld-Reschke, J., Herb, U., & Wenzlaff, K. (2013) · Springer eBooks · doi:10.1007/978-3-319-00026-8_16 · Cited by 13
“The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it” (Cole et al. 1981, p. 881).”
Background
The Responsible Conduct of Bioethics Research
Master, Z. (2011) · Accountability in Research · doi:10.1080/08989621.2011.557302 · Cited by 13
“…or institutional affiliation (Ceci and Peters, 1982; Lee and Schunn, 2011; Wenneras and Wold, 1997), low inter-reviewer reliability (Bower, 1991; Cole et al., 1981), the incompetence of reviewers (Resnik et al., 2008) and ineffectiveness in detecting research fraud and plagiarism (LaFollette,…”
Background
Introducing sub-impact factor (SIF-) sequences and an aggregated SIF-indicator for journal ranking
Xu, F., Liu, W., & Rousseau, R. (2014) · Scientometrics · doi:10.1007/s11192-014-1401-9 · Cited by 13
“A new indicator, the sub-impact factor denoted as SIF, derived sub-impact factor sequences and an aggregated SIF-indicator are proposed.”
BackgroundMethod
Quality control of epidemiological lectures online: scientific evaluation of peer review.
Linkov, F. Y., Lovalekar, M. T., & LaPorte, R. E. (2007) · PubMed · Cited by 13
Towards a Systematic Understanding of How to Institutionally Design Scientific Advisory Committees: A Conceptual Framework and Introduction to a Special Journal Issue
Hoffman, S. J., Ottersen, T., Tejpar, A., et al. (2018) · Global Challenges · doi:10.1002/gch2.201800020 · Cited by 13
Identifying Unethical Practices in Journal Publishing
Serebnick, J. (1991) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 12
Perspectives on global warming
Yearley, S., Mercer, D., Pitman, A. J., Орескес, Н., & Conway, E. M. (2012) · Metascience · doi:10.1007/s11016-011-9639-9 · Cited by 13
“Cole et al. (1981) attribute much of the variability in peer review outcomes to chance.”
BackgroundInfluential
Can’t We Do Better? A cost-benefit analysis of proposal writing in a competitive funding environment
Schweiger, G. (2023) · PLoS ONE · doi:10.1371/journal.pone.0282320 · Cited by 12
“] getting a research grant depends to a significant extent on chance” [7].”
Background
External Tests of Peer Review Validity Via Impact Measures
Gallo, S. A., & Glisson, S. R. (2018) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2018.00022 · Cited by 17
“In addition, many studies have indicated low inter-rater reliability amongst panelists (Cole and Simon, 1981) and some studies indicate that review scores and rankings are much more dependent on the individual reviewer than on the proposal (Jayasinghe et al.”
The road from “vocation”: Weber and Veblen on the purposelessness of scholarship
Turner, S. (2019) · Journal of Classical Sociology · doi:10.1177/1468795x19851375 · Cited by 12
Lotteries make science fairer
Gildenhuys, P. (2020) · Journal of Responsible Innovation · doi:10.1080/23299460.2020.1812485 · Cited by 11
The Cultivation of Social Work Knowledge: Toward a More Robust System of Peer Review
Dunleavy, D. J. (2021) · Families in Society The Journal of Contemporary Social Services · doi:10.1177/10443894211012243 · Cited by 11
“The limitations of traditional peer review (i.e., oft closed, blinded, pre-publication review) have been known for decades (see generally Cole et al., 1981; D. Crane, 1967; Horrobin, 1974, 1990; Rennie, 1986; Smith, 1997, 2006; Wager & Jefferson, 2001).”
Background
Productivity of Medical Research in Switzerland
BOVIER, P. A., Guillain, H., & Perneger, T. (2001) · Journal of Investigative Medicine · doi:10.2310/6650.2001.34093 · Cited by 12
The fate of published articles, submitted again
Bartko, J. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011213 · Cited by 17
Anonyme Kritik: Peer Review auf dem Prüfstand der Wissenschaftsforschung
Fröhlich, G. (2002) · E-LIS Repository (University of Naples Federico II) · Cited by 12
On the failure to detect previously published research
Beaver, D. D. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011225 · Cited by 12
Primary care: opportunities and threats Deregulating primary care
Coulter, A., & Mays, N. (1997) · BMJ · doi:10.1136/bmj.314.7079.506 · Cited by 12
Referee agreement in context
Hargens, L. L. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065857 · Cited by 12
Offenheit und wissenschaftliche Werke : Open Access, Open Review, Open Metrics, Open Science & Open Knowledge
Herb, U. (2012) · Publications of the UdS (Saarland University) · doi:10.22028/d291-25636 · Cited by 12
“Fazit: Gutachterurteil, Ablehnung und Annahme eines Antrags waren zufällig (S. Cole, J. R. Cole & Simon, 1981; Fröhlich, 2003).”
Background
Are 'Better' Ideas More Likely to Succeed? An Empirical Analysis of Startup EvaluationPreprint
Scott, E. L., Shu, P., & Lubynsky, R. M. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2638367 · Cited by 12
“Unlike the vast literature on the effectiveness of the peer-review process in evaluating scientific ideas (Cole et al. , 1981; Chubin & Hackett, 1990; Lamont, 2009; Li & Agha, 2015), empirical evidence on evaluation of entrepreneurial ideas is scant and inconclusive.12 Using data on Canadian inventions, Astebro & Elhe…”
Order in the court: Consistency in criminal-court decisions.
Diamond, S. S. (2004) · American Psychological Association eBooks · doi:10.1037/10085-004 · Cited by 12
An evaluation of the quality of therapeutic studies in perinatal medicine
Tyson, J. E., Furzán, J. A., REISCH, J. S., & Mize, S. G. (1983) · Obstetrics and Gynecology · Cited by 12
Talent Identification at the limits of Peer Review: an analysis of the EMBO Postdoctoral Fellowships Selection ProcessPreprint
Klaus, B., & del Álamo, D. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/481655 · Cited by 12
“group of pre-selected, high-quality applications [1, 4, 6-8].”
Peer review for improving the quality of grant applications
Demicheli, V., & Pietrantonj, C. D. (2003) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000003 · Cited by 12
When Matthew met Matilda: the Argentinean gender gap in the allocation of science and technology public grants
Fiorentin, F. A., Pereira, M., Suárez, D., & Tcach, A. (2022) · Economics of Innovation and New Technology · doi:10.1080/10438599.2021.2017289 · Cited by 12
“…2018; Langfeldt et al. 2015), the lack of capabilities within the Agencies or Councils which manages and allocates funds (David 1994), random issues (Cole, Cole, and Simon 1981), and also gender-based discriminative practices (Magua et al. 2017; Uhlmann and Cohen 2007; van der Lee and Ellemers…”
Background
Cognitive Bias in the Peer Review Process
Street, C., & Ward, K. (2019) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/3371041.3371046 · Cited by 11
Two dogmas of peer-reviewism
Roumbanis, L. (2020) · Journal of Responsible Innovation · doi:10.1080/23299460.2020.1855806 · Cited by 11
Interrater Reliability of the Peer Review Process in Management Journals
Nicolai, A. T., Schmal, S., & Schuster, C. L. (2014) · Springer eBooks · doi:10.1007/978-3-319-09785-5_7 · Cited by 11
Peer review and national need
Chapman, I. D., & Farina, C. (1983) · Research Policy · doi:10.1016/0048-7333(83)90010-0 · Cited by 11
Do we really want more “reliable” reviewers?
KRAEMER, H. C. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065870 · Cited by 11
Disagreement among journal reviewers: No cause for undue alarm
Stricker, L. J. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065985 · Cited by 11
Improving Scientific Judgments in Law and Government: A Field Experiment of Patent Peer Review
Ho, D. E., & Ouellette, L. L. (2020) · Journal of Empirical Legal Studies · doi:10.1111/jels.12249 · Cited by 9
Las citas bibliográficas en la evaluación de la actividad científica: significado, consecuencias y un marco conceptual alternativo
Crisci, J. V., & Katinas, L. (2020) · Boletín de la Sociedad Argentina de Botánica · doi:10.31055/1851.2372.v55.n3.28723 · Cited by 18
The effectiveness of Japanese public funding to generate emerging topics in life science and medicine
Ohniwa, R. L., Takeyasu, K., & Hibino, A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0290077 · Cited by 10

Authors: Cole, S., Cole, J. R., & Simon, G. A.
Affiliations: Stony Brook University (US)
Columbia University (US)
ORCID: Cole, Stephen
Venue: Science
Publisher: American Association for the Advancement of Science (AAAS)
Published: 1981-11-20
Volume: 214
Issue: 4523
Pages: 881-886
ISSN: 0036-8075 (ISSN-L)
1095-9203
DOI: 10.1126/science.7302566
OpenAlex: W2091375917
Semantic Scholar: 20d3ad461253ead52398dcc0b194c5f5af1f6e08
Cited by: 623
References: 3
Citations by year: 2026: 15 · 2025: 16 · 2024: 16 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 27 · 2019: 24 · 2018: 18 · 2017: 17 · 2016: 21 · 2015: 23 · 2014: 22 · 2013: 15 · 2012: 18
FWCI: 279.85
Type: journal article
Language: en
Open access: closed
Locations: Science (publishedVersion)
PubMed (publishedVersion)
DigitalGeorgetown (Georgetown University Library) (submittedVersion)
Topics: Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords: peer review, research funding, National Science Foundation, reviewer disagreement, reviewer selection, scientific consensus, research evaluation
MeSH terms: Peer Review, Research Support as Topic, United States

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Chance and Consensus in Peer Review

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks