Cited by 623Research, Science, and Academiascientometrics and bibliometrics researchPsychology Research and Bibliometrics

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Chance and Consensus in Peer Review

Cole, Stephen, Cole, Jonathan R., Simon, Gary A.·1981·Science

DOI 10.1126/science.7302566

Open this paper in CoChat — free account

Abstract

An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to a significant extent on chance. The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it. No evidence of systematic bias in the selection of NSF reviewers was found.

Integrity signals

Registry

Crossref · 10.1126/science.7302566

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Toward a Metric of Science: The Advent of Science Indicators.
Wilken, P. H., Elkana, Y., Lederberg, J., Merton, R. Κ., Thackray, A. W., & Zuckerman, H. (1979) · Social Forces · doi:10.2307/2577293 · Cited by 280
Peer Review and the Support of Science
Cole, S., Rubin, L., & Cole, J. R. (1977) · Scientific American · doi:10.1038/scientificamerican1077-34 · Cited by 140
Self-citation
Which researcher will get the grant?
Cole, J. R., & Cole, S. (1979) · Nature · doi:10.1038/279575a0 · Cited by 21
Self-citation

Flagged first, then by citations

Show all 3 references

Research works printed but not linked: 1

COLE, S, METRIC SCI ADVENT SC: 162 (1978).

Sorted by citations

Items 301–400 of 623

Correlations between submission and acceptance of papers in peer review journals
Ausloos, M., Nedić, O., & Dekanski, A. B. (2019) · Scientometrics · doi:10.1007/s11192-019-03026-x · Cited by 10
Face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation
Bieri, M., Roser, K., Heyard, R., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-047386 · Cited by 10
Five Models of Science, Illustrating How Selection Shapes Methods
Smaldino, P. E. (2022) · University of Pittsburgh Press eBooks · doi:10.2307/j.ctv31djr2f.5 · Cited by 9
Structural constraints and the publications dilemma: A review and some proposals
Dean, D. G. (1989) · The American Sociologist · doi:10.1007/bf02691857 · Cited by 10
The reliability of peer review for manuscript and grant submissions: “It's like déjà vu all over again!”
Cicchetti, D. V. (1993) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0003079x · Cited by 10
Happiness of economists
Feld, L. P., Necker, S., & Frey, B. S. (2014) · Applied Economics · doi:10.1080/00036846.2014.985374 · Cited by 10
Grant agencies versus the search for truth
Gordon, R. (1993) · Accountability in Research · doi:10.1080/08989629308573824 · Cited by 10
Scientific publications from departments of clinical physiology and nuclear medicine in Denmark. A bibliometric analysis of ‘impact’ in the years 1989–1994
Hansen, H. B., Brinch, K., & Henriksen, J. H. (1996) · Clinical Physiology · doi:10.1111/j.1475-097x.1996.tb01016.x · Cited by 10
What should be done improve reviewing?
Crandall, R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065766 · Cited by 10
Reforming peer review: From recycling to reflexivity
Chubin, D. E. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011274 · Cited by 11
Does the perceived quality of interdisciplinary research vary between fields?
Thelwall, M. A., Kousha, K., Stuart, E., Makita, M., Abdoli, M., Wilson, P., & Levitt, J. M. (2023) · Journal of Documentation · doi:10.1108/jd-01-2023-0012 · Cited by 10
A new approach to grant review assessments: score, then rank
Gallo, S. A., Pearce, M., Lee, C. J., & Erosheva, E. A. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00131-7 · Cited by 12
Recognition and Rewards in Academia – Recent Trends in Assessment
Kramer, B. M. R., & Bosman, J. (2024) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-80262-685-820241004 · Cited by 11
Bias in Open Peer-Review: Evidence from the English Superior Courts
Vidal, J. B. I., & Leaver, C. (2015) · The Journal of Law Economics and Organization · doi:10.1093/jleo/ewv004 · Cited by 8
Importance of factors in the review of grant proposals.
Norman, K. L. (1986) · Journal of Applied Psychology · doi:10.1037//0021-9010.71.1.156 · Cited by 10
The journal article review process as a game of chance
Glenn, N. D. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011365 · Cited by 9
Quality in research: an empirical study
Miller, R. L. (1994) · Technovation · doi:10.1016/0166-4972(94)90017-5 · Cited by 9
Collaborative Document Evaluation: An Alternative Approach To Classic Peer Review
Beel, J., & Gipp, B. (2008) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1070316 · Cited by 9
Rejecting published work: Similar fate for fiction
Ross, C. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011626 · Cited by 9
New governance arrangements in science policyPreprint
Lengwiler, M., & Simon, D. (2005) · RePEc: Research Papers in Economics · Cited by 9
Peer review in the physical sciences: An editor's view
Honig, W. M. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011432 · Cited by 9
Does Peer Review Penalize Scientific Risk Taking? Evidence from NIH Grant Renewals
Azoulay, P., & Greenblatt, W. H. (2025) · National Bureau of Economic Research · doi:10.3386/w33495 · Cited by 11
The Case for Reviewing as Collaboration and Response
Hunter, S. M. (1995) · Rhetoric Review · Cited by 8
Random grant allocation from the researchers’ perspective: Introducing the distinction into legitimate and illegitimate problems in Bourdieu’s field theory
Barlösius, E., & Philipps, A. (2022) · Social Science Information · doi:10.1177/05390184221076627 · Cited by 8
“By describing the choice of reviewers by the funding organizations as being based on luck (see also Bornmann and Daniel, 2009; Cole et al., 1981), the interviewees also underscored the difference that a science-guided choice of reviewers makes.”
Background
The determinants of election to the Presidency of the American Economic Association: Evidence from a cohort of distinguished 1950’s economists
Diamond, A. M., & Toth, R. J. (2007) · Scientometrics · doi:10.1007/s11192-006-1747-8 · Cited by 8
“Some empirical studies have defended the Merton thesis (e.g., ZUCKERMAN & MERTON, 1971), while others have implicitly criticized it (e.g., CRANE 1965; KOREN, 1986, 1987; PETERS & CECI, 1982; COLE et al., 1981; MULKAY, 1980).”
Background
Philosophy in the trenches: from naturalized to experimental philosophy (of science)
Stotz, K. C. (2009) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2009.03.011 · Cited by 8
“The answer is that a substantial percentage of the variance is explained by facts about the reviewers and almost none is explained by facts about the papers; Cole et al., 1981; Blackburn & Hakel, 2006).”
Background
Influence of Reviewer Interaction Network on Long-Term Citations: A Case Study of the Scientific Peer-Review System of the Journal of High Energy Physics
Sikdar, S., Matteo Marsili, A., Ganguly, N., & Mukherjee, A. (2017) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl.2017.7991572 · Cited by 8
“ed that blinding signicantly improved the quality of the reviews. Further, reviewers seem to fail to reach consensus while determining whether the contribution of a scientic article is novel enough [6]. This has led to serious debate in the research community regarding the credibility of the peer-review process. With…”
Background
The Social Structure of Consecration in Cultural Fields: The Influence of Status and Social Distance in Audience–Candidate Evaluative Processes
Aadland, E., Cattani, G., & Ferriani, S. (2018) · Emerald Group Publishing Limited eBooks · doi:10.1108/s0733-558x20180000055006 · Cited by 8
Quantitative analysis of a visible tip of the peer review iceberg: Book reviews in chemistry
Schubert, A. P., Zsindely, S., Telcs, A., & Braun, T. (1984) · Scientometrics · doi:10.1007/bf02025830 · Cited by 8
Productivity versus Age
Marchetti, C. (2004) · IIASA PURE (International Institute of Applied Systems Analysis) · Cited by 8
Is unreliability in peer review harmful?
Roediger, H. L., III (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065936 · Cited by 8
Bias, incompetence, or bad management?
Ziman, J. M. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011742 · Cited by 8
Peer review and theCurrent Anthropologyexperience
Belshaw, C. S. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011237 · Cited by 8
Peer review again.
Yankauer, A. (1982) · American Journal of Public Health · doi:10.2105/ajph.72.3.239 · Cited by 8
When we practice to deceive: The ethics of a metascientific inquiry
Mindick, B. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011511 · Cited by 8
Talent Management in Higher Education
Thunnissen, M., & Boselie, P. (2024) · Emerald Group Publishing Limited eBooks · doi:10.1108/9781802626858 · Cited by 13
ACADEMIC TALENT SELECTION IN GRANT REVIEW PANELS
van Arensbergen, P., van der Weijden, I., & van den Besselaar, P. (2014) · Radboud Repository (Radboud University) · Cited by 7
Why is the reliability of peer review so low?
Laming, D. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065882 · Cited by 7
What do academic metrics do to political scientists? Theorizing their roots, locating their effects
Baele, S. J., & Bettiza, G. (2017) · Politics · doi:10.1177/0263395716685727 · Cited by 7
From External Evaluation, to School Self-evaluation, to Peer Review
Godfrey, D. (2020) · Accountability and educational improvement · doi:10.1007/978-3-030-48130-8_1 · Cited by 11
Not-Last Words: Worn-Out Dichotomies in the Sociology of Science (Reply)
Gieryn, T. F. (1982) · Social Studies of Science · doi:10.1177/030631282012002009 · Cited by 7
In praise of randomness
Schönemann, P. H. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065973 · Cited by 7
The quandary of manuscript reviewing
Whitehurst, G. J. “. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011699 · Cited by 8
Scholarly Communication as a Loosely Coupled System: Reassessing Prospects for Structural Reform
Schwartz, C. A. (1994) · College & Research Libraries · doi:10.5860/crl_55_02_101 · Cited by 7
Explaining an unsurprising demonstration: High rejection rates and scarcity of space
Beyer, J. M. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011250 · Cited by 7
Efficiency and flexibility in research funding. A comparative study of funding instruments and review criteria.
Langfeldt, L., & Scordato, L. (2016) · BIBSYS Brage (BIBSYS (Norway)) · doi:10.13140/rg.2.1.4568.2165 · Cited by 7
When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer reviewPreprint
Erosheva, E. A., Martinková, P., & Lee, C. J. (2021) · doi:10.17605/osf.io/knph8 · Cited by 7
Getting funded in a highly fluctuating environment: Shifting from excellence to luck and timing
Kindsiko, E., Rõigas, K., & Niinemets, Ü. (2022) · PLoS ONE · doi:10.1371/journal.pone.0277337 · Cited by 6
“The small differences in the score in the end depend on the degree of subjectivity of the reviews [14, 35, 36].”
Background
Postdigital Peer Review: A Vulnerable Space Between Freedom and Responsibility
Veletsianos, G., Arabadzhy, G., Athilat, V., Bartucz, J., Dogra, S., Fredrickson, N. R., Goeke, M., Jeon, S., Urena, M., Jandrić, P., et al. (2025) · Postdigital Science and Education · doi:10.1007/s42438-025-00604-6 · Cited by 6
Similar‐to‐me effects in the grant application process: Applicants, panellists, and the likelihood of obtaining funds
Banal‐Estañol, A., Liu, Q., Macho‐Stadler, I., & Pérez‐Castrillo, D. (2023) · R and D Management · doi:10.1111/radm.12601 · Cited by 6
Blinding Models for Scientific Peer-Review of Biomedical Research Proposals: A Systematic Review
Qussini, S., MacDonald, R. S., Shahbal, S., & Dierickx, K. (2023) · Journal of Empirical Research on Human Research Ethics · doi:10.1177/15562646231191424 · Cited by 5
“Notably, consideration of the author’s track record is typically excluded from manuscript review but may form an important part of research proposal review (Cole et al., 1981).”
BackgroundInfluential
Research Program Peer Review: Purposes, Principles, Practices, Protocols
Kostoff, R. N. (2004) · Defense Technical Information Center (DTIC) · Cited by 6
The Calculus of Committee Composition
Libby, E., & Glass, L. (2010) · PLoS ONE · doi:10.1371/journal.pone.0012642 · Cited by 6
“Previous work has approached this problem through analyzing evaluation costs or accuracy [14,15,32–34].”
BackgroundInfluential
Peer Review: Strengthening the Weak Link of Academic Radiology
Shriki, J. E., & Bhargava, P. (2015) · Current Problems in Diagnostic Radiology · doi:10.1067/j.cpradiol.2015.02.007 · Cited by 6
“Peer review has long been recognized as a process fraught with flaws, biases, and poor reproducibility.(1-3) Criticisms of the peer review process are many.”
Revista de Contabilidad
Martínez, J. A., Martínez, F. G., & Diazaraque, J. M. M. (2011) · Cited by 6
Anomalies in the Peer-review System
Sikdar, S., Matteo Marsili, A., Ganguly, N., & Mukherjee, A. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983675 · Cited by 6
“That the reviewers often fail to reach consensus ([5]) and that rejected papers are often cited more in the long run ([2]), have already been pointed out.”
Background
Who Is (Likely) Peer-Reviewing Your Papers? A Partial Insight into the World’s Top Reviewers
Pomponi, F., D’Amico, B., & Rye, T. (2019) · Publications · doi:10.3390/publications7010015 · Cited by 6
“Cole and Simon [3] looked at research grants and found that funding depends to a significant extent on chance.”
Background
Rank-normalized journal impact factor as a predictive tool
Racki, G. (2009) · Archivum Immunologiae et Therapiae Experimentalis · doi:10.1007/s00005-009-0006-0 · Cited by 6
“However, this is still an unimpressive number compared with potential pitfalls in the course of more intuitive expert judgments (see for example (Cole et al. 1981; Garfield 1983; Rothwell and Martyn 2000)).”
Background
From Laboratory to Scientific Literature
Dirk, L. (1996) · Science Communication · doi:10.1177/1075547096018001001 · Cited by 6
Refereeing and Peer Review. Part 4. Research on the Peer Review of Grant Proposals and Suggestions for Improvement
Garfield, E. (1987) · Cited by 6
A Large Community but Few Peers: A Study of the Scientific Community in India
Haribabu, E. (1991) · Sociological Bulletin · doi:10.1177/0038022919910105 · Cited by 6
“REFERENCES Aurora, G S and N Kumar 1985 'Sociology of Science', in Survey of Research in Sociology and Social Anthropology, 1969-79, ICSSR, Vol II New Delhi Satavahan Publications Bhabha, Homi J 1966 'Science and Problems of Development', Science, 12 166-72 Cole, Stephen, Rubin Leonard and Jonathan R. Cole 1977 'Peer…”
Background
Improving research on and policies for peer-review practices
Perloff, R. M., & Perloff, R. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011584 · Cited by 6
Faculty Malfeasance: Understanding Academic Deviance
Heeren, J., & Shichor, D. (1993) · Sociological Inquiry · doi:10.1111/j.1475-682x.1993.tb00201.x · Cited by 6
Decline of U.S. cancer mortality rates: Expert estimates of past underreporting
Gori, G. B., & Cornelius, J. (1986) · Regulatory Toxicology and Pharmacology · doi:10.1016/0273-2300(86)90017-6 · Cited by 6
Deception in the study of the peer-review process
Fleiss, J. L. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011341 · Cited by 6
Canadian medical research strategy for the eighties
Forsdyke, D. R. (1983) · Medical Hypotheses · doi:10.1016/0306-9877(83)90057-9 · Cited by 6
Correction to ‘Open science and modified funding lotteries can impede the natural selection of bad science’
Smaldino, P. E., Turner, M. A., & Kallens, P. C. (2019) · Royal Society Open Science · doi:10.1098/rsos.191249 · Cited by 5
Trends in Peer ReviewPreprint
Reinhart, M., & Schendzielorz, C. (2021) · doi:10.31235/osf.io/nzsp5 · Cited by 6
Beyond Feasibility Filters: How Expertise Heterogeneity Enables Innovation RecognitionPreprint
Lane, J. N., Szajnfarber, Z., Crusan, J. C., & Menietti, M. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4117951 · Cited by 6
“…on the scientific peer review process points to the downsides of “noise” in evaluation scores and poor interrater reliability among experts (Cole and Simon 1981, Pier et al. 2018, Rothwell and Martyn 2000), we suggest that inserting deliberate variation in domain distance can encourage…”
Background
A System-Level Analysis of Conference Peer Review
Zhang, Y., Yu, F.-Y., Schoenebeck, G. R., & Kempe, D. (2022) · ACM Conference on Economics and Computation (EC) · doi:10.1145/3490486.3538235 · Cited by 7
Do Peer Reviews Predict Impact? Evidence from the American Sociological Review, 1978 to 1982
Teplitskiy, M., & Bakanic, V. (2016) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023116640278 · Cited by 7
“The great majority of these studies focus on reliability: to what extent reviewers agree with one another (Bornmann, Mutz, and Daniel 2010; Cicchetti 1991; Cole, Cole, and Simon 1981).”
Background
Smart Policies for Societies in Transition
Benner, M., Marklund, G., & Serger, S. S. (2022) · Edward Elgar Publishing eBooks · doi:10.4337/9781788970815 · Cited by 5
Ex ante evaluation of interdisciplinary research projects: A literature review
Oviedo‐García, M. Á. (2016) · Social Science Information · doi:10.1177/0539018416658147 · Cited by 5
“…question in the case of grant proposals in the Cole et al. (1981) study, as they found that, in the assessment of 150 grant proposals submitted to the National Science Foundation, the decision to fund depended to a great extent ‘upon which reviewers happen to be selected’ (Cole et al., 1981: 881).”
Background
The Peer Review Question
Atkinson, R. C. (1981) · Science · doi:10.1126/science.214.4527.1292 · Cited by 5
Rejecting published work: It couldn't happen in physics! (or could it?)
Moravcsik, M. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011535 · Cited by 5
New knowledge versus consensus – a critical note on their relationship based on the debate concerning the use of barrel-vaults in Macedonian tombs
Chilidis, K. (2007) · European Journal of Archaeology · doi:10.1177/1461957108101242 · Cited by 5
“Having said that, it is important to point out that referees very seldom agree on the intrinsic value of a manuscript (Cole et al. 1981:882; Giles 2006).”
Background
Peer review: An unflattering picture
Adams, K. M. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065687 · Cited by 5
Multiskill Research
Porter, A. L., & Rossini, F. A. (1986) · Science Communication · doi:10.1177/107554708600700301 · Cited by 5
A Study of the Board Certification Examination in Child and Adolescent Psychiatry
Tanguay, P. E., McDermott, J. F., & Philips, I. (1990) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1097/00004583-199009000-00024 · Cited by 5
Reviewer reliability: Confusing random error with systematic error or bias
Presser, S. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011602 · Cited by 5
Authorship and manuscript reviewing: The risk of bias
DeBakey, L. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011328 · Cited by 5
Journal availability and the quality of published research
Fletcher, J. M. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065808 · Cited by 5
Editorial: A critical review of peer review: The need to scrutinize the"gatekeepers" of research in exercise physiology
Robergs, R. A. (2003) · QUT ePrints (Queensland University of Technology) · Cited by 5
Influence of reviewer interaction network on long-term citations: a case study of the scientific peer-review system of the journal of high energy physicsPreprint
Sikdar, S., Matteo Marsili, A., Ganguly, N., & Mukherjee, A. (2017) · arXiv (Cornell University) · doi:10.5555/3200334.3200354 · Cited by 7
Academic Evaluation in Higher Education
Hamann, J., & Beljean, S. (2020) · Springer eBooks · doi:10.1007/978-94-017-8905-9_295 · Cited by 5
Empirical analysis of factors influencing delay in article acceptance in accounting journals
Argilés-Bosch, J. M., Ravenda, D., & García-Blandón, J. (2023) · Learned Publishing · doi:10.1002/leap.1564 · Cited by 4
“The importance of authors and institutions plays an important role in the editor ’ s decision (Cole et al., 1981; Peters & Ceci, 1982).”
BackgroundResult
Validating Peer Review in Criminal Justice Evaluation Research: Evidence from CrimeSolutions.gov
WORRALL, J. L. (2015) · Journal of Criminal Justice Education · doi:10.1080/10511253.2015.1052827 · Cited by 4
“In a classic study, Cole et al. (1981) explored the role of chance in obtaining National Science Foundation (NSF) grants.”
Background
Changing the Peer Review or Changing the Peers — Recent Development in Assessment of Large Research Collaborations
Hansson, F., & Mønsted, M. (2012) · Higher Education Policy · doi:10.1057/hep.2012.17 · Cited by 5
Peer Review Under Review — a Statistical Study on Proposal Ranking at ESO. Part I: the Pre-meeting Phase
Patat, F. (2018) · Publications of the Astronomical Society of the Pacific · doi:10.1088/1538-3873/aac463 · Cited by 4
Peer Review for Academic Research
Kuo, C. (2022) · Archives of Otorhinolaryngology-Head & Neck Surgery · doi:10.24983/scitemed.aohns.2022.00155 · Cited by 5
Journal Ratings as Predictors of Article Quality in Arts, Humanities, and Social Sciences: An Analysis Based on the Italian Research Evaluation Exercise
Bonaccorsi, A. J., Ferrara, A., & Malgarini, M. (2018) · Springer eBooks · doi:10.1007/978-3-319-68554-0_11 · Cited by 4
On the effectiveness of the scientific peer-review system: a case study of the Journal of High Energy Physics
Sikdar, S., Tehria, P., Matteo Marsili, A., Ganguly, N., & Mukherjee, A. (2018) · International Journal on Digital Libraries · doi:10.1007/s00799-018-0247-9 · Cited by 4
“Further, reviewers seem to fail to reach consensus while determining whether the contribution of a scientific article is novel enough [4].”
Background
The reliability of peer review in anthrozoology
Herzog, H. A., Podberscek, A. L., & Docherty, A. (2005) · Anthrozoös · doi:10.2752/089279305785594180 · Cited by 4
Reliability is neither to be expected nor desired in peer review
Luce, R. D. (1993) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00030776 · Cited by 4
ADVANCES IN INFORMATION SCIENCE Bias in Peer Review
Lee, C. J., Sugimoto, C. R., Guo, Z., & Cronin, B. (2013) · Cited by 4
Review bias: Positive or negative, good or bad?
Geen, R. G. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011353 · Cited by 4
Canadian medical research strategy for the eighties
Forsdyke, D. R. (1983) · Medical Hypotheses · doi:10.1016/0306-9877(83)90058-0 · Cited by 4
Collective allocation of science funding: from funding agencies to scientific agencyPreprint
Bollen, J., Crandall, D., Junk, D., et al. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.1067 · Cited by 4
Diabetologia 1965–1985 — Portrait of a journal
Berger, M. (1985) · Diabetologia · doi:10.1007/bf00281980 · Cited by 4
Research Policy: Peering into the bowels of the MRC II: review systems
Smith, R. W. (1988) · BMJ · doi:10.1136/bmj.296.6621.556 · Cited by 4

Authors: Cole, S., Cole, J. R., & Simon, G. A.
Affiliations: Stony Brook University (US)
Columbia University (US)
ORCID: Cole, Stephen
Venue: Science
Publisher: American Association for the Advancement of Science (AAAS)
Published: 1981-11-20
Volume: 214
Issue: 4523
Pages: 881-886
ISSN: 0036-8075 (ISSN-L)
1095-9203
DOI: 10.1126/science.7302566
OpenAlex: W2091375917
Semantic Scholar: 20d3ad461253ead52398dcc0b194c5f5af1f6e08
Cited by: 623
References: 3
Citations by year: 2026: 15 · 2025: 16 · 2024: 16 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 27 · 2019: 24 · 2018: 18 · 2017: 17 · 2016: 21 · 2015: 23 · 2014: 22 · 2013: 15 · 2012: 18
FWCI: 279.85
Type: journal article
Language: en
Open access: closed
Locations: Science (publishedVersion)
PubMed (publishedVersion)
DigitalGeorgetown (Georgetown University Library) (submittedVersion)
Topics: Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords: peer review, research funding, National Science Foundation, reviewer disagreement, reviewer selection, scientific consensus, research evaluation
MeSH terms: Peer Review, Research Support as Topic, United States

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Chance and Consensus in Peer Review

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks