The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Measuring the effectiveness of scientific gatekeeping
Open accessAbstract
Significance: Peer review is an institution of enormous importance for the careers of scientists and the content of published science. The decisions of gatekeepers—editors and peer reviewers—legitimize scientific findings, distribute professional rewards, and influence future research. However, appropriate data to gauge the quality of gatekeeper decision-making in science has rarely been made publicly available. Our research tracks the popularity of rejected and accepted manuscripts at three elite medical journals. We found that editors and reviewers generally made good decisions regarding which manuscripts to promote and reject. However, many highly cited articles were surprisingly rejected. Our research suggests that evaluative strategies that increase the mean quality of published science may also increase the risk of rejecting unconventional or outstanding work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Emergence of Scaling in Random Networks
Barabási, A.-L., & Albert, R. (1999) · Science · doi:10.1126/science.286.5439.509 · Cited by 36,135
The Structure of Scientific Revolutions
Kuhn, T. S., & Schlegel, R. (1963) · Physics Today · doi:10.1063/1.3050879 · Cited by 35,538
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
THE STRUCTURE OF SCIENTIFIC REVOLUTIONS
MCGUIRE, J. E. (1963) · Philosophical Books · doi:10.1111/j.1468-0149.1963.tb00813.x · Cited by 8,424
Laboratory Life: The Construction of Scientific Facts
Latour, B. (1986) · DigitalGeorgetown (Georgetown University Library) · Cited by 7,491
Competing Technologies, Increasing Returns, and Lock-In by Historical Events
Arthur, W. B. (1989) · The Economic Journal · doi:10.2307/2234208 · Cited by 7,489
Levinthal, D. A., & March, J. G. (1993) · Strategic Management Journal · doi:10.1002/smj.4250141009 · Cited by 7,283
The Matthew Effect in Science: The reward and communication systems of science are considered.
Merton, R. K. (1968) · Science · doi:10.1126/science.159.3810.56 · Cited by 6,720
Gieryn, T. F. (1983) · American Sociological Review · doi:10.2307/2095325 · Cited by 4,838
(1968) · PubMed · Cited by 4,803
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Laboratory life: The social construction of scientific facts
Austin, J. (1982) · Social Science & Medicine · doi:10.1016/0277-9536(82)90228-3 · Cited by 2,883
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447
The frequency distribution of scientific productivity
Lotka, A. J. (1926) · Journal of the Washington Academy of Sciences · Cited by 2,246
Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006) · Science · doi:10.1126/science.1121066 · Cited by 2,231 · cited 1×
“Status signals and other social factors can profoundly affect diffusion processes, as well as people’s opinions and perceptions of quality (45).”
BackgroundGompers, P. A., & Lerner, J. Y. (1999) · Cited by 2,078
What do citation counts measure? A review of studies on citing behavior
Bornmann, L., & Daniel, H. (2008) · Journal of Documentation · doi:10.1108/00220410810844150 · Cited by 1,607 · cited 1×
“However, the vast majority of citations are either positive or neutral in nature (13).”
BackgroundThe Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · cited 1×
“Scientists have expressed concern that this publication bias selectively distorts the corpus of published of science and contributes to the publication of dubious results (37, 38).”
BackgroundThe frequency distribution of scientific productivity
G.F.S. (1926) · Journal of the Franklin Institute · doi:10.1016/s0016-0032(26)91166-6 · Cited by 1,344
Processing Fads and Fashions: An Organization-Set Analysis of Cultural Industry Systems
Hirsch, P. M. (1972) · American Journal of Sociology · doi:10.1086/225192 · Cited by 1,333
How Professors Think: Inside the Curious World of Academic Judgment
Lamont, M. (2009) · doi:10.4159/9780674054158 · Cited by 1,086
Peer-review practices of psychological journals: The fate of published articles, submitted again
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011183 · Cited by 1,079 · cited 1×
“Related to the Matthew Effect (1) and underscoring the importance of social status in science, evaluators have been found to judge equivalent work from high-status sources more favorably than work from lower-status contributors (11, 19, 20).”
BackgroundPublication prejudices: An experimental study of confirmatory bias in the peer review system
Mahoney, M. J. (1977) · Cognitive Therapy and Research · doi:10.1007/bf01173636 · Cited by 1,013 · cited 1×
“Although peer review is widely cited as central to academic evaluation (2, 3), numerous scholars have expressed concern about the effectiveness of peer review, particularly regarding the tendency to protect the scientific status quo and suppress innovative findings (4, 5).”
BackgroundLee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
Dale, S. B., & Krueger, A. B. (2002) · The Quarterly Journal of Economics · doi:10.1162/003355302320935089 · Cited by 995
Rescuing US biomedical research from its systemic flaws
Alberts, B., Kirschner, M. W., Tilghman, S., & Varmus, H. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1404402111 · Cited by 817 · cited 1×
“expressed concern that very high rejection rates tend to squeeze out innovative research (25).”
BackgroundSmith, P. K., & Trope, Y. (2006) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.90.4.578 · Cited by 754
Status Signals: A Sociological Study of Market Competition
Podolny, J. M. (2010) · Project Muse (Johns Hopkins University) · doi:10.1515/9781400837878 · Cited by 747
Adaptation as Information Restriction: The Hot Stove Effect
Denrell, J., & March, J. G. (2001) · Organization Science · doi:10.1287/orsc.12.5.523.10092 · Cited by 715
The Bias Against Creativity: Why People Desire but Reject Creative Ideas
Mueller, J. S., Melwani, S., & Goncalo, J. A. (2012) · Psychological Science · doi:10.1177/0956797611421018 · Cited by 658
Chance and Consensus in Peer Review
Cole, S., Cole, J. R., & Simon, G. A. (1981) · Science · doi:10.1126/science.7302566 · Cited by 623 · cited 1×
“(56) argued that roughly one-half of a competitive National Science Foundation granting process was a matter of luck.”
BackgroundHow professors think: inside the curious world of academic judgment
(2009) · Choice Reviews Online · doi:10.5860/choice.47-1492 · Cited by 601
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496 · cited 1×
“It is generally easier to distinguish scientific work of very low quality than it is to recognize finer gradations distinguishing the best contributions in advance of publication (17, 18).”
BackgroundScientific Careers: Universalism and Particularism
Long, J. S., & Fox, M. F. (1995) · Annual Review of Sociology · doi:10.1146/annurev.so.21.080195.000401 · Cited by 486
The Hierarchy of the Sciences?
Cole, S. (1983) · American Journal of Sociology · doi:10.1086/227835 · Cited by 478
Status and Status Strain in the Professions
Abbott, A. (1981) · American Journal of Sociology · doi:10.1086/227318 · Cited by 454
The Philosophical Basis of Peer Review and the Suppression of Innovation
Horrobin, D. F. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100162024 · Cited by 436
How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists
Gans, J. S., & Shepherd, G. B. (1994) · Journal of Economic Perspectives · doi:10.1257/jep.8.1.165 · Cited by 389
Threshold models of interpersonal effects in consumer demand
Granovetter, M., & Soong, R. (1986) · Journal of Economic Behavior & Organization · doi:10.1016/0167-2681(86)90023-5 · Cited by 333
Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine
Goodman, S. N., Berlin, J., Fletcher, S. W., & Fletcher, R. H. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-121-1-199407010-00003 · Cited by 324
Evolving Standards for Academic Publishing: A q‐r Theory
Ellison, G. (2002) · Journal of Political Economy · doi:10.1086/341871 · Cited by 252
Peer review. Crude and understudied, but indispensable
Kassirer, J. P. (1994) · JAMA · doi:10.1001/jama.272.2.96 · Cited by 235
Hargens, L. L. (2000) · American Sociological Review · doi:10.2307/2657516 · Cited by 211
Turning Lemons into Lemonade: Where Is the Value in Peer Reviews?
Starbuck, W. H. (2003) · Journal of Management Inquiry · doi:10.1177/1056492603258972 · Cited by 175
Flows of Research Manuscripts Among Scientific Journals Reveal Hidden Submission Patterns
Calcagno, V., Demoinet, É., Gollner, K., Guidi, L., Ruths, D. A., & de Mazancourt, C. (2012) · Science · doi:10.1126/science.1227833 · Cited by 169
The Loser's Curse: Decision Making and Market Efficiency in the National Football League Draft
Massey, C., & Thaler, R. H. (2013) · Management Science · doi:10.1287/mnsc.1120.1657 · Cited by 163
Lee, K. P., Boyd, E. A., Holroyd-Leduc, J. M., Bacchetti, P., & Bero, L. A. (2006) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.2006.tb00418.x · Cited by 130
Self-citationNew, B. (1997) · BMJ · doi:10.1136/bmj.314.7079.498 · Cited by 130
Top performers are not the most impressive when extreme performance indicates unreliability
Denrell, J. C., & Liu, C. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1116048109 · Cited by 119
Silver, N. (2012) · Cited by 115
Assessing precision in the manuscript review process - a little better than a dice roll.
Lindsey, D. (1988) · Scientometrics · doi:10.1007/bf02020243 · Cited by 97 · cited 2×
“It is generally easier to distinguish scientific work of very low quality than it is to recognize finer gradations distinguishing the best contributions in advance of publication (17, 18).”
BackgroundDiffusing through Disciplines: Insiders, Outsiders, and Socially Influenced Citation Behavior
Lynn, F. B. (2014) · Social Forces · doi:10.1093/sf/sou069 · Cited by 50
Does the need for agreement among reviewers inhibit the publication controversial findings?
Armstrong, J. S., & Hubbard, R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065699 · Cited by 40
The role of status in evaluating research: the case of data editing
Leahey, E. (2003) · Social Science Research · doi:10.1016/j.ssresearch.2003.09.004 · Cited by 29
Radioimmunoassay: A Probe for Fine Structure of Biologic Systems
Yalow, R. S. (1992) · Scandinavian Journal of Immunology · doi:10.1111/j.1365-3083.1992.tb02828.x · Cited by 13
(2014) · Choice Reviews Online · doi:10.5860/choice.52-0385 · Cited by 9
Cited by
The science of science: From the perspective of complex systems
Zeng, A., Shen, Z., Zhou, J., Wu, J., Fan, Y., Wang, Y., & Stanley, H. E. (2017) · Physics Reports · doi:10.1016/j.physrep.2017.10.001 · Cited by 410
A multi-disciplinary perspective on emergent and future innovations in peer review
Tennant, J. P., Dugan, J. M., Graziotin, D., Jacques, D. C., Waldner, F., Mietchen, D., Elkhatib, Y., B. Collister, L., Pikas, C. K., Crick, T., et al. (2017) · F1000Research · doi:10.12688/f1000research.12037.1 · Cited by 289
Checco, A., Bracciale, L., Loreti, P., Pinfield, S., & Bianchi, G. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00703-8 · Cited by 286
The effect of publishing peer review reports on referee behavior in five scholarly journals
Bravo, G., Grimaldo, F., López-Iñesta, E., Mehmani, B., & Squazzoni, F. (2019) · Nature Communications · doi:10.1038/s41467-018-08250-2 · Cited by 273
“Excellence R Us”: university research and the fetishisation of excellence
Moore, S., Neylon, C., Paul Eve, M., Paul O’Donnell, D., & Pattinson, D. (2017) · Palgrave Communications · doi:10.1057/palcomms.2016.105 · Cited by 264
Metadata
- Authors
- Siler, K., Lee, K., & Bero, L.
- Venue
- Proceedings of the National Academy of Sciences
- Published
- 2014-12-22
- Volume
- 112
- Issue
- 2
- Pages
- 360-365
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- OpenAlex
- W2028083052
- Cited by
- 333
- References
- 56
- FWCI
- 24.02
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.