The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.
Scientific Utopia: I. Opening Scientific Communication
Abstract
Existing norms for scientific communication are rooted in anachronistic practices of bygone eras making them needlessly inefficient. We outline a path that moves away from the existing model of scientific communication to improve the efficiency in meeting the purpose of public science—knowledge accumulation. We call for six changes: (a) full embrace of digital communication; (b) open access to all published research; (c) disentangling publication from evaluation; (d) breaking the “one article, one journal” model with a grading system for evaluation and diversified dissemination outlets; (e) publishing peer review; and (f) allowing open, continuous peer review. We address conceptual and practical barriers to change and provide examples showing how the suggested practices are being used already. The critical barriers to change are not technical or financial; they are social. Although scientists guard the status quo, they also have the power to change it.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Two new Later Stone Age sites from the Final Pleistocene in the Falémé Valley, eastern Senegal
Ndiaye, M., Lespez, L., Tribolo, C., Rassé, M., Hadjas, I., Davidoux, S., Huysecom, É., & Douze, K. (2024) · Monash University Research Portal (Monash University) · doi:10.3929/ethz-b-000667478 · Cited by 23,868
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Measuring individual differences in implicit cognition: the implicit association test
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.6.1464 · Cited by 9,210
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
The American Psychological Association
Langfeld, H. S. (1919) · Science · doi:10.1126/science.49.1254.51 · Cited by 7,475
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447
Accounting for the effects of accountability.
Lerner, J. S., & Tetlock, P. E. (1999) · Psychological Bulletin · doi:10.1037/0033-2909.125.2.255 · Cited by 2,383 · cited 2×
“Further, transparency increases, not decreases, accountability and ethical behavior (Lerner & Tetlock, 1999).”
BackgroundPublication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Empirical assessment of effect of publication bias on meta-analyses
Sutton, A. J. (2000) · BMJ · doi:10.1136/bmj.320.7249.1574 · Cited by 1,398
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Peer-review practices of psychological journals: The fate of published articles, submitted again
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011183 · Cited by 1,079 · cited 5×
“…accessible freely, research can be performed to detect biases and heuristics that influence reviews and may educate the scientific community on how to overcome these influences (Peters & Ceci, 1982; Petty et al., 1999; Reich, Green, Brock, & Tetlock, 2007; Wilson, DePaulo, Mook, & Klaaren, 1993).”
BackgroundInfluentialThe Handbook of Research Synthesis
Hanka, R. (1994) · BMJ · doi:10.1136/bmj.309.6952.488a · Cited by 1,052
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 1×
“, nonsignificant) results are much more likely to end up in the file drawer (Greenwald, 1975).”
BackgroundSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 2×
“, significant) results a near necessity for publication (Fanelli, 2010, 2012; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
BackgroundErroneous analyses of interactions in neuroscience: A problem of significance
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2886 · Cited by 864 · cited 1×
“For example, Bishop ( 2012 ) identified significant design and analytic problems in a prominent neuroscience article (Temple et al, 2003 ) based on recent learning about methodology and analysis in this area (eg, Ioannidis, 2011 ; Kriegeskorte, Simmons, Bellgowan, & Baker, 2009 ; Nieuwenhuis, Forstmann, & Wagenmakers,…”
BackgroundWhen Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings
Adler, N. J., & Harzing, A.-W. (2009) · Academy of Management Learning & Education · doi:10.5465/amle.2009.37012181 · Cited by 853
Citation Advantage of Open Access Articles
Eysenbach, G. (2006) · PLoS Biology · doi:10.1371/journal.pbio.0040157 · Cited by 814
Temple, E., Deutsch, G. K., Poldrack, R. A., Miller, S. L., Tallal, P., Merzenich, M. M., & Gabrieli, J. D. E. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0030098100 · Cited by 796
Zuckerman, H., & Merton, R. K. (1971) · Minerva · doi:10.1007/bf01553188 · Cited by 789
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · cited 2×
“In particular, normative publishing practices make obtaining positive (i.e., significant) results a near necessity for publication (Fanelli, 2010, 2012; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995).”
BackgroundComparison of Registered and Published Primary Outcomes in Randomized Controlled Trials
Mathieu, S. (2009) · JAMA · doi:10.1001/jama.2009.1242 · Cited by 730
Do Open-Access Articles Have a Greater Research Impact?
Antelman, K. (2004) · College & Research Libraries · doi:10.5860/crl.65.5.372 · Cited by 678
Sunstein, C. R. (2007) · Medical Entomology and Zoology · Cited by 620
Self-Selected or Mandated, Open Access Increases Citation Impact for Higher Quality Research
Gargouri, Y., Hajjem, C., Larivière, V., Gingras, Y., Carr, L., Brody, T., & Harnad, S. (2010) · PLoS ONE · doi:10.1371/journal.pone.0013636 · Cited by 602
Challenges and opportunities of open data in ecology
Reichman, O. J., Jones, M. B., & Schildhauer, M. P. (2011) · Science · doi:10.1126/science.1197962 · Cited by 600
Implicit social cognition: from measures to mechanisms
Nosek, B. A., Hawkins, C. B., & Frazier, R. S. (2011) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.01.005 · Cited by 578 · cited 1×
“One of our core areas of expertise is implicit social cognition and implicit measurement (Nosek, Hawkins, & Frazier, 2011 ).”
BackgroundUnder what conditions does theory obstruct research progress?
Greenwald, A. G., Pratkanis, A. R., Leippe, M. R., & Baumgardner, M. H. (1986) · Psychological Review · doi:10.1037/0033-295x.93.2.216 · Cited by 574
The poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496 · cited 1×
“However, it is unusual to make a decision opposing the peer reviewers if they are unanimous, and lack of unanimity is common (Bornmann, Mutz, & Daniel, 2010; Cicchetti, 1991; Fiske & Fogg, 1990; Marsh & Ball, 1989; Peters & Ceci, 1982; Petty, Fleming, & Fabrigar, 1999).”
BackgroundThe Hierarchy of the Sciences?
Cole, S. (1983) · American Journal of Sociology · doi:10.1086/227835 · Cited by 478
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 463
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
The Philosophical Basis of Peer Review and the Suppression of Innovation
Horrobin, D. F. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100162024 · Cited by 436
How Much Better Are the Most-Prestigious Journals? The Statistics of Academic Publication
Starbuck, W. H. (2005) · Organization Science · doi:10.1287/orsc.1040.0107 · Cited by 434
Open access publishing, article downloads, and citations: randomised controlled trial
Davis, P. M., Lewenstein, B. V., Simon, D. H., Booth, J. G., & Connolly, M. J. L. (2008) · BMJ · doi:10.1136/bmj.a568 · Cited by 419
van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999) · BMJ · doi:10.1136/bmj.318.7175.23 · Cited by 415
Article 50 million: an estimate of the number of scholarly articles in existence
JINHA, A. E. (2010) · Learned Publishing · doi:10.1087/20100308 · Cited by 404
The (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · cited 2×
“Rosenthal (1979) named this the “file-drawer effect,” which can be very costly for knowledge accumulation.”
BackgroundUnpublished results hide the decline effect
Schooler, J. (2011) · Nature · doi:10.1038/470437a · Cited by 333
Excess Significance Bias in the Literature on Brain Volume Abnormalities
Ioannidis, J. P. A. (2011) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2011.28 · Cited by 299
The ups and downs of peer review
Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M., Mans, R., Mayhew, D., McGowan, S., Polter, A., et al. (2007) · Advances in Physiology Education · doi:10.1152/advan.00104.2006 · Cited by 294
Bornmann, L., Mutz, R., & Daniel, H.-D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0014331 · Cited by 290 · cited 2×
“Interrater consistency among reviewers is low (Bornmann et al., 2010; Marsh & Ball, 1989; Peters & Ceci, 1982; Petty et al., 1999).”
BackgroundPeer review for journals: Evidence on quality control, fairness, and innovation
Armstrong, J. S. (1997) · Science and Engineering Ethics · doi:10.1007/s11948-997-0017-3 · Cited by 262
Causal relationship between article citedness and journal impact
Seglen, P. O. (1994) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199401)45:1<1::aid-asi1>3.0.co;2-y · Cited by 239
Scholarly Consensus and Journal Rejection Rates
Hargens, L. L. (1988) · American Sociological Review · doi:10.2307/2095739 · Cited by 234
Phenomena of Retraction: Reasons for Retraction and Citations to the Publications
Budd, J. M., Sievert, M., & Schultz, T. R. (1998) · JAMA · doi:10.1001/jama.280.3.296 · Cited by 230
Trends in the Usage of ISI Bibliometric Data: Uses, Abuses, and Implications
Cameron, B. D. (2005) · portal: Libraries and the Academy · doi:10.1353/pla.2005.0003 · Cited by 199
Interrater agreement for journal manuscript reviews.
Whitehurst, G. J. (1984) · American Psychologist · doi:10.1037//0003-066x.39.1.22 · Cited by 189
Shining Light into Black Boxes
Morin, A., Urban, J., Adams, P. D., Foster, I., Sali, A., Baker, D., & Sliz, P. (2012) · Science · doi:10.1126/science.1218263 · Cited by 189
Opening up BMJ peer review: A beginning that should lead to complete transparency
Smith, R. (1999) · BMJ · doi:10.1136/bmj.318.7175.4 · Cited by 189
Scientific Journal Publishing: Yearly Volume and Open Access Availability.
Björk, B., Roos, A., & Lauri, M. (2009) · Työväentutkimus Vuosikirja · Cited by 189
Incongruence between test statistics and P values in medical papers
García-Berthou, E., & Alcaraz, C. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-13 · Cited by 181
Cognitive neuroscience 2.0: building a cumulative science of human brain function
Yarkoni, T., Poldrack, R. A., Van Essen, D. C., & Wager, T. D. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.08.004 · Cited by 174
Economic Implications of Alternative Scholarly Publishing Models Exploring the costs and benefits
Houghton, J., Rasmussen, B., Sheehan, P. J., Oppenheim, C., Morris, A., Creaser, C., Greenwood, H. E., Summers, M. A., & Gourlay, A. (2009) · Victoria University Research Repository (Victoria University) · Cited by 167
Bell, S., & Shank, J. D. (2004) · College & Research Libraries News · doi:10.5860/crln.65.7.7297 · Cited by 161
The invisible hand of peer review
Harnad, S. R. (1998) · Nature · doi:10.1038/nature28029 · Cited by 144
Fiske, D. W., & Fogg, L. F. (1990) · American Psychologist · doi:10.1037/0003-066x.45.5.591 · Cited by 142
Reinventing Discovery: The New Era of Networked Science
Nielsen, M. A. (2011) · Princeton University Press eBooks · doi:10.1515/9781400839452 · Cited by 139
The Peer Review Process Used to Evaluate Manuscripts Submitted to Academic Journals
Marsh, H. W., & Ball, S. (1989) · The Journal of Experimental Education · doi:10.1080/00220973.1989.10806503 · Cited by 120
How often are our numbers wrong?
Rosenthal, R. (1978) · American Psychologist · doi:10.1037/0003-066x.33.11.1005 · Cited by 115
Evidence on peer review—scientific quality control or smokescreen?
Goldbeck-Wood, S. (1999) · BMJ · doi:10.1136/bmj.318.7175.44 · Cited by 98
Empirical developments in retraction
Redman, B. K., Yarandi, H. N., & Merz, J. F. (2008) · Journal of Medical Ethics · doi:10.1136/jme.2007.023069 · Cited by 91
Psychology must learn a lesson from fraud case
Wicherts, J. M. (2011) · Nature · doi:10.1038/480007a · Cited by 86
Letting the daylight in: Reviewing the reviewers and other ways to maximize transparency in science
Wicherts, J. M., Kievit, R., Bakker, M., & Borsboom, D. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00020 · Cited by 70
The Review Process at PSPB: Correlates of Interreviewer Agreement and Manuscript Acceptance
Petty, R. E., Fleming, M. A., & Fabrigar, L. R. (1999) · Personality and Social Psychology Bulletin · doi:10.1177/0146167299025002005 · Cited by 61
Houghton, J., Rasmussen, B., Sheehan, P. J., Oppenheim, C., Morris, A., Creaser, C., Greenwood, H. E., Summers, M. A., & Gourlay, A. (2009) · Loughborough University Institutional Repository (Loughborough University) · Cited by 53
Scientists' Evaluations of Research: the Biasing Effects of the Importance of the Topic
Wilson, T. D., DePaulo, B. M., Mook, D. G., & Klaaren, K. J. (1993) · Psychological Science · doi:10.1111/j.1467-9280.1993.tb00572.x · Cited by 53
Trust Your Science? Open Your Data and Code
Stodden, V. (2011) · AMSTAT news: the membership magazine of the American Statistical Association · doi:10.7916/d8cj8q0p · Cited by 48
Harnad, S. R. (2017) · Routledge eBooks · doi:10.4324/9781315257778-6 · Cited by 46
Holden, G. W., Rosenberg, G., Barker, K., & Onghena, P. (2006) · Research on Social Work Practice · doi:10.1177/1049731506292570 · Cited by 46
Annual Report 1996 - Financial Statements
Uat, G. (2017) · Cited by 44
Houghton, J. (2009) · SSRN Electronic Journal · doi:10.2139/ssrn.1492578 · Cited by 40
Inconsistencies between reported test statistics and p‐values in two psychiatry journals
Berle, D., & Starcevic, V. (2007) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.225 · Cited by 38
Glynn, J. R., & Thomas, S. L. (2013) · The Lancet · doi:10.1016/s0140-6736(13)61244-1 · Cited by 33
Diffusion of treatment research: does open access matter?
Hardisty, D. J., & Haaga, D. A. F. (2008) · Journal of Clinical Psychology · doi:10.1002/jclp.20492 · Cited by 27
How Often are Our Statistics Wrong? a Statistics Class Exercise
Rossi, J. S. (1987) · Teaching of Psychology · doi:10.1207/s15328023top1402_8 · Cited by 15
Biases in research evaluation: Inflated assessment, oversight, or error-type weighting?
Reich, D. A., Green, M. C., Brock, T. C., & Tetlock, P. E. (2006) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2006.06.001 · Cited by 14
Cited by
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Self-citationA manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationThe Chicago face database: a free stimulus set of faces and norming data
Ma, D. S., Correll, J., & Wittenbrink, B. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0532-5 · Cited by 1,788
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,627
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,372
Metadata
- Authors
- Nosek, B. A., & Bar-Anan, Y.
- Venue
- Psychological Inquiry
- Publisher
- Informa UK Limited
- Published
- 2012-07
- Volume
- 23
- Issue
- 3
- Pages
- 217-243
- ISSN
- 1047-840X (ISSN-L)
- 1532-7965
- OpenAlex
- W2136528623
- Cited by
- 368
- References
- 80
- FWCI
- 41.53
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.