CoChat
Sign Up
Cited by 471Meta-analysis and systematic reviewsAcademic integrity and plagiarismPsychology Research and Bibliometrics

Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention

Open access
Ioannidis, John P.A., Munafò, Marcus R., Fusar-Poli, Paolo, Nosek, Brian A., David, Sean P.·2014·Trends in Cognitive Sciences
DOI 10.1016/j.tics.2014.02.010PMC PMC4078993
Open this paper in CoChat — free account

Abstract

Recent systematic reviews and empirical evaluations of the cognitive sciences literature suggest that publication and other reporting biases are prevalent across diverse domains of cognitive science. In this review, we summarize the various forms of publication and reporting biases and other questionable research practices, and overview the available methods for probing into their existence. We discuss the available empirical evidence for the presence of such biases across the neuroimaging, animal, other preclinical, psychological, clinical trials, and genetics literature in the cognitive sciences. We also highlight emerging solutions (from study design to data analyses and reporting) to prevent bias and improve the fidelity in the field of cognitive science research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.tics.2014.02.010

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results.

    “Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 12­23 ).”

    From Estimating the reproducibility of psychological science (2015) · cites this paper as [18]

    The Definitions of Biases section identifies selective outcome reporting and selective analysis reporting as forms of bias. The Pre-publication review practices section calls for disclosure of experimental conditions, but does not specifically identify failure to specify the conditions necessary or sufficient to obtain results as a problematic practice. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The original studies may have inflated effect sizes due to publication, selection, reporting, or other biases.

    “An important qualification about this result is the possibility that the original studies have inflated effect sizes due to publication, selection, reporting, or other biases ( 9, 12­23 ).”

    From Estimating the reproducibility of psychological science (2015) · cites this paper as [18]

  • Concerns about reproducibility are widespread across disciplines.

    “Concerns about reproducibility are widespread across disciplines ( 9­21 ).”

    From Estimating the reproducibility of psychological science (2015) · cites this paper as [18]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 98 · Cited by 471

Research works printed but not linked: 65

  1. [2]Dwan K Systematic review of the empirical evidence of study publication bias and outcome reporting bias PLoS One 2008 3 e3081 18769481 PMC2518111
  2. [3]Vogel G Scientific misconduct. Psychologist accused of fraud on 'astonishing scale' Science 2011 334 579 22053018
  3. [4]Vogel G Scientific misconduct. Fraud charges cast doubt on claims of DNA damage from cell phone fields Science 2008 321 1144 1145 18755946
  4. [5]Vogel G Developmental biology. Fraud investigation clouds paper on early cell fate Science 2006 314 1367 1369 17138872
  5. [6]Saquib N Practices and impact of primary outcome adjustment in randomized controlled trials: meta-epidemiologic study BMJ 2013 347 f4313 23851720 PMC3709831
  6. [7]Turner EH Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database PLoS Med 2012 9 e1001189 22448149 PMC3308934
  7. [8]Turner EH Selective publication of antidepressant trials and its influence on apparent efficacy N Engl J Med 2008 358 252 260 18199864
  8. [9]Dwan K Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review PLoS One 2013 8 e66844 23861749 PMC3702538
  9. [11]Lau J The case of the misleading funnel plot BMJ 2006 333 597 600 16974018 PMC1570006
  10. [12]Peters JL Comparison of two methods to detect publication bias in metaanalysis JAMA 2006 295 676 680 16467236
  11. [14]Copas JB Malley PF A robust P-value for treatment effect in meta-analysis with publication bias Stat Med 2008 27 4267 4278 18384185
  12. [15]Copas J Jackson D A bound for publication bias based on the fraction of unpublished studies Biometrics 2004 60 146 153 15032784
  13. [17]Pfeiffer T Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias PLoS One 2011 6 e18362 21479240 PMC3066227
  14. [18]Gadbury GL Allison DB Inappropriate fiddling with statistical analyses to obtain a desirable p-value: tests to detect its presence in published literature PLoS One 2012 7 e46363 23056287 PMC3466248
  15. [19]Ioannidis JP Trikalinos TA An exploratory test for an excess of significant findings Clin Trials 2007 4 245 253 17715249
  16. [20]Ioannidis JPA Clarifications on the application and interpretation of the test for excess significance and its extensions J Math Psychol 2013 (in press)
  17. [21]Francis G The psychology of replication and replication in psychology Perspectives in Psychological Science 2012 7 585 594 26168115
  18. [25]Ioannidis JP Commentary: Adjusting for bias: a user's guide to performing plastic surgery on meta-analyses of observational studies Int J Epidemiol 2011 40 777 779 21233141
  19. [26]Thompson S A proposed method of bias adjustment for meta-analyses of published observational studies Int J Epidemiol 2011 40 765 777 21186183 PMC3147067
  20. [27]Jennings RG Van Horn JD Publication bias in neuroimaging research: implications for meta-analyses Neuroinformatics 2012 10 67 80 21643733 PMC4368431
  21. [29]Borgwardt S Why are psychiatric imaging methods clinically unreliable? Conclusions and practical guidelines for authors, editors and reviewers Behav Brain Funct 2012 8 46 22938624 PMC3478220
  22. [31]Sayo A Study factors influencing ventricular enlargement in schizophrenia: a 20 year follow-up meta-analysis Neuroimage 2012 59 154 167 21787868 PMC3319053
  23. [32]David SP Potential reporting bias in FMRI studies of the brain PLoS One 2013 8 e70104 23936149 PMC3723634
  24. [35]ter Riet G Publication bias in laboratory animal research: a survey on magnitude, drivers, consequences and potential solutions PLoS One 2012 7 e43404 22957028 PMC3434185
  25. [37]Ioannidis JP Extrapolating from animals to humans Sci Transl Med 2012 4 151ps115 22972841
  26. [40]Sena E How can we improve the pre-clinical development of drugs for stroke? Trends Neurosci 2007 30 433 439 17765332
  27. [44]Fanelli D Ioannidis JP US studies may overestimate effect sizes in softer research Proc Natl Acad Sci U S A 2013 110 15031 15036 23980165 PMC3773789
  28. [47]John LK Measuring the prevalence of questionable research practices with incentives for truth telling Psychol Sci 2012 23 524 532 22508865
  29. [51]Stamatakis E Undue industry influences that distort healthcare research, strategy, expenditure and practice: a review Eur J Clin Invest 2013 43 469 475 23521369
  30. [52]Etter JF The impact of pharmaceutical company funding on results of randomized trials of nicotine replacement therapy for smoking cessation: a metaanalysis Addiction 2007 102 815 822 17493109
  31. [53]Cuijpers P Comparison of psychotherapies for adult depression to pill placebo control groups: a meta-analysis Psychol Med 2013 1 11 23552610
  32. [54]Cuijpers P Comparison of psychotherapies for adult depression to pill placebo control groups: a meta-analysis Psychol Med 2013 44 685 695 23552610
  33. [56]Munafò MR Reliability and replicability of genetic association studies Addiction 2009 104 1439 1440 19686515
  34. [57]Ioannidis JP Replication validity of genetic association studies Nat Genet 2001 29 306 309 11600885
  35. [58]Munafò MR Flint J Dissecting the genetic architecture of human personality Trends Cogn Sci 2011 15 395 400 21831694
  36. [59]Hirschhorn JN Genomewide association studies--illuminating biologic pathways N Engl J Med 2009 360 1699 1701 19369661
  37. [60]Evangelou E Ioannidis JP Meta-analysis methods for genome-wide association studies and beyond Nat Rev Genet 2013 14 379 389 23657481
  38. [61]Panagiotou OA The power of meta-analysis in genome-wide association studies Annu Rev Genomics Hum Genet 2013 14 441 465 23724904 PMC4040957
  39. [62]Anderson MS Normative dissonance in science: results from a national survey of u.s. Scientists J Empir Res Hum Res Ethics 2007 2 3 14 19385804
  40. [63]Nosek BA Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability Perspectives on Psychological Science 2012 7 615 631 PMC10540222 26168121
  41. [66]Ioannidis JP The importance of potential studies that have not existed and registration of observational data sets JAMA 2012 308 575 576 22871867
  42. [68]Dal-Ré R Should prospective registration of observational research be only a matter of willingness? Sci Transl Med 2013 (in press) 24553383
  43. [70]Schmidt S Shall we really do it again? The powerful concept of replication is neglected in the social sciences 2009 Review of General Psychology 13 90 100
  44. [72]Makel MC Replications in psychology research: How often do they really occur? Perspectives on Psychological Science 2012 7 537 542 26168110
  45. [73]Evangelou E Perceived information gain from randomized trials correlates with publication in high-impact factor journals J Clin Epidemiol 2012 65 1274 1281 22959593
  46. [75]Chambers CD Registered reports: a new publishing initiative at Cortex Cortex 2013 49 609 610 23347556
  47. [78]Wadman M NIH mulls rules for validating key results Nature 2013 500 14 16 23903729
  48. [79]Open-Science Collaboration An open, large-scale collaborative effort to estimate the reproducibility of psychological science Perspectives on Psychological Science 2012 7 657 660 26168127
  49. [80]Open-Science-Collaboration Stodden V Leisch F Peng R The Reproducibility Project: A model of large-scale collaboration for empirical research on reproducibility Implementing Reproducible Computational Research (A Volume in the R Series) 2013 Taylor & Francis
  50. [81]Klein RA Ratliff KA Investigating variation in replicability: A“Many Labs” replication project. Registered design Social Psychology 2014 (in press)
  51. [82]Frank MC Saxe R Teaching replication Perspectives on Psychological Science 2012 7 600 604 26168118
  52. [83]Grahe JE Harnessing the undiscovered resource of student research projects Perspectives on Pyschological Science 2012 7 26168119
  53. [84]Poldrack RA Guidelines for reporting an fMRI study Neuroimage 2008 40 409 414 18191585 PMC2287206
  54. [85]Neuliep JW Crandell R Editorial bias against replication research Journal of Social Behavior and Personality 1990 5 85 90
  55. [86]Eich E Business not as usual Psychological Science 2014 (in press) 24285431
  56. [87]Carp J The secret lives of experiments: methods reporting in the fMRI literature Neuroimage 2012 63 289 300 22796459
  57. [88]Steen RG Retractions in the scientific literature: do authors deliberately commit research fraud? J Med Ethics 2010 37 113 117 21081306
  58. [89]Koole SL Lakens D Rewarding replications: A sure and simple way to improve psychological science Perspectives on Psychological Science 2012 7 608 614 26168120
  59. [92]Ioannidis JP Trikalinos TA Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials J Clin Epidemiol 2005 58 543 549 15878467
  60. [93]Kirkham JJ The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews BMJ 2010 340 c365 20156912
  61. [94]Hahn S Assessing the potential for bias in meta-analysis due to selective reporting of subgroup analyses within studies Stat Med 2000 19 3325 3336 11122498
  62. [95]Moreno SG Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications BMJ 2009 339 b2981 19666685 PMC2723217
  63. [96]Vedula SS Outcome reporting in industry-sponsored trials of gabapentin for off-label use N Engl J Med 2009 361 1963 1971 19907043
  64. [97]Ioannidis JP Why most discovered true associations are inflated Epidemiology 2008 19 640 648 18633328
  65. [98]Young NS Why current publication practices may distort science PLoS Med 2008 5 e201 18844432 PMC2561077

Authors
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P.
Affiliations
  • Stanford Medicine (US)
  • Center for Open Science (US)
Corresponding author
Ioannidis, John P.A.
ORCID
Venue
Trends in Cognitive Sciences
Publisher
Elsevier BV
Published
2014-05
Volume
18
Issue
5
Pages
235-241
ISSN
1364-6613 (ISSN-L)
DOI
10.1016/j.tics.2014.02.010
PMC
PMC4078993
OpenAlex
W2128246772
Cited by
471
References
98
Citations by year
2026: 16 · 2025: 25 · 2024: 27 · 2023: 29 · 2022: 29 · 2021: 52 · 2020: 46 · 2019: 46 · 2018: 49 · 2017: 44 · 2016: 56 · 2015: 45 · 2014: 7
FWCI
13.87
Type
journal article
Language
en
Open access
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations
  • Trends in Cognitive Sciences (publishedVersion)
  • PubMed (publishedVersion)
  • Explore Bristol Research (publishedVersion)
  • PubMed Central (acceptedVersion)
  • Research Portal (King's College London) (submittedVersion)
  • Bristol Research (University of Bristol) (submittedVersion)
Funders
  • British Heart Foundation
  • National Institute on Aging (R24AG048124)
  • National Institute on Drug Abuse (DA-017441, R01DA017441)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
publication bias, reporting bias, questionable research practices, cognitive science, bias detection, bias prevention, research integrity, neuroimaging research, animal studies, clinical trials, genetic research, research design
MeSH terms
Mental Disorders, Humans, Bias, Prevalence, Publication Bias, Cognitive Science

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review