Publication Bias: The Problem That Won't Go Away
Abstract
Conclusions about the efficacy and safety of medical interventions are based on data presented in the scientific literature. The validity of these conclusions is threatened if publication bias results from investigators or editors making decisions about publishing study results on the basis of the direction or strength of the study findings. This paper reports meta-analyses performed using data from four prospective investigations in which a total of 997 initiated studies were followed to learn of study results, publication status, and reasons for nonpublication. The analysis indicates that there is a positive association between "significant" study results and publication (OR = 2.88; 95% confidence interval [CI] 2.13 to 3.90). When the analysis was restricted to controlled trials (n = 280), an even stronger relationship between "significant" results and publication was observed (OR = 6.15; 95% CI 2.24 to 16.92), with randomized trials (n = 200) apparently no less susceptible to publication bias than controlled trials in general (OR = 8.72; 95% CI 1.91 to 39.81). In every case, failure to publish was investigator-based, and not due to editorial decisions. The results of clinical trials should not be suppressed in this way. Development of registration systems for randomized trials is essential if this problem is to be minimized in future.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Publication bias is likely when drawing generalized conclusions from the published results because null findings may not be reported and statistically significant findings tend to be published.
“Furthermore, with null findings possibly not being reported and statistically significant findings tending to be published, publication bias is likely when drawing generalized conclusions from these published results [131].”
From Genetics of child aggression, a systematic review (2024) · cites this paper as [131]
Cites
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · see citing works
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · see citing works
Self-citationSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · see citing works
Dickersin, K., Yuan, M.-Q., & Meinert, C. (1992) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.1992.03480030052036 · Cited by 900 · see citing works
Self-citationPublication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814 · see citing works
Publication bias: the case for an international registry of clinical trials.
Simes, R. J. (1986) · Journal of Clinical Oncology · doi:10.1200/jco.1986.4.10.1529 · Cited by 747 · see citing works
Publication bias and clinical trials
Dickersin, K., Chan, S., Chalmersx, T. C., Sacks, H. S., & Smith, H. (1987) · Controlled Clinical Trials · doi:10.1016/0197-2456(87)90155-3 · Cited by 715 · see citing works
Self-citationUnderreporting Research Is Scientific Misconduct
Chalmers, I. (1990) · JAMA · doi:10.1001/jama.263.10.1405 · Cited by 634 · see citing works
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.2307/2282137 · Cited by 524 · see citing works
Getting to grips with Archie Cochrane's agenda.
Chalmers, I., Dickersin, K., & Chalmers, T. C. (1992) · BMJ · doi:10.1136/bmj.305.6857.786 · Cited by 376 · see citing works
Self-citationA cohort study of summary reports of controlled trials.
Chalmers, I., Adams, M., Dickersin, K., Hetherington, J., Tarnow-Mordi, W., Meinert, C., Tonascia, S., & Chalmers, T. (1990) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.1990.03440100117017 · Cited by 200 · see citing works
Self-citationBIAS AGAINST THE NULL HYPOTHESIS: THE REPRODUCTIVE HAZARDS OF COCAINE
Koren, G., Shear, H. H., Graham, K., & Einarson, T. R. (1989) · The Lancet · doi:10.1016/s0140-6736(89)92044-8 · Cited by 173 · see citing works
Bias Against Negative Studies in Newspaper Reports of Medical Research
Koren, G. (1991) · JAMA · doi:10.1001/jama.1991.03470130104037 · Cited by 144 · see citing works
Dickersin, K. (1988) · Controlled Clinical Trials · doi:10.1016/0197-2456(88)90010-4 · Cited by 58 · see citing works
Self-citationToward prospective registration of clinical trials
Meinert, C. L. (1988) · Controlled Clinical Trials · doi:10.1016/0197-2456(88)90002-5 · Cited by 57 · see citing works
A proposal for registering clinical triais
Piantadosi, S., & Byar, D. P. (1988) · Controlled Clinical Trials · doi:10.1016/0197-2456(88)90011-6 · Cited by 19 · see citing works
New Journal Will Publish Without Paper
Palca, J. W. (1991) · Science · doi:10.1126/science.1896854 · Cited by 12 · see citing works
Making clinical trialists register
(1991) · The Lancet · doi:10.1016/0140-6736(91)90371-u · Cited by 12 · see citing works
Regulatory process effects clinical trial registration in Spain
Dickersin, K., & López, F. J. G. (1992) · Controlled Clinical Trials · doi:10.1016/0197-2456(92)90207-g · Cited by 11 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465 · see citing works
The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052 · see citing works
Dissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039 · see citing works
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751 · see citing works
Ioannidis, J. P. A. (1998) · JAMA · doi:10.1001/jama.279.4.281 · Cited by 632 · see citing works
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603 · see citing works
Formulating Questions and Locating Primary Studies for Inclusion in Systematic Reviews
Counsell, C. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-5-199709010-00008 · Cited by 505 · see citing works
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453 · see citing works
Trial Publication after Registration in ClinicalTrials.Gov: A Cross-Sectional Analysis
Ross, J. S., Mulvey, G. K., Hines, E. M., Nissen, S. E., & Krumholz, H. M. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000144 · Cited by 401 · see citing works
Publication Bias in Editorial Decision Making
Olson, C. M. (2002) · JAMA · doi:10.1001/jama.287.21.2825 · Cited by 395 · see citing works
Factors that limit the quality, number and progress of randomised controlled trials
Prescott, Counsell, Gillespie, Grant, Russell, Kiauka, Colthart, Ross, Shepherd, & Russell (1999) · Health Technology Assessment · doi:10.3310/hta3200 · Cited by 390 · see citing works
Cures That Harm: Unanticipated Outcomes of Crime Prevention Programs
McCord, J. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716202250781 · Cited by 338 · see citing works
Publication Bias (The "File-Drawer Problem") in Scientific Inference
Scargle, J. D. (1999) · arXiv · doi:10.48550/arxiv.physics/9909033 · Cited by 313 · see citing works
Dechartres, A., Boutron, I., Trinquart, L., Charles, P., & Ravaud, P. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-1-201107050-00006 · Cited by 307 · see citing works
Lilford, R. J., Chilton, P. J., Hemming, K., Girling, A. J., Taylor, C. A., & Barach, P. (2010) · BMJ · doi:10.1136/bmj.c4413 · Cited by 146 · see citing works
Bafeta, A., Dechartres, A., Trinquart, L., Yavchitz, A., Boutron, I., & Ravaud, P. (2012) · BMJ · doi:10.1136/bmj.e813 · Cited by 129 · see citing works
Hahn, S., Williamson, P. R., & Hutton, J. L. (2002) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2002.00314.x · Cited by 97 · see citing works
Alcohol and intimate partner violence: do we have enough information to act?
Gil-González, D., Vives-Cases, C., Álvarez-Dardet, C., & Latour-Pérez, J. (2006) · European Journal of Public Health · doi:10.1093/eurpub/ckl016 · Cited by 98 · see citing works
Empirical evaluation of which trial characteristics are associated with treatment effect estimates
Dechartres, A., Trinquart, L., Faber, T., & Ravaud, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.005 · Cited by 85 · see citing works
Kebede, M. M., Le Cornet, C., & Fortner, R. T. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1589 · Cited by 50 · see citing works
Genetics of child aggression, a systematic review
Koyama, E., Kant, T., Takata, A., Kennedy, J. L., & Zai, C. C. (2024) · Translational Psychiatry · doi:10.1038/s41398-024-02870-7 · Cited by 33 · see citing works
Examining the predictive validity of NIH peer review scores
Lindner, M. D., & Nakamura, R. K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0126938 · Cited by 24 · see citing works
- Authors
- DICKERSIN, K., & MIN, Y.
- Affiliations
- University of Maryland, Baltimore (US)
- Johns Hopkins University (US)
- Venue
- Annals of the New York Academy of Sciences
- Publisher
- Wiley
- Published
- 1993-12
- Volume
- 703
- Issue
- 1
- Pages
- 135-148
- ISSN
- 0077-8923 (ISSN-L)
- 1749-6632
- OpenAlex
- W2075571844
- Cited by
- 427
- References
- 19
- Citations by year
- 2026: 10 · 2025: 17 · 2024: 27 · 2023: 23 · 2022: 10 · 2021: 15 · 2020: 14 · 2019: 11 · 2018: 11 · 2017: 19 · 2016: 10 · 2015: 17 · 2014: 15 · 2013: 13 · 2012: 19
- FWCI
- 9.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Annals of the New York Academy of Sciences (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmaceutical industry and healthcare
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- publication bias, clinical trial registration, significance testing, research integrity
- MeSH terms
- Clinical Trials as Topic, Female, Humans, Male, Professional Staff Committees, Prospective Studies, Publishing, Registries, Research Design, Research Support as Topic, Reproducibility of Results, Bias, Confidence Intervals, Logistic Models, Odds Ratio
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review