Publication bias—a problem in interpreting medical data
Abstract
Publication bias, the phenomenon in which studies with positive results are more likely to be published than studies with negative results, is a serious problem in the interpretation of scientific research. Various hypothetical models have been studied which clarify the potential for bias and highlight characteristics which make a study especially susceptible to bias. Empirical investigations have supported the hypothesis that bias exists and have provided a quantitative assessment of the magnitude of the problem. The use of meta‐analysis as a research tool has focused attention on the issue, since naive methodologies in this area are especially susceptible to bias. In this paper we review the available research, discuss alternative suggestions for conducting unbiased meta‐analysis and suggest some scientific policy measures which could improve the quality of published data in the long term.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Truncated distributions have been used to study hidden tests (publication bias) in medical research.
“Truncated distributions have been used to study hidden tests (i.e., publication bias) in medical research.”
From … and the Cross-Section of Expected Returns (2016) · cites this paper as Begg and Berlin (1988)
Journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results.
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
From The Extent and Consequences of P-Hacking in Science (2015) · cites this paper as [4]
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · see citing works
Peer-review practices of psychological journals: The fate of published articles, submitted again
Peters, D. P., & Ceci, S. J. (1982) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00011183 · Cited by 1,079 · see citing works
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 3× · see citing works
“This graph of sample size versus effectsize should be symmetrical in the absence of bias, but will tend to be skewed in its presence (Light and Pillemer (1984); see Figs 1 and 2 and further discussion in Section 5.”
BackgroundInfluentialSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · see citing works
Bias in treatment assignment in controlled clinical trials
Chalmers, T. C., Celano, P., Sacks, H. S., & Smith, H. (1983) · New England Journal of Medicine · doi:10.1056/nejm198312013092204 · Cited by 864 · see citing works
Source of funding and outcome of clinical trials
Davidson, R. A. (1986) · Journal of General Internal Medicine · doi:10.1007/bf02602327 · Cited by 397 · see citing works
Hedges, L. V. (1984) · Journal of Educational Statistics · doi:10.3102/10769986009001061 · Cited by 241 · see citing works
Toward prospective registration of clinical trials
Meinert, C. L. (1988) · Controlled Clinical Trials · doi:10.1016/0197-2456(88)90002-5 · Cited by 57 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955 · see citing works
Self-citationDuval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · see citing works
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120 · see citing works
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709 · see citing works
Why Don't Physicians Follow Clinical Practice Guidelines?: A Framework for Improvement
Cabana, M. D., Rand, C. S., Powe, N. R., Wu, A. W., Wilson, M. H., Abboud, P.-A. C., & Rubin, H. R. (1999) · JAMA · doi:10.1001/jama.282.15.1458 · Cited by 6,677 · see citing works
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779 · see citing works
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · see citing works
Self-citationDeeks, J. J., Macaskill, P., & Irwig, L. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.01.016 · Cited by 2,910 · see citing works
Design and analysis of pilot studies: recommendations for good practice
Lancaster, G. A., Dodd, S., & Williamson, P. R. (2004) · Journal of Evaluation in Clinical Practice · doi:10.1111/j..2002.384.doc.x · Cited by 2,572 · see citing works
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267 · see citing works
… and the Cross-Section of Expected Returns
Harvey, C. R., Liu, Y., & Zhu, H. (2016) · Review of Financial Studies · doi:10.1093/rfs/hhv059 · Cited by 2,119 · see citing works
GRADE guidelines: 5. Rating the quality of evidence—publication bias
Guyatt, G. H., Oxman, A. D., Montori, V., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Djulbegovic, B., Atkins, D., Falck-Ytter, Y., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.011 · Cited by 1,819 · see citing works
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · see citing works
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · see citing works
“…that current scientific practices create strong incentives to publish statistically significant (i.e., “ positive ” ) results, and there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [4 – 10].”
BackgroundThe Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · see citing works
Publication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291 · see citing works
A comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286 · see citing works
Rosenberg, M. S. (2005) · Evolution · doi:10.1111/j.0014-3820.2005.tb01004.x · Cited by 1,285 · see citing works
Cumulative Meta-Analysis of Therapeutic Trials for Myocardial Infarction
Lau, J., Antman, E. M., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992) · New England Journal of Medicine · doi:10.1056/nejm199207233270406 · Cited by 1,140 · see citing works
Dissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039 · see citing works
Stanley, T. D., & Doucouliagos, H. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1095 · Cited by 916 · see citing works
Food cue reactivity and craving predict eating and weight gain: A meta-analytic review
Boswell, R. G., & Kober, H. (2016) · Obesity Reviews · doi:10.1111/obr.12354 · Cited by 703 · see citing works
The Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634 · see citing works
Publication Bias in Research Synthesis: Sensitivity Analysis Using A Priori Weight Functions.
Vevea, J. L., & Woods, C. M. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.428 · Cited by 461 · see citing works
Kliem, S., Kröger, C., & Kosfelder, J. (2010) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0021015 · Cited by 474 · see citing works
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453 · see citing works
Investigating and Dealing with Publication and Other Biases
Sterne, J. A. C., Egger, M., & Smith, G. D. (2001) · Systematic Reviews in Health Care · doi:10.1002/9780470693926.ch11 · Cited by 449 · see citing works
Gurevitch, J., Curtis, P. S., & Jones, M. H. (2001) · Advances in Ecological Research · doi:10.1016/s0065-2504(01)32013-5 · Cited by 421 · see citing works
Publication and related biases.
Song, Eastwood, Gilbody, Duley, & Sutton (2000) · Health Technology Assessment · doi:10.3310/hta4100 · Cited by 427 · see citing works
Publication Bias: The Problem That Won't Go Away
DICKERSIN, K., & MIN, Y. (1993) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1993.tb26343.x · Cited by 427 · see citing works
Moreno, S. G., Sutton, A. J., Ades, A., Stanley, T. D., Abrams, K. R., Peters, J. L., & Cooper, N. J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-2 · Cited by 373 · see citing works
Greenland, S. (2001) · Biostatistics · doi:10.1093/biostatistics/2.4.463 · Cited by 359 · see citing works
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359 · see citing works
Assessing risk of bias due to missing results in a synthesis
Page, M. J., Higgins, J. P., & Sterne, J. A. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch13 · Cited by 402 · see citing works
Updated tests for small-study effects in meta-analyses
Harbord, R. M., Harris, R. J., & Sterne, J. A. C. (2009) · The Stata Journal: Promoting communications on statistics and Stata · doi:10.1177/1536867x0900900202 · Cited by 307 · see citing works
Relation of the alliance with outcomes in youth psychotherapy: a meta-analysis
McLeod, B. D. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.02.001 · Cited by 301 · see citing works
Modelling publication bias in meta-analysis: a review
Sutton, A. J., Song, F., Gilbody, S. M., & Abrams, K. R. (2000) · Statistical Methods in Medical Research · doi:10.1177/096228020000900503 · Cited by 288 · see citing works
Page, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035.pub2 · Cited by 281 · see citing works
Panagioti, M., Gooding, P. A., & Tarrier, N. (2012) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2012.02.009 · Cited by 280 · see citing works
Publication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266 · see citing works
Gooding, P., & Tarrier, N. (2009) · Behaviour Research and Therapy · doi:10.1016/j.brat.2009.04.002 · Cited by 267 · see citing works
The Relationship Between Deliberate Practice and Performance in Sports: A Meta-Analysis
Macnamara, B. N., Moreau, D., & Hambrick, D. Z. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616635591 · Cited by 248 · see citing works
Limitations of PET-PEESE and Other Meta-Analysis Methods
Stanley, T. D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693062 · Cited by 229 · see citing works
Van Norman, G. A. (2016) · JACC: Basic to Translational Science · doi:10.1016/j.jacbts.2016.06.003 · Cited by 223 · see citing works
Publication bias as a threat to the validity of meta-analytic results
Rothstein, H. R. (2008) · Journal of Experimental Criminology · doi:10.1007/s11292-007-9046-9 · Cited by 216 · see citing works
Outcome selection bias in meta-analysis
Williamson, P. R., Gamble, C., Altman, D. G., & Hutton, J. L. (2005) · Statistical Methods in Medical Research · doi:10.1191/0962280205sm415oa · Cited by 188 · see citing works
Song, F., Parekh-Bhurke, S., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, A. J., Hing, C. B., & Harvey, I. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-79 · Cited by 175 · see citing works
Hedges, L. V. (1992) · Journal of Educational Statistics · doi:10.3102/10769986017004279 · Cited by 186 · see citing works
The impact of Covid-19 on student achievement: Evidence from a recent meta-analysis
Di Pietro, G. (2023) · Educational Research Review · doi:10.1016/j.edurev.2023.100530 · Cited by 173 · see citing works
“The appropriateness of the RE model is confirmed as I2 has a value of 100%.14 This suggests that all the variability in the effect-size estimates is due to heterogeneity as opposed to sampling error. Additionally, we also look at τ2 (between-study variance),15 which denotes the variability in the underlying true effec…”
InfluentialPublication Bias in Meta-Analysis: Confidence Intervals for Rosenthal’s Fail-Safe Number
Fragkos, K. C., Tsagris, M., & Frangos, C. C. (2014) · International Scholarly Research Notices · doi:10.1155/2014/825383 · Cited by 159 · see citing works
The expectation effect and cardiac pacing for refractory vasovagal syncope
Sud, S., Massel, D., Klein, G. J., Leong-Sit, P., Yee, R., Skanes, A. C., Gula, L. J., & Krahn, A. D. (2007) · The American Journal of Medicine · doi:10.1016/j.amjmed.2006.05.046 · Cited by 151 · see citing works
Finding the power to reduce publication bias
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2017) · Statistics in Medicine · doi:10.1002/sim.7228 · Cited by 131 · see citing works
Worsening file-drawer problem in the abstracts of natural, medical and social science databases
Pautasso, M. (2010) · Scientometrics · doi:10.1007/s11192-010-0233-5 · Cited by 136 · see citing works
Detecting and adjusting for small‐study effects in meta‐analysis
Rücker, G., Carpenter, J. R., & Schwarzer, G. (2011) · Biometrical Journal · doi:10.1002/bimj.201000151 · Cited by 132 · see citing works
Fedewa, M. V., Hathaway, E. D., Ward-Ritacco, C. L., Williams, T. D., & Dobbs, W. C. (2018) · Sports Medicine · doi:10.1007/s40279-018-0897-1 · Cited by 135 · see citing works
Using Dissertations to Examine Potential Bias in Child and Adolescent Clinical Trials.
McLeod, B. D., & Weisz, J. R. (2004) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.72.2.235 · Cited by 125 · see citing works
JOURNAL QUALITY, EFFECT SIZE, AND PUBLICATION BIAS IN META-ANALYSIS
Murtaugh, P. A. (2002) · Ecology · doi:10.1890/0012-9658(2002)083[1162:jqesap]2.0.co;2 · Cited by 126 · see citing works
Effects of sewerage on diarrhoea and enteric infections: a systematic review and meta-analysis
Norman, G., Pedley, S., & Takkouche, B. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70123-7 · Cited by 115 · see citing works
Association of Physical Activity Level and Stroke Outcomes in Men and Women: A Meta-Analysis
Diep, L., Kwagyan, J., Kurantsin-Mills, J., Weir, R., & Jayam-Trouth, A. (2010) · Journal of Women's Health · doi:10.1089/jwh.2009.1708 · Cited by 117 · see citing works
Hedger, N., Gray, K. L. H., Garner, M., & Adams, W. J. (2016) · Psychological Bulletin · doi:10.1037/bul0000054 · Cited by 115 · see citing works
Hahn, S., Williamson, P. R., & Hutton, J. L. (2002) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2002.00314.x · Cited by 97 · see citing works
Villar, J., Mackey, M. E., Carroli, G., & Donner, A. (2001) · Statistics in Medicine · doi:10.1002/sim.1096 · Cited by 96 · see citing works
Chiang, C.-H., Jaroenlapnopparat, A., Colak, S. C., Yu, C.-C., Xanthavanij, N., Wang, T.-H., See, X. Y., Lo, S.-W., Ko, A., Chang, Y.-C., et al. (2025) · Gastroenterology · doi:10.1053/j.gastro.2025.06.003 · Cited by 105 · see citing works
Bishop, D. V. M., & Thompson, P. A. (2016) · PeerJ · doi:10.7717/peerj.1715 · Cited by 90 · see citing works
Sala, G., Aksayli, N. D., Tatlidil, K. S., Gondo, Y., & Gobet, F. (2019) · Intelligence · doi:10.1016/j.intell.2019.101386 · Cited by 63 · see citing works
Mueller, K. F., Meerpohl, J. J., Briel, M., Antes, G., von Elm, E., Lang, B., Motschall, E., Schwarzer, G., & Bassler, D. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.015 · Cited by 48 · see citing works
Blood-Based Oxidation Markers in Medicated and Unmedicated Schizophrenia Patients: A Meta-Analysis
Goh, X. X., Tang, P. Y., & Tee, S. F. (2022) · Asian Journal of Psychiatry · doi:10.1016/j.ajp.2021.102932 · Cited by 24 · see citing works
Moderator effects of mobile users’ pedagogical role on science learning: a meta-analysis.
Shi, L., & Kopcha, T. J. (2022) · British Journal of Educational Technology · doi:10.1111/bjet.13210 · Cited by 17 · see citing works
Does Retrieval Practice Protect Memory Against Stress? A Meta-Analysis
Mihaylova, M., Zocca, M., Kliegel, M., & Rothen, N. (2025) · Swiss Psychology Open · doi:10.5334/spo.101 · Cited by 1 · see citing works
- Authors
- Begg, C. B., & Berlin, J. A.
- Affiliations
- Dana-Farber Cancer Institute (US)
- Venue
- Journal of the Royal Statistical Society. Series A (Statistics in Society)
- Publisher
- Oxford University Press (OUP)
- Published
- 1988
- Volume
- 151
- Issue
- 3
- Pages
- 419
- ISSN
- 0964-1998 (ISSN-L)
- OpenAlex
- W2028299022
- Cited by
- 814
- References
- 9
- Citations by year
- 2026: 14 · 2025: 44 · 2024: 37 · 2023: 27 · 2022: 36 · 2021: 26 · 2020: 18 · 2019: 25 · 2018: 22 · 2017: 24 · 2016: 30 · 2015: 35 · 2014: 32 · 2013: 34 · 2012: 30
- FWCI
- 21.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the Royal Statistical Society Series A (Statistics in Society) (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- publication bias, negative results, meta-analysis, meta-analysis bias, research quality, science policy, publication practices
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
