Quantifying publication bias in meta-analysis
Open accessAbstract
Summary: Publication bias is a serious problem in systematic reviews and meta-analyses, which can affect the validity and generalization of conclusions. Currently, approaches to dealing with publication bias can be distinguished into two classes: selection models and funnel-plot-based methods. Selection models use weight functions to adjust the overall effect size estimate and are usually employed as sensitivity analyses to assess the potential impact of publication bias. Funnel-plot-based methods include visual examination of a funnel plot, regression and rank tests, and the nonparametric trim and fill method. Although these approaches have been widely used in applications, measures for quantifying publication bias are seldom studied in the literature. Such measures can be used as a characteristic of a meta-analysis; also, they permit comparisons of publication biases between different meta-analyses. Egger's regression intercept may be considered as a candidate measure, but it lacks an intuitive interpretation. This article introduces a new measure, the skewness of the standardized deviates, to quantify publication bias. This measure describes the asymmetry of the collected studies’ distribution. In addition, a new test for publication bias is derived based on the skewness. Large sample properties of the new measure are studied, and its performance is illustrated using simulations and three case studies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139 · see citing works
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763 · see citing works
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955 · see citing works
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · see citing works
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709 · see citing works
A basic introduction to fixed-effect and random-effects models for meta-analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.12 · Cited by 6,793 · see citing works
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779 · see citing works
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955 · see citing works
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282 · see citing works
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267 · cited 3× · see citing works
“…statistical tests have been proposed for publication bias in the funnel plot, such as Begg’s rank test (Begg and Mazumdar, 1994) and Egger’s regression test (Egger et al., 1997) and its extensions (e.g., Macaskill et al., 2001; Rothstein et al., 2005; Harbord et al., 2006; Peters et al., 2006).”
MethodResultInvestigating and dealing with publication and other biases in meta-analysis
Sterne, J. A. C., Egger, M., & Smith, G. D. (2001) · BMJ · doi:10.1136/bmj.323.7304.101 · Cited by 2,225 · see citing works
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053 · see citing works
Empirical assessment of effect of publication bias on meta-analyses
Sutton, A. J. (2000) · BMJ · doi:10.1136/bmj.320.7249.1574 · Cited by 1,398 · see citing works
A comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286 · see citing works
Bias in treatment assignment in controlled clinical trials
Chalmers, T. C., Celano, P., Sacks, H. S., & Smith, H. (1983) · New England Journal of Medicine · doi:10.1056/nejm198312013092204 · Cited by 864 · see citing works
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818 · see citing works
Adjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809 · see citing works
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007) · Statistics in Medicine · doi:10.1002/sim.2889 · Cited by 716 · see citing works
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359 · see citing works
Modelling publication bias in meta-analysis: a review
Sutton, A. J., Song, F., Gilbody, S. M., & Abrams, K. R. (2000) · Statistical Methods in Medical Research · doi:10.1177/096228020000900503 · Cited by 288 · see citing works
Citation bias favoring statistically significant studies was present in medical research
Jannot, A.-S., Agoritsas, T., Gayet-Ageron, A., & Perneger, T. V. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.09.015 · Cited by 193 · see citing works
Flagged first, then by citations
Research works printed but not linked: 21
- Altman DG 2002 Poor-quality medical research: what can journals do? JAMA 287 2765 2767 12038906
- Antille A Kersting G Zucchini W 1982 Testing symmetry Journal of the American Statistical Association 77 639 646
- Begg CB Berlin JA 1988 Publication bias: a problem in interpreting medical data Journal of the Royal Statistical Society. Series A (Statistics in Society) 151 419 463
- DerSimonian R Laird N 1986 Meta-analysis in clinical trials Controlled Clinical Trials 7 177 188 3802833
- Egger M Davey Smith G Schneider M Minder C 1997 Bias in meta-analysis detected by a simple, graphical test BMJ 315 629 634 9310563 PMC2127453
- Gøtzsche PC 1987 Reference bias in reports of drug trials BMJ 295 654 656 3117277 PMC1257776
- Higgins JPT 2008 Commentary: Heterogeneity in meta-analysis should be expected and appropriately quantified International Journal of Epidemiology 37 1158 1160 18832388
- Hróbjartsson A Gøtzsche PC 2010 Placebo interventions for all clinical conditions Cochrane Database of Systematic Reviews 1 Art. No.: CD003974 PMC7156905 20091554
- Kicinski M Springate DA Kontopantelis E 2015 Publication bias in meta-analyses from the Cochrane Database of Systematic Reviews Statistics in Medicine 34 2781 2793 25988604
- Light RJ Pillemer DB 1984 Summing Up: The Science of Reviewing Research Harvard University Press Cambridge, MA
- Liu CJ Latham NK 2009 Progressive resistance strength training for improving physical function in older adults Cochrane Database of Systematic Reviews 3 Art. No.: CD002759 PMC4324332 19588334
- MacGillivray HL 1986 Skewness and asymmetry: measures and orderings The Annals of Statistics 14 994 1011
- McWilliams TP 1990 A distribution-free test for symmetry based on a runs statistic Journal of the American Statistical Association 85 1130 1133
- Normand SLT 1999 Meta-analysis: formulating, evaluating, combining, and reporting Statistics in Medicine 18 321 359 10070677
- Peters JL Sutton AJ Jones DR Abrams KR Rushton L 2006 Comparison of two methods to detect publication bias in meta-analysis JAMA 295 676 680 16467236
- Rothstein HR Sutton AJ Borenstein M 2005 Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments John Wiley & Sons Chichester, UK
- Rücker G Schwarzer G Carpenter J 2008 Arcsine test for publication bias in meta-analyses with binary outcomes Statistics in Medicine 27 746 763 17592831
- Silliman NP 1997a Hierarchical selection models with applications in meta-analysis Journal of the American Statistical Association 92 926 936
- Stead LF Perera R Bullen C Mant D Hartmann-Boyce J Cahill K Lancaster T 2012 Nicotine replacement therapy for smoking cessation Cochrane Database of Systematic Reviews 11 Art. No.: CD000146 23152200
- Sutton AJ Higgins JPT 2008 Recent developments in meta-analysis Statistics in Medicine 27 625 650 17590884
- Whitehead A Whitehead J 1991 A general parametric approach to the meta-analysis of randomized clinical trials Statistics in Medicine 10 1665 1677 1792461
Cited by
Sorted by citations
Shi, L., & Lin, L. (2019) · Medicine · doi:10.1097/md.0000000000015987 · Cited by 818 · see citing works
Self-citationAlnefeesi, Y., Chen-Li, D., Krane, E., Jawad, M. Y., Rodrigues, N. B., Ceban, F., Di Vincenzo, J. D., Meshkat, S., Ho, R. C. M., Gill, H., et al. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.04.037 · Cited by 186 · see citing works
A meta-analysis on heart rate variability biofeedback and depressive symptoms
Pizzoli, S. F. M., Marzorati, C., Gatti, D., Monzani, D., Mazzocco, K., & Pravettoni, G. (2021) · Scientific Reports · doi:10.1038/s41598-021-86149-7 · Cited by 170 · see citing works
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165 · see citing works
Martins, D., Rademacher, L., Gabay, A. S., Taylor, R., Richey, J. A., Smith, D. V., Goerlich, K. S., Nawijn, L., Cremers, H. R., Wilson, R., et al. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.12.034 · Cited by 119 · see citing works
Effects of digital games on student motivation in mathematics: A meta‐analysis in K‐12
Fadda, D., Pellegrini, M., Vivanet, G., & Zandonella Callegher, C. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12618 · Cited by 135 · see citing works
Rodríguez-Nieto, G., Seer, C., Sidlauskaite, J., Vleugels, L., Van Roy, A., Hardwick, R., & Swinnen, S. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119665 · Cited by 89 · see citing works
Chiang, C.-H., Jaroenlapnopparat, A., Colak, S. C., Yu, C.-C., Xanthavanij, N., Wang, T.-H., See, X. Y., Lo, S.-W., Ko, A., Chang, Y.-C., et al. (2025) · Gastroenterology · doi:10.1053/j.gastro.2025.06.003 · Cited by 105 · see citing works
Park, J., Ntelis, S., Yunasan, E., Downton, K. D., Yip, T. C.-F., Munir, K. M., & Haq, N. (2023) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/clinem/dgad471 · Cited by 69 · see citing works
Daraj, L. R., AlGhareeb, M., Almutawa, Y. M., Trabelsi, K., & Jahrami, H. (2023) · Healthcare · doi:10.3390/healthcare11142066 · Cited by 64 · see citing works
Kahoot! Game‐based digital learning platform: A comprehensive meta‐analysis
Özdemir, O. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13084 · Cited by 41 · see citing works
Permana, H., Yanto, T. A., & Hariyanto, T. I. (2022) · Diabetes & Metabolic Syndrome: Clinical Research & Reviews · doi:10.1016/j.dsx.2022.102640 · Cited by 38 · see citing works
Application of network meta-analysis in the field of physical activity and health promotion
Su, X., McDonough, D. J., Chu, H., Quan, M., & Gao, Z. (2020) · Journal of Sport and Health Science · doi:10.1016/j.jshs.2020.07.011 · Cited by 38 · see citing works
Self-citationSinopoli, A., Sciurti, A., Isonne, C., Santoro, M. M., & Baccolini, V. (2024) · Nutrients · doi:10.3390/nu16091345 · Cited by 40 · see citing works
“For this reason, further studies are necessary to confirm our results, especially well-conducted RCTs that would contribute to populating the funnel plot and probably mitigate any potential presence of publication bias [84].”
ResultChen, Y. C., Chen, W.-C., Liu, C.-W., Huang, W.-Y., Lu, I., Lin, C. W., Huang, R. Y., Chen, J. S., & Huang, C. H. (2023) · European Review of Aging and Physical Activity · doi:10.1186/s11556-023-00333-4 · Cited by 34 · see citing works
Neo, H. S., Tan, J. H., Ang, W. H. D., & Lau, Y. (2022) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2022.110790 · Cited by 32 · see citing works
Cabezas-Klinger, H., Fernandez-Daza, F. F., & Mina-Paz, Y. (2025) · Behavioral Sciences · doi:10.3390/bs15111450 · Cited by 18 · see citing works
“We assessed publication-bias risk using Egger’s regression test and the funnel plot (Lin & Chu, 2018; Egger et al., 1997) (Figures 5 and 6).”
Park, S., Tang, A., Pollock, C., & Sakakibara, B. M. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-055527 · Cited by 12 · see citing works
Heidari‐Soureshjani, R., Nasrabadi, A. N., Zakerimoghadam, M., Mohammadi, T., & Rasti, A. (2023) · Health Science Reports · doi:10.1002/hsr2.1536 · Cited by 10 · see citing works
Ibrahim, A. A. E., McWilliams, D. F., Smith, S. L., Chaplin, W. J., Salimian, M., Georgopoulos, V., Kouraki, A., & Walsh, D. A. (2025) · Annals of Physical and Rehabilitation Medicine · doi:10.1016/j.rehab.2024.101894 · Cited by 9 · see citing works
Mundim de Oliveira, I., Santos, R. C. D., Alves Silva, R., Figueiredo Alves, R. R., Teodoro Martins, B. C., & Soares, L. R. (2024) · Revista Brasileira de Ginecologia e Obstetrícia · doi:10.61622/rbgo/2024rbgo28 · Cited by 4 · see citing works
Clinical Trials Gone Missing—A Potential Source for Publication Bias in Dentistry
Tomack, J., Mascardo, K., Chen, C.-Y., Chen, T., Li, X., Kim, D. M., & Machtei, E. E. (2024) · Publications · doi:10.3390/publications12030023
“When referring to clinical trials, withholding negative results from publication not only distorts the literature but may even potentially have major public health implications [2,3].”
Background
- Authors
- Lin, L., & Chu, H.
- Affiliations
- University of Minnesota (US)
- Corresponding author
- Lin, Lifeng
- Chu, Haitao
- Venue
- Biometrics
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-11-15
- Volume
- 74
- Issue
- 3
- Pages
- 785-794
- ISSN
- 0006-341X (ISSN-L)
- 1541-0420
- PMC
- PMC5953768
- OpenAlex
- W2769488285
- Cited by
- 1,674
- References
- 21
- Citations by year
- 2026: 276 · 2025: 357 · 2024: 353 · 2023: 224 · 2022: 178 · 2021: 155 · 2020: 101 · 2019: 19 · 2018: 8 · 2017: 2
- FWCI
- 31.81
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Biometrics (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- NIAID (R21 AI103012, HC, LL, NIDCR R03 DE024750, NLM R21 LM012197, U01 DK106786)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- publication bias, meta-analysis, funnel plot, trim and fill, selection model, standardized effect size, large sample properties
- MeSH terms
- Animals, Computer Simulation, Humans, Reference Standards, Regression Analysis, Meta-Analysis as Topic, Models, Statistical, Ocular Motility Disorders, Publication Bias, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
