Empirical Comparison of Publication Bias Tests in Meta-Analysis
Open accessAbstract
Background Decision makers rely on meta-analytic estimates to trade off benefits and harms. Publication bias impairs the validity and generalizability of such estimates. The performance of various statistical tests for publication bias has been largely compared using simulation studies and has not been systematically evaluated in empirical data. Methods This study compares seven commonly used publication bias tests (i.e., Begg's rank test, trim-and-fill, Egger's, Tang's, Macaskill's, Deeks', and Peters' regression tests) based on 28,655 meta-analyses available in the Cochrane Library. Results Egger's regression test detected publication bias more frequently than other tests (15.7% in meta-analyses of binary outcomes and 13.5% in meta-analyses of non-binary outcomes). The proportion of statistically significant publication bias tests was greater for larger meta-analyses, especially for Begg's rank test and the trim-and-fill method. The agreement among Tang's, Macaskill's, Deeks', and Peters' regression tests for binary outcomes was moderately strong (most κ's were around 0.6). Tang's and Deeks' tests had fairly similar performance (κ > 0.9). The agreement among Begg's rank test, the trim-and-fill method, and Egger's regression test was weak or moderate (κ < 0.5). Conclusions Given the relatively low agreement between many publication bias tests, meta-analysts should not rely on a single test and may apply multiple tests with various assumptions. Non-statistical approaches to evaluating publication bias (e.g., searching clinical trials registries, records of drug approving agencies, and scientific conference proceedings) remain essential.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Funnel plot asymmetry was visually assessed for 5 or more studies.
“Funnel plot asymmetry, involving 5 or more studies, was visually assessed due to biased results in statistical tests for prevalence outcomes in single-group studies (15,16).”
From Treatment Preferences in Patients With Hypothyroidism (2025) · cites this paper as [15]
The Analysis Approach section says the paper considered meta-analyses containing at least five studies. The Introduction discusses inflated false-positive rates for some publication-bias tests with binary outcomes. It does not say that funnel plots were visually assessed for prevalence outcomes in single-group studies; its analysis concerns statistical tests in Cochrane meta-analyses. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955 · see citing works
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709 · see citing works
A basic introduction to fixed-effect and random-effects models for meta-analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.12 · Cited by 6,793 · see citing works
The combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380 · see citing works
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779 · see citing works
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955 · see citing works
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282 · see citing works
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267 · cited 1× · see citing works
“, design flaws such as inadequate allocation concealment) and they commonly enroll high-risk individuals, the funnel plot can be viewed as an approach to evaluating small study effects in general, rather than publication bias in particular.(14,31,32) In addition, the P value plots in Figures 2 and 3 indicate that some…”
BackgroundInvestigating and dealing with publication and other biases in meta-analysis
Sterne, J. A. C., Egger, M., & Smith, G. D. (2001) · BMJ · doi:10.1136/bmj.323.7304.101 · Cited by 2,225 · see citing works
Empirical assessment of effect of publication bias on meta-analyses
Sutton, A. J. (2000) · BMJ · doi:10.1136/bmj.320.7249.1574 · Cited by 1,398 · see citing works
Publication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291 · see citing works
A comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286 · see citing works
The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052 · see citing works
Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events
Bradburn, M. J., Deeks, J. J., Berlin, J. A., & Russell Localio, A. (2007) · Statistics in Medicine · doi:10.1002/sim.2528 · Cited by 938 · see citing works
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007) · Statistics in Medicine · doi:10.1002/sim.2889 · Cited by 716 · see citing works
Moreno, S. G., Sutton, A. J., Ades, A., Stanley, T. D., Abrams, K. R., Peters, J. L., & Cooper, N. J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-2 · Cited by 373 · see citing works
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359 · see citing works
Flagged first, then by citations
Research works printed but not linked: 25
- Begg CB, Berlin JA. Publication bias: a problem in interpreting medical data. J R Stat Soc Ser A (Stat Soc). 1988;151(3):419-63.
- Kicinski M. Publication bias in recent meta-analyses. PLoS ONE. 2013;8(11):e81823.
- Lin L, Chu H. Quantifying publication bias in meta-analysis. Biometrics. 2017. https://doi.org/10.1111/biom.12817. · DOI · Paper page
- Eyding D, Lelgemann M, Grouven U, et al. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ. 2010;341:c4737.
- Murad MH, Montori VM, Ioannidis JPA, et al. How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature. JAMA. 2014;312(2):171-79.
- Light RJ, Pillemer DB. Summing Up: The Science of Reviewing Research. Cambridge: Harvard University Press; 1984.
- Egger M, Davey Smith G, Schneider M, et al. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629-34.
- Rothstein HR, Sutton AJ, Borenstein M. Publication Bias in Meta- Analysis: Prevention, Assessment and Adjustments. Chichester: Wiley; 2005.
- Tang J-L, Liu JLY. Misleading funnel plot for detection of bias in meta- analysis. J Clin Epidemiol. 2000;53(5):477-84.
- Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol. 2005;58(9):882-93.
- Peters JL, Sutton AJ, Jones DR, et al. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006;295(6):676-80.
- Bürkner PC, Doebler P. Testing for publication bias in diagnostic meta- analysis: a simulation study. Stat Med. 2014;33(18):3061-77.
- Walter SD, Cook RJ. A comparison of several point estimators of the odds ratio in a single 2 × 2 contingency table. Biometrics. 1991;47(3):795-811.
- Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Chichester: Wiley; 2008.
- Sweeting MJ, Sutton AJ, Paul LC. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Stat Med. 2004;23(9):1351-75.
- Cohen J. A coefficient of agreement for nominal scales. Educ Psychosoc Meas. 1960;20(1):37-46.
- Sterne JAC, Egger M, Davey Smith G. Investigating and dealing with p u b l i c a t i o n a n d o t h e r b i a s e s i n m e t a -a n a l y s i s . B M J . 2001;323(7304):101-05.
- Lin L, Chu H, Hodges JS. Alternative measures of between-study heterogeneity in meta-analysis: reducing the impact of outlying studies. Biometrics. 2017;73(1):156-66.
- Normand S-LT. Meta-analysis: formulating, evaluating, combining, and reporting. Stat Med. 1999;18(3):321-59.
- Peters JL, Sutton AJ, Jones DR, et al. Assessing publication bias in meta-analyses in the presence of between-study heterogeneity. J R Stat Soc Ser A (Stat Soc). 2010;173(3):575-91.
- Higgins JPT, Thompson SG, Deeks JJ, et al. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557-60.
- Higgins JPT. Commentary: heterogeneity in meta-analysis should be expected and appropriately quantified. Int J Epidemiol. 2008;37(5):1158- 60.
- Ioannidis JPA, Patsopoulos NA, Rothstein HR. Reasons or excuses for avoiding meta-analysis in forest plots. BMJ. 2008;336(7658):1413-15.
- Whitehead A, Whitehead J. A general parametric approach to the meta-analysis of randomized clinical trials. Stat Med. 1991;10(11):1665-77.
- Alonso-Coello P, Schünemann HJ, Moberg J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016.
Cited by
Sorted by citations
Ntoumanis, N., Ng, J. Y. Y., Prestwich, A., Quested, E., Hancox, J. E., Thøgersen-Ntoumani, C., Deci, E. L., Ryan, R. M., Lonsdale, C., & Williams, G. C. (2021) · Health Psychology Review · doi:10.1080/17437199.2020.1718529 · Cited by 1,154 · see citing works
“Having noted this, it should also be stated that tests of funnel plot asymmetry typically have low power (e.g., Lin et al., 2018).”
BackgroundShi, L., & Lin, L. (2019) · Medicine · doi:10.1097/md.0000000000015987 · Cited by 818 · see citing works
Self-citationAssessing risk of bias due to missing results in a synthesis
Page, M. J., Higgins, J. P., & Sterne, J. A. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch13 · Cited by 402 · see citing works
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165 · see citing works
Self-citationA proposed framework to guide evidence synthesis practice for meta-analysis with zero-events studies
Xu, C., Furuya-Kanamori, L., Zorzela, L., Lin, L., & Vohra, S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.012 · Cited by 117 · see citing works
Self-citationJu, K., Lin, L., Chu, H., Cheng, L.-L., & Xu, C. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01035-6 · Cited by 61 · see citing works
Self-citationWu, S., Wang, J., Guo, Q., Lan, H., Sun, Y., Ren, M., Liu, Y., Wang, P., Wang, L., Su, R., et al. (2023) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2023.03.002 · Cited by 56 · see citing works
Self-citationAlami, A., Krewski, D., Farhat, N., Mattison, D., Wilson, K., Gravel, C. A., Farrell, P. J., Crispo, J. A. G., Haddad, N., Perez-Lloret, S., et al. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-065687 · Cited by 44 · see citing works
The Influence of Active and Passive Procrastination on Academic Performance: A Meta-Analysis
Kooren, N. S., Van Nooijen, C., & Paas, F. (2024) · Education Sciences · doi:10.3390/educsci14030323 · Cited by 33 · see citing works
“These tests are most accurate in detecting publication bias and small study bias when compared to the Tang’s regression test and Begg’s regression test [50].”
Ammar, A., Trabelsi, K., Boujelbane, M. A., Boukhris, O., Glenn, J. M., Chtourou, H., & Schöllhorn, W. I. (2023) · Educational Research Review · doi:10.1016/j.edurev.2023.100537 · Cited by 23 · see citing works
Jalali, A., Ziapour, A., Ezzati, E., Kazemi, S., & Kazeminia, M. (2024) · American Journal of Health Promotion · doi:10.1177/08901171231224101 · Cited by 19 · see citing works
Treatment Preferences in Patients With Hypothyroidism
de Lima Beltrão, F. E., Carvalhal, G., de Almeida Beltrão, D. C., de Lima Beltrão, F. E., Ribeiro, M. O., Ettleson, M. D., Ramos, H. E., & Bianco, A. C. (2025) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/clinem/dgae651 · Cited by 15 · see citing works
Ran, A. R., Ding, J. L., Tang, Z., Lam, C., Nguyen, T. X., Zhou, J., Zhang, S., Fang, D., Yang, D., Ng, V., et al. (2026) · Diabetes Care · doi:10.2337/dc25-1493 · Cited by 7 · see citing works
- Authors
- Lin, L., Chu, H., Murad, M. H., Hong, C., Qu, Z., Cole, S. R., & Chen, Y.
- Affiliations
- Florida State University (US)
- University of Minnesota (US)
- Mayo Clinic (US)
- Harvard University (US)
- Beijing Normal University (CN)
- University of North Carolina at Chapel Hill (US)
- University of Pennsylvania (US)
- Corresponding author
- Chu, Haitao
- Venue
- Journal of General Internal Medicine
- Published
- 2018-04-16
- Volume
- 33
- Issue
- 8
- Pages
- 1260-1267
- ISSN
- 0884-8734 (ISSN-L)
- 1525-1497
- PMC
- PMC6082203
- OpenAlex
- W2801697966
- Cited by
- 464
- References
- 20
- Citations by year
- 2026: 83 · 2025: 114 · 2024: 83 · 2023: 67 · 2022: 42 · 2021: 33 · 2020: 23 · 2019: 15 · 2018: 2
- FWCI
- 13.92
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Journal of General Internal Medicine (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Carolina Digital Repository (University of North Carolina at Chapel Hill) (submittedVersion)
- UNC Libraries
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- publication bias, trim and fill, test agreement, meta-analysis validity, binary outcomes
- MeSH terms
- Systematic Reviews as Topic, Humans, Meta-Analysis as Topic, Publication Bias, Empirical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
