Graphical augmentations to sample‐size‐based funnel plot in meta‐analysis
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Assessing publication bias is a critical procedure in meta‐analyses for rating the synthesized overall evidence. Because statistical tests for publication bias are usually not powerful and only give P values that inform either the presence or absence of the bias, examining the asymmetry of funnel plots has been popular to investigate potentially missing studies and the direction of the bias. Most funnel plots present treatment effects against their standard errors, and the contours depicting studies' significance levels have been used in the plots to distinguish publication bias from other factors (such as heterogeneity and subgroup effects) that may cause the plots' asymmetry. However, treatment effects and their standard errors are frequently associated even if no publication bias exists (eg, both variables depend on the four data cells in a 2 × 2 table for the odds ratio), so standard‐error‐based funnel plots may lead to false positive conclusions when such association may not be negligible. In addition, the missingness of studies may relate to their sample sizes besides P values (which are partly determined by standard errors); studies with more samples are more likely published. Therefore, funnel plots based on sample sizes can be an alternative tool. However, the contours for standard‐error‐based funnel plots cannot be directly applied to sample‐size‐based ones. This article introduces contours for sample‐size‐based funnel plots of various effect sizes, which may help meta‐analysts properly interpret such plots' asymmetry. We provide five examples to illustrate the use of the proposed contours.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Deeks, J. J., Macaskill, P., & Irwig, L. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.01.016 · Cited by 2,910 · cited 1×
“To control the false positive rate, many alternative regression tests have been introduced by replacing the standard errors with the sample sizes or some functions of the sample sizes.(12,13,24,35) For example, Tang and Liu(24) (henceforth Tang's test) suggest”
MethodPublication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282 · cited 2×
“In addition, simulation studies showed that the statistical tests had poor power in most situations.(16) Empirical studies based on a large collection of real meta‐analyses also indicated that various tests had low or moderate agreement for detecting publication bias.”
BackgroundMethodA modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267 · cited 1×
“Consequently, some meta‐ analysts refer publication bias to as “small‐study effects” to better describe the funnel plots' asymmetry.(16,23) To detect the association between the studies' effect sizes and their sample sizes, meta‐analysts can alter to use the sample‐size‐based funnel plots, which present the studies' s…”
MethodPeters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053
GRADE guidelines: 5. Rating the quality of evidence—publication bias
Guyatt, G. H., Oxman, A. D., Montori, V., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Djulbegovic, B., Atkins, D., Falck-Ytter, Y., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.011 · Cited by 1,819
Quantifying publication bias in meta-analysis
Lin, L., & Chu, H. (2018) · Biometrics · doi:10.1111/biom.12817 · Cited by 1,674
Self-citationEffectiveness of brief alcohol interventions in primary care populations
Kaner, E. F., Beyer, F. R., Muirhead, C., Campbell, F., Pienaar, E. D., Bertholet, N., Daeppen, J. B., Saunders, J. B., & Burnand, B. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004148.pub4 · Cited by 1,665
Publication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291 · cited 1×
“For example, studies with smaller P values tend to be published more likely than those with larger P values.(5) Various methods are available to deal with publication bias.”
BackgroundA comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286 · cited 2×
“To control the false positive rate, many alternative regression tests have been introduced by replacing the standard errors with the sample sizes or some functions of the sample sizes.(12,13,24,35) For example, Tang and Liu(24) (henceforth Tang's test) suggest”
BackgroundMethodThe appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052 · cited 1×
“Third, using the funnel plot (based either on the standard error or on the sample size) to assess publication bias has inherent limitations, because the plot's asymmetry may be due to various causes other than publication bias.(20,56) Even if contours of different significance levels are added to the standard‐error‐ o…”
MethodEffectiveness of brief alcohol interventions in primary care populations
Kaner, E. F. S., Dickinson, H. O., Beyer, F. R., Campbell, F., Schlesinger, C., Heather, N., Saunders, J. B., Burnand, B., & Pienaar, E. D. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004148.pub3 · Cited by 746
Statistical Power, Sample Size, and Their Reporting in Randomized Controlled Trials
Moher, D. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520020048013 · Cited by 608
Bayesian approaches to random-effects meta-analysis: a comparative study
Smith, T. C., Spiegelhalter, D. J., & Thomas, A. (1995) · Statistics in Medicine · doi:10.1002/sim.4780142408 · Cited by 568
Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study
Nuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W. S., Tschannen, B., Altman, D. G., Egger, M., & Juni, P. (2010) · BMJ · doi:10.1136/bmj.c3515 · Cited by 559
Impact of inspiratory muscle training in patients with COPD: what is the evidence?
Gosselink, R., De Vos, J., van den Heuvel, S. P., Segers, J., Decramer, M., & Kwakkel, G. (2011) · European Respiratory Journal · doi:10.1183/09031936.00031810 · Cited by 524
Empirical Comparison of Publication Bias Tests in Meta-Analysis
Lin, L., Chu, H., Murad, M. H., Hong, C., Qu, Z., Cole, S. R., & Chen, Y. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-018-4425-7 · Cited by 464 · cited 1×
“Empirical studies based on a large collection of real meta‐analyses also indicated that various tests had low or moderate agreement for detecting publication bias.(17) Therefore, it may be improper to assess publication bias by completely relying on a single statistical test.”
BackgroundSelf-citationMisleading funnel plot for detection of bias in meta-analysis
Tang, J.-L., & Liu, J. L. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00204-8 · Cited by 420
A meta‐analysis of declines in local species richness from human disturbances
Murphy, G. E. P., & Romanuk, T. N. (2014) · Ecology and Evolution · doi:10.1002/ece3.909 · Cited by 390
Tonelli, M., Manns, B., & Feller-Kopman, D. (2002) · American Journal of Kidney Diseases · doi:10.1053/ajkd.2002.36318 · Cited by 308
Warn, D. E., Thompson, S. G., & Spiegelhalter, D. J. (2002) · Statistics in Medicine · doi:10.1002/sim.1189 · Cited by 277
Group-based parent training programmes for improving parental psychosocial health
Barlow, J., Smailagic, N., Huband, N., Roloff, V., & Bennett, C. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002020.pub3 · Cited by 262
Song, F., Parekh-Bhurke, S., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, A. J., Hing, C. B., & Harvey, I. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-79 · Cited by 175
Pharmacotherapy for social anxiety disorder (SAnD)
Williams, T., Hattingh, C. J., Kariuki, C. M., Tromp, S. A., van Balkom, A. J., Ipser, J. C., & Stein, D. J. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001206.pub3 · Cited by 84
Isojarvi, J., Wood, H., Lefebvre, C., & Glanville, J. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1294 · Cited by 38
Cited by
Shi, L., & Lin, L. (2019) · Medicine · doi:10.1097/md.0000000000015987 · Cited by 818
Self-citationMusuuza, J. S., Watson, L., Parmasad, V., Putman-Buehler, N., Christensen, L., & Safdar, N. (2021) · PLOS ONE · doi:10.1371/journal.pone.0251170 · Cited by 544
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
“PB can be better identified if regions with observed studies correspond to areas of high statistical significance.(95,96) While these modifications have served to improve type I error rates, other issues persist.”
ResultPelland, J. C., Remmert, J. F., Robinson, Z. P., Hinson, S. R., & Zourdos, M. C. (2026) · Sports Medicine · doi:10.1007/s40279-025-02344-w · Cited by 18
Yunita, Y., Juandi, D., Hasanah, A., & Suparman (2022) · AIP Conference Proceedings · doi:10.1063/5.0102458 · Cited by 5
Metadata
- Authors
- Lin, L.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2019-02-07
- Volume
- 10
- Issue
- 3
- Pages
- 376-388
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2911778644
- Cited by
- 50
- References
- 26
- FWCI
- 3.34
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.