Sensitivity Analysis for Publication Bias in Meta-Analyses
Open accessAbstract
Summary: We propose sensitivity analyses for publication bias in meta-analyses. We consider a publication process such that ‘statistically significant’ results are more likely to be published than negative or “non-significant” results by an unknown ratio, η. Our proposed methods also accommodate some plausible forms of selection based on a study's standard error. Using inverse probability weighting and robust estimation that accommodates non-normal population effects, small meta-analyses, and clustering, we develop sensitivity analyses that enable statements such as ‘For publication bias to shift the observed point estimate to the null, “significant” results would need to be at least 30 fold more likely to be published than negative or “non-significant” results’. Comparable statements can be made regarding shifting to a chosen non-null value or shifting the confidence interval. To aid interpretation, we describe empirical benchmarks for plausible values of η across disciplines. We show that a worst-case meta-analytic point estimate for maximal publication bias under the selection model can be obtained simply by conducting a standard meta-analysis of only the negative and ‘non-significant’ studies; this method sometimes indicates that no amount of such publication bias could ‘explain away’ the results. We illustrate the proposed methods by using real meta-analyses and provide an R package: PublicationBias.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Mathur and VanderWeele's sensitivity analysis seems to be relatively unbiased with values of η below 20 (i.e., a publication probability 20 times higher for positive than null or negative results).
“Finally, the Mathur and VanderWeele's sensitivity analysis seems to be relatively unbiased with values of η below 20 (i.e., a publication probability 20 times higher for positive than null or negative results; Mathur & VanderWeele, 2020).”
From Please don't stop the music: A meta-analysis of the cognitive and academic benefits of instrumental musical training in childhood and adolescence (2022) · cites this paper as Mathur & VanderWeele, 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 2× · see citing works
“…minimum number of unpublished studies with a mean point estimate of 0 (or another fixed value) that would need to exist for their inclusion in the meta-analysis to reduce the pooled point estimate to ‘statistical non-significance’ (Rosenthal, 1979) or to a given effect size threshold (Orwin, 1983).”
BackgroundSensitivity analysis in observational research: Introducing the E-value
VanderWeele, T. J., & Ding, P. (2017) · Annals of Internal Medicine · doi:10.7326/m16-2607 · Cited by 6,350 · see citing works
Self-citationMethods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542 · see citing works
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282 · see citing works
A Fail-Safe N for Effect Size in Meta-Analysis
Orwin, R. G. (1983) · Journal of Educational Statistics · doi:10.2307/1164923 · Cited by 2,064 · see citing works
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053 · see citing works
Anderson, C. A., Shibuya, A., Ihori, N., Swing, E. L., Bushman, B. J., Sakamoto, A., Rothstein, H. R., & Saleem, M. (2010) · Psychological Bulletin · doi:10.1037/a0018251 · Cited by 2,048 · see citing works
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845 · see citing works
Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 1× · see citing works
“…it would be possible to conduct maximum likelihood sensitivity analyses under the standard parametric random-effects model, invoking the additional assumptions that, among the published studies, γi ∼IID N.0, τ2/ and i ∼IID N.0, σ2i / (e.g. Brockwell and Gordon (2001) and Viechtbauer (2005)).”
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · see citing works
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · cited 2× · see citing works
“…that we modelled selection by using a single cut-off at p = 0:05 both for ease of interpretation and because it conforms well to empirical findings on how applied researchers and statisticians interpret and report p-values (Head et al., 2015; Masicampo and Lalande, 2012; McShane and Gal, 2017).”
ResultField, A. P., & Gillett, R. (2010) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711010x502733 · Cited by 1,162 · see citing works
The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052 · cited 1× · see citing works
“In practice, most meta-analyses are far too small to apply selection models; for example, the percentage of meta-analyses comprising fewer than 10 studies probably exceeds 50% in the Cochrane database (Ioannidis and Trikalinos, 2007) and in medical journals (Sterne et al., 2000).”
BackgroundInverse probability weighted estimation for general missing data problems
Wooldridge, J. M. (2007) · Journal of Econometrics · doi:10.1016/j.jeconom.2007.02.002 · Cited by 1,029 · cited 1× · see citing works
“In the on-line supplement (section 1.4), we derive a parametric specification that is directly analogous to inverse probability weighting for survey sampling or missing data for general M-estimators (Wooldridge, 2007).”
MethodInfluentialConsequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 2× · see citing works
“, 2004; Coursol and Wagner, 1986) and selective submission of papers (Franco et al., 2014; Greenwald, 1975), as well as editors’ and reviewers’ biases (although some empirical work suggests that the latter may be a weaker influence than author decision making; see Lee et al.”
BackgroundStanley, T. D., & Doucouliagos, H. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1095 · Cited by 916 · see citing works
Publication Bias in Research Synthesis: Sensitivity Analysis Using A Priori Weight Functions.
Vevea, J. L., & Woods, C. M. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.428 · Cited by 461 · cited 4× · see citing works
“…meta-analysis are only mildly attenuated even under severe publication bias, then the results might be considered fairly robust to publication bias (Vevea and Woods, 2005), whereas if the results are severely attenuated under mild publication bias the results might be considered sensitive to…”
BackgroundMethodResultInfluentialMcShane, B. B., Böckenholt, U., & Hansen, K. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616662243 · Cited by 398 · see citing works
Publication Bias in Editorial Decision Making
Olson, C. M. (2002) · JAMA · doi:10.1001/jama.287.21.2825 · Cited by 395 · see citing works
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388 · see citing works
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359 · see citing works
Ferguson, C. J., & Kilburn, J. (2010) · Psychological Bulletin · doi:10.1037/a0018566 · Cited by 311 · see citing works
A peculiar prevalence of <i>p</i> values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266 · see citing works
Hilgard, J., Engelhardt, C. R., & Rouder, J. N. (2017) · Psychological Bulletin · doi:10.1037/bul0000074 · Cited by 162 · cited 1× · see citing works
“…could be explained away by publication bias; other researchers suggested that publication bias might largely explain these results (Ferguson and Kilburn, 2010; Hilgard et al., 2017), whereas the original authors argued that the results were in fact robust to publication bias (Kepes et al., 2017).)”
Result
Flagged first, then by citations
Research works printed but not linked: 21
- Bergmann, C., Tsuji, S., Piccinini, P. E., Lewis, M. L., Braginsky, M., Frank, M. C. and Cristia, A. (2018) Promoting replicability in developmental research through meta‐analyses: insights from language acquisition research. Chld Devlpmnt, 89, 1996– 2009. PMC6282795 29736962
- Boehm, J. K., Chen, Y., Koga, H., Mathur, M. B., Vie, L. L. and Kubzansky, L. D. (2018) Is optimism associated with healthier cardiovascular‐related behavior?: Meta‐analyses of 3 health behaviors. Circuln Res., 122, 1119– 1134. 29650630
- Bom, P. R. and Rachinger, H. (2019) A kinked meta‐regression model for publication bias correction. Res. Synth. Meth., 10, 497– 514. 31039283
- Brockwell, S. E. and Gordon, I. R. (2001) A comparison of statistical methods for meta‐analysis. Statist. Med., 20, 825– 840. 11252006
- Ding, P. and VanderWeele, T. J. (2016) Sensitivity analysis without assumptions. Epidemiology, 27, no. 3, 368– 377. 26841057 PMC4820664
- Egger, M., Smith, G. D., Schneider, M. and Minder, C. (1997) Bias in meta‐analysis detected by a simple, graphical test. Br. Med. J., 315, 629– 634. 9310563 PMC2127453
- Fisher, Z. and Tipton, E. (2015) Robumeta: an R‐package for robust variance estimation in meta‐analysis. Preprint arXiv:1503.02220. University of North Carolina at Chapel Hill, Chapel Hill. · arXiv
- Hedges, L. V., Tipton, E. and Johnson, M. C. (2010) Robust variance estimation in metaregression with dependent effect size estimates. Res. Synth. Meth., 1, 39– 65. 26056092
- Jin, Z.‐C., Zhou, X.‐H. and He, J. (2015) Statistical methods for dealing with publication bias in meta‐analysis. Statist. Med., 34, 343– 360. 25363575
- Johnson, V. E., Payne, R. D., Wang, T., Asher, A. and Mandal, S. (2017) On the reproducibility of psychological science. J. Am. Statist. Ass., 112, 1– 10. PMC5976261 29861517
- Kepes, S., Bushman, B. J. and Anderson, C. A. (2017) Violent video game effects remain a societal concern: Reply to Hilgard, Engelhardt, and Rouder (2017). Perspect. Psychol. Sci., 143, no. 7, 775– 782. 28639811
- Lee, K. P., Boyd, E. A., Holroyd‐Leduc, J. M., Bacchetti, P. and Bero, L. A. (2006) Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med. J. Aust., 184, 621– 626. 16803442
- Lewis, M., Braginsky, M., Tsuji, S., Bergmann, C., Piccinini, P. E., Cristia, A. and Frank, M. C. (2017) Quantitative synthesis of early language acquisition using meta‐analysis. Preprint (Available from https://psyarxiv.com/htsjm.)
- Li, X., Dai, D., Chen, B., Tang, H., Xie, X. and Wei, W. (2018) Efficacy of PI3K/AKT/mTOR pathway inhibitors for the treatment of advanced solid cancers: a literature‐based meta‐analysis of 46 randomised control trials. PLOS One, 13, no. 2, article e0192464. PMC5800666 29408858
- Mathur, M. B. and VanderWeele, T. J. (2018) New metrics for meta‐analyses of heterogeneous effects. Statist. Med., 38, 1336– 1342. PMC6519385 30513552
- Mathur, M. B. and VanderWeele, T. J. (2020a) Estimating publication bias in meta‐analyses: a meta‐meta‐analysis across disciplines and journal tiers. Preprint Quantitative Sciences Unit, Stanford University, Stanford. (Available from https://osf.io/p3xyd/.)
- Mathur, M. B. and VanderWeele, T. J. (2020b) Robust metrics and sensitivity analyses for meta‐analyses of heterogeneous effects. Epidemiology, 31, 356– 358. 32141922 PMC7963111
- McShane, B. B. and Gal, D. (2017) Statistical significance and the dichotomization of evidence. J. Am. Statist. Ass., 112, 885– 895. PMC5769160 29348701
- Tipton, E. (2015) Small‐sample adjustments for robust variance estimation with meta‐regression. Psychol. Meth., 20, no. 3, 375– 393. 24773356
- VanderWeele, T. J., Mathur, M. B. and Ding, P. (2019) Correcting misinterpretations of the e‐value. Ann. Intern. Med., 170, 131– 132. 30597489
- Wang, C.‐C. and Lee, W.‐C. (2019) A simple method to estimate prediction intervals and predictive distributions: summarizing meta‐analyses beyond means and confidence intervals. Res. Synth. Meth, 10, 255– 266. 30835918
Cited by
Sorted by citations
Noetel, M., Sanders, T., Gallardo-Gómez, D., Taylor, P., del Pozo Cruz, B., van den Hoek, D., Smith, J. J., Mahoney, J., Spathis, J., Moresi, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-075847 · Cited by 707 · see citing works
Digital interventions for the treatment of depression: a meta-analytic review
Moshe, I., Terhorst, Y., Philippi, P., Domhardt, M., Cuijpers, P., Cristea, I., Pulkki-Råback, L., Baumeister, H., & Sander, L. B. (2021) · Psychological Bulletin · doi:10.1037/bul0000334 · Cited by 390 · see citing works
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165 · see citing works
Li, J., Li, C., Puts, M., Wu, Y.-C., Lyu, M.-M., Yuan, B., & Zhang, J.-P. (2023) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2023.104447 · Cited by 127 · see citing works
Román-Caballero, R., Vadillo, M. A., Trainor, L. J., & Lupiáñez, J. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100436 · Cited by 118 · see citing works
Cerna-Turoff, I., Fang, Z., Meierkord, A., Wu, Z., Yanguela, J., Bangirana, C. A., & Meinck, F. (2021) · Trauma, Violence, & Abuse · doi:10.1177/1524838020985532 · Cited by 92 · see citing works
Panunzi, S., Maltese, S., De Gaetano, A., Capristo, E., Bornstein, S. R., & Mingrone, G. (2021) · Obesity Reviews · doi:10.1111/obr.13230 · Cited by 88 · see citing works
Boen, R., Quintana, D. S., Ladouceur, C. D., & Tamnes, C. K. (2022) · Psychophysiology · doi:10.1111/psyp.14003 · Cited by 40 · see citing works
- Authors
- Mathur, M. B., & VanderWeele, T. J.
- Affiliations
- Stanford University (US)
- Harvard University (US)
- Corresponding author
- Mathur, Maya B.
- Venue
- Journal of the Royal Statistical Society Series C: Applied Statistics
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-08-28
- Volume
- 69
- Issue
- 5
- Pages
- 1091-1119
- ISSN
- 0035-9254 (ISSN-L)
- 1467-9876
- PMC
- PMC7590147
- OpenAlex
- W2906361607
- Cited by
- 354
- References
- 25
- Citations by year
- 2026: 67 · 2025: 85 · 2024: 80 · 2023: 42 · 2022: 39 · 2021: 27 · 2020: 8 · 2019: 2 · 2017: 1 · 2015: 1 · 2014: 1
- FWCI
- 16.83
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the Royal Statistical Society Series C (Applied Statistics) (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- National Institutes of Health (R01 CA222147)
- John E. Fetzer Memorial Trust (R2020-16)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- publication bias, sensitivity analysis, meta-analysis, inverse probability weighting, robust estimation, non-significant results, clustering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
