The DOI matches its Crossref record, with a small difference in title or year; 49 of the 49 references it cites were found in a registry or an index.
Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes
Open accessAbstract
We review and evaluate selection methods, a prominent class of techniques first proposed by Hedges (1984) that assess and adjust for publication bias in meta-analysis, via an extensive simulation study. Our simulation covers both restrictive settings as well as more realistic settings and proceeds across multiple metrics that assess different aspects of model performance. This evaluation is timely in light of two recently proposed approaches, the so-called p-curve and p-uniform approaches, that can be viewed as alternative implementations of the original Hedges selection method approach. We find that the p-curve and p-uniform approaches perform reasonably well but not as well as the original Hedges approach in the restrictive setting for which all three were designed. We also find they perform poorly in more realistic settings, whereas variants of the Hedges approach perform well. We conclude by urging caution in the application of selection methods: Given the idealistic model assumptions underlying selection methods and the sensitivity of population average effect size estimates to them, we advocate that selection methods should be used less for obtaining a single estimate that purports to adjust for publication bias ex post and more for sensitivity analysis—that is, exploring the range of estimates that result from assuming different forms of and severity of publication bias.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Consistent with those findings, the relative probabilities estimated by selection method analyses (e.g., McShane et al., 2016) also suggest a general lack of publication bias.”
From Examining domains of psychological flexibility and inflexibility as treatment mechanisms in acceptance and commitment therapy: A comprehensive systematic and meta-analytic review (2024) · cites this paper as McShane et al., 2016
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On the other hand, this could also be an overestimate of average study power, as heterogeneity in effect sizes can cause the p-curve to overestimate mean power (Brunner & Schimmack, 2020; McShane et al., 2016).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as McShane et al., 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we ran selection method analyses for each meta-analytic effect using R functions developed and validated by McShane et al. (2016).”
From Examining domains of psychological flexibility and inflexibility as treatment mechanisms in acceptance and commitment therapy: A comprehensive systematic and meta-analytic review (2024) · cites this paper as McShane et al., 2016
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, the credibility of such results is contingent on a lack of selective reporting or on the ability to effectively correct for missing evidence (McShane et al., 2016; Stanley and Doucouliagos, 2014).”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as McShane et al., 2016
The cited source says what the claim attributes to it.
In the citing paper
“In general, cumulative or meta-analytic evidence obtained from multiple independently conducted experiments provides a better basis for assessing the reliability of findings than evidence from a single experiment.”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as McShane et al., 2016
The cited source says what the claim attributes to it.
In the citing paper
“p-uniform does not model heterogeneity, and this can lead it to overestimate the average effect size of all studies (McShane et al., 2016).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as McShane et al., 2016
The cited source says what the claim attributes to it.
In the citing paper
“Bias adjustments can improve performance relative to a meta-analysis that does not adjust for publication bias (Carter et al., 2019; McShane et al., 2016; Moreno et al., 2009; Simonsohn et al., 2014).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as McShane et al., 2016
In the citing paper
“Statisticians have proposed several statistical approaches to assess how robust meta-analyses are to various assumptions about the extent and nature of missing results, including trim-andfill 67 , selection models 68 , and regression-based adjustment methods 69 .”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [69]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 2×
“…& Mazumdar, 1994; Egger, Smith, Schneider, & Minder, 1997; Sterne & Egger, 2005), Rosenthal’s fail-safe N method and its many variants (Becker, 2005; Rosenthal, 1979), the trim-and-fill method (Duval, 2005; Duval & Tweedie, 2000a, 2000b), and the Precision-Effect Test– Precision-Effect Estimate…”
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Mathematical Statistics: Basic Ideas and Selected Topics.
Wachter, K. W., Bickel, P. J., & Doksum, K. A. (1979) · Journal of the American Statistical Association · doi:10.2307/2286373 · Cited by 1,886
Sur les fonctions convexes et les inégalités entre les valeurs moyennes
Jensen, J. L. (1906) · Acta Mathematica · doi:10.1007/bf02418571 · Cited by 1,886
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 4×
“These concerns date back centuries (Boyle, 1661/1965; Dickersin, 2005; Editors, 1909; Ferriar & Simmons, 1792; Hall, 1959; Lane & Dunlap, 1978; Sterling, 1959) and relate not only to issues surrounding the size, direction, and statistical significance of study results but also to issues surrounding the availability an…”
BackgroundResultInfluentialStanley, T. D., & Doucouliagos, H. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1095 · Cited by 916
Regression Methods to Detect Publication and Other Bias in Meta‐Analysis
Sterne, J. A. C., & Egger, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch6 · Cited by 826 · cited 1×
“…plot (Sterne, Becker, & Egger, 2005), nonparametric and regression-based tests (Begg & Mazumdar, 1994; Egger, Smith, Schneider, & Minder, 1997; Sterne & Egger, 2005), Rosenthal’s fail-safe N method and its many variants (Becker, 2005; Rosenthal, 1979), the trim-and-fill method (Duval, 2005;…”
The Philosophy of Multiple Comparisons
Tukey, J. W. (1991) · Statistical Science · doi:10.1214/ss/1177011945 · Cited by 773
“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734
p -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717
Duval, S. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch8 · Cited by 496
Publication Bias in Research Synthesis: Sensitivity Analysis Using A Priori Weight Functions.
Vevea, J. L., & Woods, C. M. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.428 · Cited by 461 · cited 3×
“…less for obtaining a single estimate that purports to adjust for publication bias ex post and more for sensitivity analysis—that is, exploring the range of estimates that result from assuming different forms of and severity of publication bias (see Vevea & Woods, 2005, and Hedges & Vevea, 2005).”
BackgroundMethodPigott, T. D. (2012) · doi:10.1007/978-1-4614-2278-5 · Cited by 453
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453
Inference for Non-random Samples
Copas, J. B., & Li, H. G. (1997) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/1467-9868.00055 · Cited by 398
Selection Models and the File Drawer Problem
Iyengar, S., & Greenhouse, J. B. (1988) · Statistical Science · doi:10.1214/ss/1177013012 · Cited by 388
A general linear model for estimating effect size in the presence of publication bias
Vevea, J. L., & Hedges, L. (1995) · Psychometrika · doi:10.1007/bf02294384 · Cited by 367
Modeling Publication Selection Effects in Meta-Analysis
Hedges, L. V. (1992) · Statistical Science · doi:10.1214/ss/1177011364 · Cited by 359
Sterne, J. A. C., Becker, B. J., & Egger, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch5 · Cited by 341
Publication Bias: Recognizing the Problem, Understanding Its Origins and Scope, and Preventing Harm
Dickersin, K. (2006) · doi:10.1002/0470870168.ch2 · Cited by 333
Meta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265
A sensitivity analysis for publication bias in systematic reviews
Copas, J. B., & Shi, J. Q. (2001) · Statistical Methods in Medical Research · doi:10.1177/096228020101000402 · Cited by 264
Statistical methods for dealing with publication bias in meta‐analysis
Jin, Z., Zhou, X., & He, J. (2014) · Statistics in Medicine · doi:10.1002/sim.6342 · Cited by 262
Hedges, L. V. (1984) · Journal of Educational Statistics · doi:10.3102/10769986009001061 · Cited by 241
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616650874 · Cited by 219
An Approach for Assessing Publication Bias Prior to Performing a Meta-Analysis
Dear, K., & Begg, C. (1992) · Statistical Science · doi:10.1214/ss/1177011363 · Cited by 202
R.Black, T. (2002) · SAGE Publications Ltd eBooks · doi:10.4135/9780857020208.n7 · Cited by 198
Failsafe N or File‐Drawer Number
Becker, B. J. (2005) · Wiley eBooks · doi:10.1002/0470870168.ch7 · Cited by 179
What Works?: Selectivity Models and Meta-Analysis
Copas, J. B. (1999) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/1467-985x.00123 · Cited by 168
Estimating effect size: Bias resulting from the significance criterion in editorial decisions
Lane, D. M., & Dunlap, W. P. (1978) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1978.tb00578.x · Cited by 161
The Connection Between Varying Treatment Effects and the Crisis of Unreplicable Research
Gelman, A. (2015) · Journal of Management · doi:10.1177/0149206314525208 · Cited by 125
Hedges, L. V., & Vevea, J. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch9 · Cited by 123
You Cannot Step Into the Same River Twice
McShane, B. B., & Böckenholt, U. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614548513 · Cited by 96
Self-citationBlinding Us to the Obvious? The Effect of Statistical Training on the Evaluation of Evidence
McShane, B. B., & Gal, D. (2015) · Management Science · doi:10.1287/mnsc.2015.2212 · Cited by 77
Self-citationweightr: Estimating Weight-Function Models for Publication Bias
Coburn, K. M., & Vevea, J. L. (2016) · doi:10.32614/cran.package.weightr · Cited by 61
Planning sample sizes when effect sizes are uncertain: The power-calibrated effect size approach.
McShane, B. B., & Böckenholt, U. (2015) · Psychological Methods · doi:10.1037/met0000036 · Cited by 56
Self-citationA Likelihood-Based Sensitivity Analysis for Publication Bias in Meta-Analysis
Copas, J. B. (2012) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/j.1467-9876.2012.01049.x · Cited by 41
Robert Boyle on Natural Philosophy, an Essay with Selections from his Writings
Hall, M. B., & Phillips, J. (1965) · American Journal of Physics · doi:10.1119/1.1971036 · Cited by 23
(1909) · New England Journal of Medicine · doi:10.1056/nejm190908191610809 · Cited by 5
Research works printed but not linked: 4
- Boyle R., 1965, The sceptical chymist
- Ferriar J., 1792, Medical histories and reflection
- Fisher Ronald A, 1938, Sankhya, 4, 14
- Sterling T. D., 1959, Journal of the American Statistical Association, 54, 30
Cited by
Sorted by citations
Salk, R. H., Hyde, J. S., & Abramson, L. Y. (2017) · Psychological Bulletin · doi:10.1037/bul0000102 · Cited by 2,622
Debunking: A Meta-Analysis of the Psychological Efficacy of Messages Countering Misinformation
Chan, M.-P. S., Jones, C. R., Hall Jamieson, K., & Albarracín, D. (2017) · Psychological Science · doi:10.1177/0956797617714579 · Cited by 898
A social comparison theory meta-analysis 60+ years on.
Gerber, J. P., Wheeler, L., & Suls, J. (2018) · Psychological Bulletin · doi:10.1037/bul0000127 · Cited by 808
Self-regulation in childhood as a predictor of future outcomes: A meta-analytic review.
Robson, D. A., Allen, M. S., & Howard, S. J. (2020) · Psychological Bulletin · doi:10.1037/bul0000227 · Cited by 759
Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis
Shrout, P. E., & Rodgers, J. L. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011845 · Cited by 830
Metadata
- Authors
- McShane, B. B., Böckenholt, U., & Hansen, K. T.
- Affiliations
- Northwestern University (US)
- University of California San Diego (US)
- Corresponding author
- McShane, Blakeley B.
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2016-09
- Volume
- 11
- Issue
- 5
- Pages
- 730-749
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2524737933
- Cited by
- 398
- References
- 49
- Citations by year
- 2026: 30 · 2025: 43 · 2024: 43 · 2023: 46 · 2022: 50 · 2021: 46 · 2020: 44 · 2019: 50 · 2018: 23 · 2017: 21 · 2016: 2
- FWCI
- 29.71
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, selection methods, p-curve, meta-analysis, sensitivity analysis, effect size estimation, simulation study
- MeSH terms
- Computer Simulation, Data Interpretation, Statistical, Humans, Meta-Analysis as Topic, Models, Statistical, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
