The DOI matches its Crossref record, with a small difference in title or year; 49 of the 49 references it cites were found in a registry or an index.
Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes
Open accessAbstract
We review and evaluate selection methods, a prominent class of techniques first proposed by Hedges (1984) that assess and adjust for publication bias in meta-analysis, via an extensive simulation study. Our simulation covers both restrictive settings as well as more realistic settings and proceeds across multiple metrics that assess different aspects of model performance. This evaluation is timely in light of two recently proposed approaches, the so-called p-curve and p-uniform approaches, that can be viewed as alternative implementations of the original Hedges selection method approach. We find that the p-curve and p-uniform approaches perform reasonably well but not as well as the original Hedges approach in the restrictive setting for which all three were designed. We also find they perform poorly in more realistic settings, whereas variants of the Hedges approach perform well. We conclude by urging caution in the application of selection methods: Given the idealistic model assumptions underlying selection methods and the sensitivity of population average effect size estimates to them, we advocate that selection methods should be used less for obtaining a single estimate that purports to adjust for publication bias ex post and more for sensitivity analysis—that is, exploring the range of estimates that result from assuming different forms of and severity of publication bias.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Consistent with those findings, the relative probabilities estimated by selection method analyses (e.g., McShane et al., 2016) also suggest a general lack of publication bias.”
From Examining domains of psychological flexibility and inflexibility as treatment mechanisms in acceptance and commitment therapy: A comprehensive systematic and meta-analytic review (2024) · cites this paper as McShane et al., 2016
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On the other hand, this could also be an overestimate of average study power, as heterogeneity in effect sizes can cause the p-curve to overestimate mean power (Brunner & Schimmack, 2020; McShane et al., 2016).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as McShane et al., 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we ran selection method analyses for each meta-analytic effect using R functions developed and validated by McShane et al. (2016).”
From Examining domains of psychological flexibility and inflexibility as treatment mechanisms in acceptance and commitment therapy: A comprehensive systematic and meta-analytic review (2024) · cites this paper as McShane et al., 2016
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, the credibility of such results is contingent on a lack of selective reporting or on the ability to effectively correct for missing evidence (McShane et al., 2016; Stanley and Doucouliagos, 2014).”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as McShane et al., 2016
The cited source says what the claim attributes to it.
In the citing paper
“In general, cumulative or meta-analytic evidence obtained from multiple independently conducted experiments provides a better basis for assessing the reliability of findings than evidence from a single experiment.”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as McShane et al., 2016
The cited source says what the claim attributes to it.
In the citing paper
“p-uniform does not model heterogeneity, and this can lead it to overestimate the average effect size of all studies (McShane et al., 2016).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as McShane et al., 2016
The cited source says what the claim attributes to it.
In the citing paper
“Bias adjustments can improve performance relative to a meta-analysis that does not adjust for publication bias (Carter et al., 2019; McShane et al., 2016; Moreno et al., 2009; Simonsohn et al., 2014).”
From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as McShane et al., 2016
In the citing paper
“Statisticians have proposed several statistical approaches to assess how robust meta-analyses are to various assumptions about the extent and nature of missing results, including trim-andfill 67 , selection models 68 , and regression-based adjustment methods 69 .”
From Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: a review (2021) · cites this paper as [69]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 2×
“…& Mazumdar, 1994; Egger, Smith, Schneider, & Minder, 1997; Sterne & Egger, 2005), Rosenthal’s fail-safe N method and its many variants (Becker, 2005; Rosenthal, 1979), the trim-and-fill method (Duval, 2005; Duval & Tweedie, 2000a, 2000b), and the Precision-Effect Test– Precision-Effect Estimate…”
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Research works printed but not linked: 4
- Boyle R., 1965, The sceptical chymist
- Ferriar J., 1792, Medical histories and reflection
- Fisher Ronald A, 1938, Sankhya, 4, 14
- Sterling T. D., 1959, Journal of the American Statistical Association, 54, 30
Cited by
Sorted by citations
Salk, R. H., Hyde, J. S., & Abramson, L. Y. (2017) · Psychological Bulletin · doi:10.1037/bul0000102 · Cited by 2,622
Debunking: A Meta-Analysis of the Psychological Efficacy of Messages Countering Misinformation
Chan, M.-P. S., Jones, C. R., Hall Jamieson, K., & Albarracín, D. (2017) · Psychological Science · doi:10.1177/0956797617714579 · Cited by 898
A social comparison theory meta-analysis 60+ years on.
Gerber, J. P., Wheeler, L., & Suls, J. (2018) · Psychological Bulletin · doi:10.1037/bul0000127 · Cited by 808
Self-regulation in childhood as a predictor of future outcomes: A meta-analytic review.
Robson, D. A., Allen, M. S., & Howard, S. J. (2020) · Psychological Bulletin · doi:10.1037/bul0000227 · Cited by 759
Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis
Shrout, P. E., & Rodgers, J. L. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011845 · Cited by 830
Metadata
- Authors
- McShane, B. B., Böckenholt, U., & Hansen, K. T.
- Affiliations
- Northwestern University (US)
- University of California San Diego (US)
- Corresponding author
- McShane, Blakeley B.
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2016-09
- Volume
- 11
- Issue
- 5
- Pages
- 730-749
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2524737933
- Cited by
- 398
- References
- 49
- Citations by year
- 2026: 30 · 2025: 43 · 2024: 43 · 2023: 46 · 2022: 50 · 2021: 46 · 2020: 44 · 2019: 50 · 2018: 23 · 2017: 21 · 2016: 2
- FWCI
- 29.71
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, selection methods, p-curve, meta-analysis, sensitivity analysis, effect size estimation, simulation study
- MeSH terms
- Computer Simulation, Data Interpretation, Statistical, Humans, Meta-Analysis as Topic, Models, Statistical, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
