CoChat
Sign Up
Cited by 398Meta-analysis and systematic reviewsPsychology Research and Bibliometricsscientometrics and bibliometrics research

Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes

Open access
McShane, Blakeley B., Böckenholt, Ulf, Hansen, Karsten T.·2016·Perspectives on Psychological Science
DOI 10.1177/1745691616662243
Open this paper in CoChat — free accountOpen PDF

Abstract

We review and evaluate selection methods, a prominent class of techniques first proposed by Hedges (1984) that assess and adjust for publication bias in meta-analysis, via an extensive simulation study. Our simulation covers both restrictive settings as well as more realistic settings and proceeds across multiple metrics that assess different aspects of model performance. This evaluation is timely in light of two recently proposed approaches, the so-called p-curve and p-uniform approaches, that can be viewed as alternative implementations of the original Hedges selection method approach. We find that the p-curve and p-uniform approaches perform reasonably well but not as well as the original Hedges approach in the restrictive setting for which all three were designed. We also find they perform poorly in more realistic settings, whereas variants of the Hedges approach perform well. We conclude by urging caution in the application of selection methods: Given the idealistic model assumptions underlying selection methods and the sensitivity of population average effect size estimates to them, we advocate that selection methods should be used less for obtaining a single estimate that purports to adjust for publication bias ex post and more for sensitivity analysis—that is, exploring the range of estimates that result from assuming different forms of and severity of publication bias.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/1745691616662243

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The relative probabilities estimated by selection method analyses also suggest a general lack of publication bias.

    “Consistent with those findings, the relative probabilities estimated by selection method analyses (e.g., McShane et al., 2016) also suggest a general lack of publication bias.”

    From Examining domains of psychological flexibility and inflexibility as treatment mechanisms in acceptance and commitment therapy: A comprehensive systematic and meta-analytic review (2024) · cites this paper as McShane et al., 2016

    The Early selection methods section explains how selection methods estimate relative publication likelihoods. But the Modeling Considerations section says statistically significant results are overrepresented in the published literature, contrary to the claimed general lack of publication bias. The paper does not report relative-probability estimates supporting that conclusion. (LLM, conf=0.84)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • The p-curve estimate of average study power may be an overestimate because heterogeneity in effect sizes can cause the p-curve to overestimate mean power.

    “On the other hand, this could also be an overestimate of average study power, as heterogeneity in effect sizes can cause the p-curve to overestimate mean power (Brunner & Schimmack, 2020; McShane et al., 2016).”

    From Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies (2023) · cites this paper as McShane et al., 2016

    The Modeling Considerations and Summary sections say that heterogeneity can make p-curve estimates upwardly biased. They identify the biased quantity as the population average effect size, however, not average study power; the copy does not establish that p-curve overestimates mean power. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Selection method analyses were run for each meta-analytic effect using R functions developed and validated by McShane et al. (2016).

    “we ran selection method analyses for each meta-analytic effect using R functions developed and validated by McShane et al. (2016).”

    From Examining domains of psychological flexibility and inflexibility as treatment mechanisms in acceptance and commitment therapy: A comprehensive systematic and meta-analytic review (2024) · cites this paper as McShane et al., 2016

    In the Method p-Curve p-Uniform Selection section (page unavailable), the authors say they provide their code and a simple example in the Supplemental Material. The paper evaluates selection methods through simulations, but our copy does not establish that McShane et al. developed and validated the particular R functions used by the citing authors; it identifies the R package “weightr” as Coburn and Vevea’s implementation. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The credibility of cumulative or meta-analytic results is contingent on a lack of selective reporting or on the ability to effectively correct for missing evidence.

    “However, the credibility of such results is contingent on a lack of selective reporting or on the ability to effectively correct for missing evidence (McShane et al., 2016; Stanley and Doucouliagos, 2014).”

    From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as McShane et al., 2016

  • Cumulative or meta-analytic evidence from multiple independently conducted experiments provides a better basis for assessing reliability of findings than evidence from a single experiment.

    “In general, cumulative or meta-analytic evidence obtained from multiple independently conducted experiments provides a better basis for assessing the reliability of findings than evidence from a single experiment.”

    From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as McShane et al., 2016

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Editors. (1909). The reporting of unsuccessful cases [Editorial]. The Boston Medical and Surgical Journal, 161, 263-264.
  2. Fisher, Ronald A (1938). Presidential address, First Indian Statistical Conference. Sankhya, 4, 14-17.

Cited by

Sorted by citations

Related

Authors
McShane, B. B., Böckenholt, U., & Hansen, K. T.
Affiliations
  • Northwestern University (US)
  • University of California San Diego (US)
Corresponding author
McShane, Blakeley B.
ORCID
Venue
Perspectives on Psychological Science
Publisher
SAGE Publications
Published
2016-09
Volume
11
Issue
5
Pages
730-749
ISSN
  • 1745-6916 (ISSN-L)
  • 1745-6924
DOI
10.1177/1745691616662243
OpenAlex
W2524737933
Cited by
398
References
49
Citations by year
2026: 30 · 2025: 43 · 2024: 43 · 2023: 46 · 2022: 50 · 2021: 46 · 2020: 44 · 2019: 50 · 2018: 23 · 2017: 21 · 2016: 2
FWCI
29.71
Type
journal article
Language
en
Open access
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
  • Perspectives on Psychological Science (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
Keywords
publication bias, selection methods, p-curve, meta-analysis, sensitivity analysis, effect size estimation, simulation study
MeSH terms
Computer Simulation, Data Interpretation, Statistical, Humans, Meta-Analysis as Topic, Models, Statistical, Publication Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review