CoChat
Sign Up
Cited by 294Meta-analysis and systematic reviewsStatistical Methods in EpidemiologyPsychology Research and Bibliometrics

Dealing with effect size multiplicity in systematic reviews and meta‐analyses

Open access
López‐López, José A., Page, Matthew J., Lipsey, Mark W., Higgins, Julian P.T.·2018·Research Synthesis Methods
DOI 10.1002/jrsm.1310
Open this paper in CoChat — free account

Abstract

Systematic reviews often encounter primary studies that report multiple effect sizes based on data from the same participants. These have the potential to introduce statistical dependency into the meta‐analytic data set. In this paper, we provide a tutorial on dealing with effect size multiplicity within studies in the context of meta‐analyses of intervention and association studies, recommending a three‐step approach. The first step is to define the research question and consider the extent to which it mainly reflects interest in mean effect sizes (which we term a convergent approach) or an interest in exploring heterogeneity (which we term a divergent approach). A second step is to identify the types of multiplicities that appear in the initial database of effect sizes relevant to the research question, and we propose a categorization scheme to differentiate them. The third step is to select a strategy for dealing with each type of multiplicity. The researcher can choose between a reductionist meta‐analytic approach, which is characterized by inclusion of a single effect size per study, and an integrative approach, characterized by inclusion of multiple effect sizes per study. We present an overview of available analysis strategies for dealing with effect size multiplicity within studies and provide recommendations intended to help researchers decide which strategy might be preferable in particular situations. Last, we offer caveats and cautions about addressing the challenges multiplicity poses for systematic reviews and meta‐analyses.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/jrsm.1310

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Multiple results compatible with an outcome domain in a study might arise when study investigators report results based on multiple analysis populations, methods for handling missing data, or methods for handling confounding.

    “Multiple results compatible with an outcome domain in a study might also arise when study investigators report results based on multiple analysis populations (such as all participants randomised, all participants receiving a specific amount of treatment), methods for handling missing data (such as multiple imputation, last-observation-carried-forward), or methods for handling confounding (such as”

    From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [69]

    The “Time points” passage (page unavailable) identifies multiple results from intention-to-treat versus completer populations and from unadjusted versus confounder-adjusted analyses. The cited passages do not address multiplicity arising from different methods for handling missing data, such as multiple imputation versus last-observation-carried-forward. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Although robust variance estimation was used to address multiplicity and statistical dependencies, this method might raise issues of disproportionate influence from studies that contribute a large number of effect sizes relative to those that contribute fewer, particularly when the overall number of studies and effect sizes is relatively small.

    “Although we used robust variance estimation to address multiplicity and statistical dependencies, this method might raise issues of disproportionate influence from studies that contribute a large number of effect sizes relative to those that contribute fewer, and particularly when the overall number of studies and effect sizes is relatively small [71].”

    From Parenting interventions to promote early child development in the first three years of life: A global systematic review and meta-analysis (2021) · cites this paper as [71]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Reviewers might seek all results that were compatible with each outcome definition from each study or use a process to select a subset of the results.

    “Reviewers might seek all results that were compatible with each outcome definition from each study or use a process to select a subset of the results.65 69”

    From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [69]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. [24]24. Hedges LV. Meta-analysis. In: Rao CR, Sinharay S, editors. Handbook of Statistics. 26. Amsterdam: Elsevier; 2007. p. 919-53.
  2. [56]56. López-López JA, Van den Noortgate W, Tanner-Smith EE, Wilson SJ, Lipsey MW. Assessing meta-regression methods for examining moderator relationships with dependent effect sizes: A Monte Carlo simulation. Res Synth Methods 2017;8(4):435-50.

Cited by

Sorted by citations

Related

Authors
López‐López, J. A., Page, M. J., Lipsey, M. W., & Higgins, J. P. T.
Affiliations
  • Department of Population Health Sciences, Bristol Medical School University of Bristol Bristol UK
  • University of Bristol (GB)
  • Vanderbilt University (US)
ORCID
Venue
Research Synthesis Methods
Publisher
Wiley
Published
2018-07-27
Volume
9
Issue
3
Pages
336-351
ISSN
  • 1759-2879 (ISSN-L)
  • 1759-2887
DOI
10.1002/jrsm.1310
OpenAlex
W2810076098
Semantic Scholar
9fdae62b538b5bd9d4a8096441d7f025633479cd
Cited by
294
References
55
Citations by year
2026: 42 · 2025: 52 · 2024: 52 · 2023: 51 · 2022: 43 · 2021: 29 · 2020: 13 · 2019: 11 · 2018: 1
FWCI
2.67
Type
journal article
Language
en
Open access
Licence
Locations
  • Research Synthesis Methods (publishedVersion)
  • PubMed (publishedVersion)
  • Bristol Research (University of Bristol) (submittedVersion, other-oa)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
statistical dependence, heterogeneity, research question formulation, intervention research, association studies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review