CoChat
Sign Up
Cited by 1,074Meta-analysis and systematic reviewsStatistical Methods in EpidemiologyStatistical Methods and Bayesian Inference

A comparison of heterogeneity variance estimators in simulated random‐effects meta‐analyses

Open access
Langan, Dean, Higgins, Julian P.T., Jackson, Dan, Bowden, Jack, Veroniki, Areti Angeliki, Kontopantelis, Evangelos, Viechtbauer, Wolfgang, Simmonds, Mark·2019·Research Synthesis Methods
DOI 10.1002/jrsm.1316
Open this paper in CoChat — free accountOpen PDF

Abstract

Studies combined in a meta‐analysis often have differences in their design and conduct that can lead to heterogeneous results. A random‐effects model accounts for these differences in the underlying study effects, which includes a heterogeneity variance parameter. The DerSimonian‐Laird method is often used to estimate the heterogeneity variance, but simulation studies have found the method can be biased and other methods are available. This paper compares the properties of nine different heterogeneity variance estimators using simulated meta‐analysis data. Simulated scenarios include studies of equal size and of moderate and large differences in size. Results confirm that the DerSimonian‐Laird estimator is negatively biased in scenarios with small studies and in scenarios with a rare binary outcome. Results also show the Paule‐Mandel method has considerable positive bias in meta‐analyses with large differences in study size. We recommend the method of restricted maximum likelihood (REML) to estimate the heterogeneity variance over other methods. However, considering that meta‐analyses of health studies typically contain few studies, the heterogeneity variance estimate should not be used as a reliable gauge for the extent of heterogeneity in a meta‐analysis. The estimated summary effect of the meta‐analysis and its confidence interval derived from the Hartung‐Knapp‐Sidik‐Jonkman method are more robust to changes in the heterogeneity variance estimate and show minimal deviation from the nominal coverage of 95% under most of our simulated scenarios.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/jrsm.1316

Cites

52 of 52 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The random-effect model with restricted maximum likelihood provides less biased estimates than other methods.

    “Meta-analysis was conducted using a random-effect model with restricted maximum likelihood, as the estimator of true heterogeneity, which seems to provide us with less biased estimates than other methods.33”

    From Psychological interventions for depression and anxiety: a systematic review and meta-analysis of Iranian chronic pain trials. (2020) · cites this paper as [33]

    The REML results (p. 13) report relatively low bias in heterogeneity-variance estimates and less negative bias than DerSimonian–Laird in some scenarios. The Discussion (p. 16) describes REML as having properties similar to another well-performing estimator, so the source does not establish that it is less biased than other methods generally. It also reports similar bias across estimators for summary-effect estimates with rare events (p. 13). (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • For random-effects models, many methods are available, and their performance has been shown to differ depending on the characteristics of the meta-analysis, such as the number and size of the included studies.

    “For random-effects models, many methods are available, and their performance has been shown to differ depending on the characteristics of the meta-analysis (such as the number and size of the included studies113 114).”

    From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [114]

  • A random-effects model typically cannot adequately estimate heterogeneity with only two effects.

    “a random-effects model typically cannot adequately estimate heterogeneity with only two effects (Langan et al., 2019)”

    From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Langan et al., 2019

  • The Knapp-Hartung method assumes a t-distribution of the pooled effect rather than a normal distribution, which reduces the chance of obtaining false-positive results.

    “This method assumes a t-distribution of the pooled effect rather than a normal distribution, which reduces the chance of obtaining false-positive results (Langan et al., 2019).”

    From The state of hiring discrimination: A meta-analysis of (almost) all recent correspondence experiments (2023) · cites this paper as Langan et al., 2019

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. [27]27. Viechtbauer W. Conducting meta-analyses in R with the metaphor package. J Stat Softw. 2010; 36(3): 1-48.

Cited by

Sorted by citations

Related

Authors
Langan, D., Higgins, J. P. T., Jackson, D., Bowden, J., Veroniki, A. A., Kontopantelis, E., Viechtbauer, W., & Simmonds, M.
Affiliations
  • University of Manchester (GB)
  • University of Bristol (GB)
  • AstraZeneca (United Kingdom) (GB)
  • St. Michael's Hospital (CA)
  • Maastricht University (NL)
  • University of York (GB)
Corresponding author
Langan, Dean
ORCID
Venue
Research Synthesis Methods
Publisher
Wiley
Published
2018-09-06
Volume
10
Issue
1
Pages
83-98
ISSN
  • 1759-2879 (ISSN-L)
  • 1759-2887
DOI
10.1002/jrsm.1316
OpenAlex
W2885675882
Semantic Scholar
264ababb39dc4b8737425a30c9fb75c5d8eb2bf4
Cited by
1,074
References
51
Citations by year
2026: 214 · 2025: 173 · 2024: 198 · 2023: 144 · 2022: 140 · 2021: 113 · 2020: 60 · 2019: 28 · 2018: 2
FWCI
38.65
Type
journal article
Language
en
Open access
Licence
onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations
  • Research Synthesis Methods (publishedVersion)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, other-oa)
  • Research Explorer (The University of Manchester) (publishedVersion, other-oa)
  • White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
  • Bristol Research (University of Bristol) (submittedVersion, other-oa)
  • UCL Discovery (University College London) (submittedVersion)
  • Research Publications (Maastricht University) (submittedVersion, other-oa)
Funders
  • National Institute for Health and Care Research (NF-SI-0617-10145)
  • Medical Research Council (MC_UU_00011/2, MR/K006665/1)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Keywords
random-effects meta-analysis, estimation bias, binary outcomes, confidence interval coverage
MeSH terms
Systematic Reviews as Topic, Algorithms, Analysis of Variance, Computer Simulation, Data Interpretation, Statistical, Humans, Probability, Research Design, Software, Meta-Analysis as Topic, Reproducibility of Results, Models, Statistical, Selection Bias, Likelihood Functions, Odds Ratio

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review