The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
A comparison of heterogeneity variance estimators in simulated random‐effects meta‐analyses
Open accessAbstract
Studies combined in a meta‐analysis often have differences in their design and conduct that can lead to heterogeneous results. A random‐effects model accounts for these differences in the underlying study effects, which includes a heterogeneity variance parameter. The DerSimonian‐Laird method is often used to estimate the heterogeneity variance, but simulation studies have found the method can be biased and other methods are available. This paper compares the properties of nine different heterogeneity variance estimators using simulated meta‐analysis data. Simulated scenarios include studies of equal size and of moderate and large differences in size. Results confirm that the DerSimonian‐Laird estimator is negatively biased in scenarios with small studies and in scenarios with a rare binary outcome. Results also show the Paule‐Mandel method has considerable positive bias in meta‐analyses with large differences in study size. We recommend the method of restricted maximum likelihood (REML) to estimate the heterogeneity variance over other methods. However, considering that meta‐analyses of health studies typically contain few studies, the heterogeneity variance estimate should not be used as a reliable gauge for the extent of heterogeneity in a meta‐analysis. The estimated summary effect of the meta‐analysis and its confidence interval derived from the Hartung‐Knapp‐Sidik‐Jonkman method are more robust to changes in the heterogeneity variance estimate and show minimal deviation from the nominal coverage of 95% under most of our simulated scenarios.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Meta-analysis was conducted using a random-effect model with restricted maximum likelihood, as the estimator of true heterogeneity, which seems to provide us with less biased estimates than other methods.33”
From Psychological interventions for depression and anxiety: a systematic review and meta-analysis of Iranian chronic pain trials. (2020) · cites this paper as [33]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For random-effects models, many methods are available, and their performance has been shown to differ depending on the characteristics of the meta-analysis (such as the number and size of the included studies113 114).”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [114]
The cited source says what the claim attributes to it.
In the citing paper
“a random-effects model typically cannot adequately estimate heterogeneity with only two effects (Langan et al., 2019)”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Langan et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“This method assumes a t-distribution of the pooled effect rather than a normal distribution, which reduces the chance of obtaining false-positive results (Langan et al., 2019).”
From The state of hiring discrimination: A meta-analysis of (almost) all recent correspondence experiments (2023) · cites this paper as Langan et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Self-citationConducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Self-citationBorenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Self-citationBorenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
Self-citationHunter, J., & Schmidt, F. (2004) · doi:10.4135/9781412985031 · Cited by 3,023
Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems
Harville, D. A. (1977) · Journal of the American Statistical Association · doi:10.1080/01621459.1977.10480998 · Cited by 2,667
IntHout, J., Ioannidis, J. P., & Borm, G. F. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-25 · Cited by 2,419
Random-effects model for meta-analysis of clinical trials: An update
DerSimonian, R., & Kacker, R. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.04.004 · Cited by 2,374
Improved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075
What to add to nothing? Use and avoidance of continuity corrections in meta‐analysis of sparse data
J. Sweeting, M., J. Sutton, A., & C. Lambert, P. (2004) · Statistics in Medicine · doi:10.1002/sim.1761 · Cited by 1,734
Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699 · cited 3×
“The package metafor 226 (Viechtbauer, 2010) was used to run simulated meta-analyses and calculate heterogeneity 227 variance estimates from methods coded in this package, bespoke code was used for those that 228 are not.”
MethodInfluentialSelf-citationMethods to estimate the between‐study variance and its uncertainty in meta‐analysis
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1164 · Cited by 1,685
Self-citationParametric empirical Bayes inference: theory and applications
Morris, C. N. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477920 · Cited by 1,448
Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events
Bradburn, M. J., Deeks, J. J., Berlin, J. A., & Russell Localio, A. (2007) · Statistics in Medicine · doi:10.1002/sim.2528 · Cited by 938
A refined method for the meta-analysis of controlled clinical trials with binary outcome
Hartung, J., & Knapp, G. (2001) · Statistics in Medicine · doi:10.1002/sim.1009 · Cited by 775
Stijnen, T., Hamza, T. H., & Özdemir, P. (2010) · Statistics in Medicine · doi:10.1002/sim.4040 · Cited by 757
Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems
Harville, D. A. (1977) · Journal of the American Statistical Association · doi:10.2307/2286796 · Cited by 595
Consensus Values and Weighting Factors
Paule, R. C., & Mandel, J. (1982) · Journal of Research of the National Bureau of Standards · doi:10.6028/jres.087.022 · Cited by 580
Bayesian approaches to random-effects meta-analysis: a comparative study
Smith, T. C., Spiegelhalter, D. J., & Thomas, A. (1995) · Statistics in Medicine · doi:10.1002/sim.4780142408 · Cited by 568
Bowden, J., Tierney, J. F., Copas, A. J., & Burdett, S. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-41 · Cited by 555
Self-citationRöver, C., Knapp, G., & Friede, T. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0091-1 · Cited by 527
A Re-Analysis of the Cochrane Library Data: The Dangers of Unobserved Heterogeneity in Meta-Analyses
Kontopantelis, E., Springate, D. A., & Reeves, D. (2013) · PLoS ONE · doi:10.1371/journal.pone.0069930 · Cited by 525
Self-citationA LIKELIHOOD APPROACH TO META-ANALYSIS WITH RANDOM EFFECTS
Hardy, R., & Thompson, S. G. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19960330)15:6 · Cited by 522
Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change
Cornell, J. E., Mulrow, C. D., Localio, R., Stack, C. B., Meibohm, A. R., Guallar, E., & Goodman, S. N. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2886 · Cited by 453 · cited 1×
“Other methods include the profile likelihood method,(21) which has also been shown as a better alternative to the Wald method in simulated meta‐analysis data(42) and recommended elsewhere.(43) Second, a continuity correction of 0.”
MethodA simple confidence interval for meta-analysis
Sidik, K., & Jonkman, J. N. (2002) · Statistics in Medicine · doi:10.1002/sim.1262 · Cited by 431 · cited 1×
“This method is equivalent to the t-distribution method above, but its variance is multiplied by 217 a scaling factor ∑ ai k i=1 (θi − θ) 2 (k − 1) ⁄ (Sidik and Jonkman, 2002; Wiksten et al., 2016).”
MethodDavey, J., Turner, R. M., Clarke, M. J., & Higgins, J. P. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-160 · Cited by 404
Self-citationA comparison of heterogeneity variance estimators in combining results of studies
Sidik, K., & Jonkman, J. N. (2007) · Statistics in Medicine · doi:10.1002/sim.2688 · Cited by 380
Kontopantelis, E., & Reeves, D. (2010) · Statistical Methods in Medical Research · doi:10.1177/0962280210392008 · Cited by 264
Self-citationSimple Heterogeneity Variance Estimation for Meta-Analysis
Sidik, K., & Jonkman, J. N. (2005) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/j.1467-9876.2005.00489.x · Cited by 254
Confidence intervals for the overall effect size in random-effects meta-analysis.
Sánchez‐Meca, J., & Marín‐Martínez, F. (2008) · Psychological Methods · doi:10.1037/1082-989x.13.1.31 · Cited by 242
A comparison of seven random‐effects models for meta‐analyses that estimate the summary odds ratio
Jackson, D., Law, M. W.-M., Stijnen, T., Viechtbauer, W., & White, I. R. (2018) · Statistics in Medicine · doi:10.1002/sim.7588 · Cited by 209
Self-citationConfidence intervals for random effects meta-analysis and robustness to publication bias
Henmi, M., & Copas, J. B. (2010) · Statistics in Medicine · doi:10.1002/sim.4029 · Cited by 167
Metaan: Random-effects Meta-analysis
Kontopantelis, E., & Reeves, D. (2010) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x1001000307 · Cited by 150
Self-citationLangan, D., Higgins, J. P. T., & Simmonds, M. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1198 · Cited by 149
Self-citationMeta-Analysis of Rare Binary Adverse Event Data
Bhaumik, D. K., Amatya, A. K., Normand, S. T., Greenhouse, J. B., Kaizar, E. E., Neelon, B., & Gibbons, R. D. (2012) · Journal of the American Statistical Association · doi:10.1080/01621459.2012.664484 · Cited by 136
A decade of individual participant data meta-analyses: A review of current practice
Simmonds, M. C., Stewart, G. B., & Stewart, L. A. (2015) · Contemporary Clinical Trials · doi:10.1016/j.cct.2015.06.012 · Cited by 131
Self-citationValid Inference in Random Effects Meta‐Analysis
Follmann, D. A., & Proschan, M. A. (1999) · Biometrics · doi:10.1111/j.0006-341x.1999.00732.x · Cited by 128
Higgins, J. P. T., & Spiegelhalter, D. J. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.96 · Cited by 123
Self-citationSome general points in estimating heterogeneity variance with the DerSimonian-Laird estimator
Böhning, D. (2002) · Biostatistics · doi:10.1093/biostatistics/3.4.445 · Cited by 122
Kontopantelis, E., & Reeves, D. J. (2012) · Statistical Methods in Medical Research · doi:10.1177/0962280211413451 · Cited by 117
Self-citationA general framework for the use of logistic regression models in meta-analysis
Simmonds, M. C., & Higgins, J. P. T. (2014) · Statistical Methods in Medical Research · doi:10.1177/0962280214534409 · Cited by 109
Self-citationAvoiding zero between‐study variance estimates in random‐effects meta‐analysis
Chung, Y., Rabe‐Hesketh, S., & Choi, I. (2013) · Statistics in Medicine · doi:10.1002/sim.5821 · Cited by 109
Malzahn, U. (2000) · Biometrika · doi:10.1093/biomet/87.3.619 · Cited by 84
Hartung–Knapp method is not always conservative compared with fixed‐effect meta‐analysis
Wiksten, A. S., Rücker, G., & Schwarzer, G. (2016) · Statistics in Medicine · doi:10.1002/sim.6879 · Cited by 78
Estimation of between-trial variance in sequential meta-analyses: A simulation study
Novianti, P. W., Roes, K. C. B., & van der Tweel, I. (2013) · Contemporary Clinical Trials · doi:10.1016/j.cct.2013.11.012 · Cited by 71
Estimating Heterogeneity Variance in Meta-Analysis
Rukhin, A. L. (2012) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2012.01047.x · Cited by 70
Restricted maximum likelihood estimation of a common mean and the Mandel–Paule algorithm
Rukhin, A. L., Biggerstaff, B. J., & Vangel, M. G. (2000) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(99)00098-1 · Cited by 65
An empirical comparison of heterogeneity variance estimators in 12 894 meta-analyses
Langan, D., Higgins, J. P. T., & Simmonds, M. C. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1140 · Cited by 51
Self-citationJackson, D., Turner, R., Rhodes, K. M., & Viechtbauer, W. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-103 · Cited by 50
Self-citationReducing the Number of Unjustified Significant Results in Meta-analysis
Hartung, J. G., & Makambi, K. H. (2003) · Communications in Statistics - Simulation and Computation · doi:10.1081/sac-120023884 · Cited by 39
On Estimating Residual Heterogeneity in Random-Effects Meta-Regression: A Comparative Study
Panityakul, T., Bumrungsup, C., & Knapp, G. (2013) · Journal of Statistical Theory and Applications · doi:10.2991/jsta.2013.12.3.4 · Cited by 37
Flagged first, then by citations
Cited by
Sorted by citations
Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
Analysing data and undertaking meta‐analyses
Deeks, J. J., Higgins, J. P., Altman, D. G., & on behalf of the Cochrane Statistical Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch10 · Cited by 2,672
Self-citationAnalysing Data and Undertaking Meta‐Analyses
Deeks, J. J., Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch9 · Cited by 1,800
Self-citationPage, M. J., Moher, D., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/gwdhk · Cited by 916
Estimating the sample mean and standard deviation from commonly reported quantiles in meta-analysis
McGrath, S., Zhao, X., Steele, R. J., Thombs, B. D., Benedetti, A., Levis, B., Riehm, K. E., Saadat, N., Levis, A. W., Azar, M., et al. (2020) · Statistical Methods in Medical Research · doi:10.1177/0962280219889080 · Cited by 869
“The restricted maximum likelihood method was used to estimate heterogeneity in all meta-analyses.(30) The QE and BC methods were less biased for estimating the pooled mean compared to the 2528 Statistical Methods in Medical Research 29(9)”
Method
Metadata
- Authors
- Langan, D., Higgins, J. P. T., Jackson, D., Bowden, J., Veroniki, A. A., Kontopantelis, E., Viechtbauer, W., & Simmonds, M.
- Affiliations
- University of Manchester (GB)
- University of Bristol (GB)
- AstraZeneca (United Kingdom) (GB)
- St. Michael's Hospital (CA)
- Maastricht University (NL)
- University of York (GB)
- Corresponding author
- Langan, Dean
- ORCID
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2018-09-06
- Volume
- 10
- Issue
- 1
- Pages
- 83-98
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2885675882
- Semantic Scholar
- 264ababb39dc4b8737425a30c9fb75c5d8eb2bf4
- Cited by
- 1,074
- References
- 51
- Citations by year
- 2026: 214 · 2025: 173 · 2024: 198 · 2023: 144 · 2022: 140 · 2021: 113 · 2020: 60 · 2019: 28 · 2018: 2
- FWCI
- 38.65
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Research Synthesis Methods (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Bristol Research (University of Bristol) (submittedVersion, other-oa)
- UCL Discovery (University College London) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Funders
- National Institute for Health and Care Research (NF-SI-0617-10145)
- Medical Research Council (MC_UU_00011/2, MR/K006665/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- random-effects meta-analysis, estimation bias, binary outcomes, confidence interval coverage
- MeSH terms
- Systematic Reviews as Topic, Algorithms, Analysis of Variance, Computer Simulation, Data Interpretation, Statistical, Humans, Probability, Research Design, Software, Meta-Analysis as Topic, Reproducibility of Results, Models, Statistical, Selection Bias, Likelihood Functions, Odds Ratio
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
