Comparison of four heterogeneity measures for meta‐analysis
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Rationale, aims, and objectives: Heterogeneity is a critical issue in meta‐analysis, because it implies the appropriateness of combining the collected studies and impacts the reliability of the synthesized results. The Q test is a traditional method to assess heterogeneity; however, because it does not have an intuitive interpretation for clinicians and often has low statistical power, many meta‐analysts alter to use some measures, such as the I2 statistic, to quantify the extent of heterogeneity. This article aims at providing a summary of available tools to assess heterogeneity and comparing their performance. Methods: We reviewed four heterogeneity measures (I2, , , and ) and illustrated how they could be treated as test statistics like the Q statistic. These measures were compared with respect to statistical power based on simulations driven by three real‐data examples. The pairwise agreement among the four measures was also evaluated using Cohen's κ coefficient. Results: Generally, was slightly more powerful than the Q test, while its type I error rate might be slightly inflated. The power of I2 was fairly close to that of Q. The and statistics might have low powers in some cases. Because the differences between the powers of I2, , and Q were often tiny, meta‐analysts might not expect I2 and to yield significant heterogeneity if the Q test failed to do so. In addition, I2 and had fairly good agreement based on the simulated meta‐analyses, but all other pairs of heterogeneity measures generally had poor agreement. Conclusion: The I2 and statistics are recommended for measuring heterogeneity. Meta‐analysts should use the heterogeneity measures as descriptive statistics which have intuitive interpretations from the clinical perspective, instead of determining the significance of heterogeneity simply based on their magnitudes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380
Basics of meta‐analysis: I 2 is not an absolute measure of heterogeneity
Borenstein, M., Higgins, J. P. T., Hedges, L. V., & Rothstein, H. R. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1230 · Cited by 1,848
Meta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709 · cited 1×
“Meta‐analysis has been a popular statistical method to synthesize evidence from different studies in clinical research.(1,2) Assessing heterogeneity between the collected studies in a meta‐analysis plays a critical role in examining whether the studies may be properly combined and the synthesized results are reliable.”
MethodUndue reliance on I 2 in assessing heterogeneity may mislead
Rücker, G., Schwarzer, G., Carpenter, J. R., & Schumacher, M. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-79 · Cited by 1,287
Uncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254
The heterogeneity statistic I2 can be biased in small meta-analyses
von Hippel, P. T. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0024-z · Cited by 1,229
A comparison of heterogeneity variance estimators in simulated random‐effects meta‐analyses
Langan, D., Higgins, J. P. T., Jackson, D., Bowden, J., Veroniki, A. A., Kontopantelis, E., Viechtbauer, W., & Simmonds, M. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1316 · Cited by 1,074 · cited 1×
“Alternative estimates, such as those based on the restricted maximum likelihood or Bayesian analysis, may be better options to be used in the heterogeneity measures.(43,44) Third, as in conventional meta‐analysis methods, the within‐study variances were treated as true values in our analysis and the four heterogeneity…”
MethodSystematic Review: Why sources of heterogeneity in meta-analysis should be investigated
Thompson, S. G. (1994) · BMJ · doi:10.1136/bmj.309.6965.1351 · Cited by 1,017
Commentary: Heterogeneity in meta-analysis should be expected and appropriately quantified
Higgins, J. P. T. (2008) · International Journal of Epidemiology · doi:10.1093/ije/dyn204 · Cited by 995
Turner, R. M., Davey, J., Clarke, M. J., Thompson, S. G., & Higgins, J. P. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dys041 · Cited by 837 · cited 1×
“Alternative estimates, such as those based on the restricted maximum likelihood or Bayesian analysis, may be better options to be used in the heterogeneity measures.(43,44) Third, as in conventional meta‐analysis methods, the within‐study variances were treated as true values in our analysis and the four heterogeneity…”
MethodInterpretation of tests of heterogeneity and bias in meta-analysis
Ioannidis, J. P. A. (2008) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2008.00986.x · Cited by 696
Myung, S.-K., Ju, W., Cho, B., Oh, S.-W., Park, S. M., Koo, B.-K., Park, B.-J., & for the Korean Meta-Analysis (KORMA) Study Group (2013) · BMJ · doi:10.1136/bmj.f10 · Cited by 467 · cited 2×
“1 for those based on the meta‐analysis by Myung et al.(38) The between‐study standard deviation was not further increased because the statistical power would be too high (eg, above 80% or 90%), and the differences between the measures would be minimal.”
MethodInfluentialReasons or excuses for avoiding meta-analysis in forest plots
Ioannidis, J. P. A., Patsopoulos, N. A., & Rothstein, H. R. (2008) · BMJ · doi:10.1136/bmj.a117 · Cited by 457
Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change
Cornell, J. E., Mulrow, C. D., Localio, R., Stack, C. B., Meibohm, A. R., Guallar, E., & Goodman, S. N. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2886 · Cited by 453 · cited 1×
“However, although the method‐of‐moments estimate has been popular so far, it has been found to be inferior in some cases.(42) Alternative estimates, such as those based on the restricted maximum likelihood or Bayesian analysis, may be better options to be used in the heterogeneity measures.”
BackgroundDavey, J., Turner, R. M., Clarke, M. J., & Higgins, J. P. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-160 · Cited by 404
Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis
Takkouche, B., Cadarso-Suarez, C., & Spiegelman, D. (1999) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009981 · Cited by 380 · cited 1×
“To avoid the problems in deriving the measures' exact null distributions, the parametric resampling method can be applied to calculate the P values of the measures, as well as the Q test.(19,32,33)”
MethodMakani, H., Bangalore, S., Desouza, K. A., Shah, A., & Messerli, F. H. (2013) · BMJ · doi:10.1136/bmj.f360 · Cited by 244 · cited 2×
“2% under the setting based on the meta‐analysis by Makani et al.(37) As for statistical power, the I(2) statistic produced nearly the same results with the Q test.”
ResultInfluential
Cited by
Body image concerns in women with polycystic ovary syndrome: a systematic review and meta-analysis
Davitadze, M., Malhotra, K., Khalil, H., Hebbar, M., Tay, C. T., Mousa, A., PCOS SEva team, Robinson, L., Sheikh, J., Melson, E., et al. (2023) · European Journal of Endocrinology · doi:10.1093/ejendo/lvad110 · Cited by 57
Meta‐Analysis of Urban Non‐Point Source Pollution From Road and Roof Runoff Across China
Wang, Y., Li, C., Qiao, J., Hu, Y., Zhang, Q., Yin, J., & Slater, L. (2025) · Earth's Future · doi:10.1029/2024ef005296 · Cited by 22
The role of blood CXCL12 level in prognosis of coronary artery disease: A meta-analysis
Zhang, S., Ding, Y., Feng, F., & Gao, Y. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.938540 · Cited by 13
Wang, Y., Lan, B., Zhang, S., Mu, Y., Mi, J., Yu, J., Zhan, Q., Luo, B., Lin, F., Teng, J., et al. (2025) · Diabetology & Metabolic Syndrome · doi:10.1186/s13098-025-01877-4 · Cited by 8
“I² statistic was used to assess statistical heterogeneity. τ 2 statistic was used to quantify the degree of heterogeneity [25, 26].”
Metadata
- Authors
- Lin, L.
- Venue
- Journal of Evaluation in Clinical Practice
- Publisher
- Wiley
- Published
- 2019-06-24
- Volume
- 26
- Issue
- 1
- Pages
- 376-384
- ISSN
- 1356-1294 (ISSN-L)
- 1365-2753
- OpenAlex
- W2955544044
- Cited by
- 184
- References
- 17
- FWCI
- 4.51
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.