Comparison of four heterogeneity measures for meta‐analysis
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Rationale, aims, and objectives: Heterogeneity is a critical issue in meta‐analysis, because it implies the appropriateness of combining the collected studies and impacts the reliability of the synthesized results. The Q test is a traditional method to assess heterogeneity; however, because it does not have an intuitive interpretation for clinicians and often has low statistical power, many meta‐analysts alter to use some measures, such as the I2 statistic, to quantify the extent of heterogeneity. This article aims at providing a summary of available tools to assess heterogeneity and comparing their performance. Methods: We reviewed four heterogeneity measures (I2, , , and ) and illustrated how they could be treated as test statistics like the Q statistic. These measures were compared with respect to statistical power based on simulations driven by three real‐data examples. The pairwise agreement among the four measures was also evaluated using Cohen's κ coefficient. Results: Generally, was slightly more powerful than the Q test, while its type I error rate might be slightly inflated. The power of I2 was fairly close to that of Q. The and statistics might have low powers in some cases. Because the differences between the powers of I2, , and Q were often tiny, meta‐analysts might not expect I2 and to yield significant heterogeneity if the Q test failed to do so. In addition, I2 and had fairly good agreement based on the simulated meta‐analyses, but all other pairs of heterogeneity measures generally had poor agreement. Conclusion: The I2 and statistics are recommended for measuring heterogeneity. Meta‐analysts should use the heterogeneity measures as descriptive statistics which have intuitive interpretations from the clinical perspective, instead of determining the significance of heterogeneity simply based on their magnitudes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380
Basics of meta‐analysis: I 2 is not an absolute measure of heterogeneity
Borenstein, M., Higgins, J. P. T., Hedges, L. V., & Rothstein, H. R. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1230 · Cited by 1,848
Meta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709 · cited 1×
“Meta‐analysis has been a popular statistical method to synthesize evidence from different studies in clinical research.(1,2) Assessing heterogeneity between the collected studies in a meta‐analysis plays a critical role in examining whether the studies may be properly combined and the synthesized results are reliable.”
MethodUndue reliance on I 2 in assessing heterogeneity may mislead
Rücker, G., Schwarzer, G., Carpenter, J. R., & Schumacher, M. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-79 · Cited by 1,287
Uncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254
The heterogeneity statistic I2 can be biased in small meta-analyses
von Hippel, P. T. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0024-z · Cited by 1,229
A comparison of heterogeneity variance estimators in simulated random‐effects meta‐analyses
Langan, D., Higgins, J. P. T., Jackson, D., Bowden, J., Veroniki, A. A., Kontopantelis, E., Viechtbauer, W., & Simmonds, M. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1316 · Cited by 1,074 · cited 1×
“Alternative estimates, such as those based on the restricted maximum likelihood or Bayesian analysis, may be better options to be used in the heterogeneity measures.(43,44) Third, as in conventional meta‐analysis methods, the within‐study variances were treated as true values in our analysis and the four heterogeneity…”
MethodSystematic Review: Why sources of heterogeneity in meta-analysis should be investigated
Thompson, S. G. (1994) · BMJ · doi:10.1136/bmj.309.6965.1351 · Cited by 1,017
Commentary: Heterogeneity in meta-analysis should be expected and appropriately quantified
Higgins, J. P. T. (2008) · International Journal of Epidemiology · doi:10.1093/ije/dyn204 · Cited by 995
Turner, R. M., Davey, J., Clarke, M. J., Thompson, S. G., & Higgins, J. P. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dys041 · Cited by 837 · cited 1×
“Alternative estimates, such as those based on the restricted maximum likelihood or Bayesian analysis, may be better options to be used in the heterogeneity measures.(43,44) Third, as in conventional meta‐analysis methods, the within‐study variances were treated as true values in our analysis and the four heterogeneity…”
Method
Cited by
Body image concerns in women with polycystic ovary syndrome: a systematic review and meta-analysis
Davitadze, M., Malhotra, K., Khalil, H., Hebbar, M., Tay, C. T., Mousa, A., PCOS SEva team, Robinson, L., Sheikh, J., Melson, E., et al. (2023) · European Journal of Endocrinology · doi:10.1093/ejendo/lvad110 · Cited by 57
Meta‐Analysis of Urban Non‐Point Source Pollution From Road and Roof Runoff Across China
Wang, Y., Li, C., Qiao, J., Hu, Y., Zhang, Q., Yin, J., & Slater, L. (2025) · Earth's Future · doi:10.1029/2024ef005296 · Cited by 22
The role of blood CXCL12 level in prognosis of coronary artery disease: A meta-analysis
Zhang, S., Ding, Y., Feng, F., & Gao, Y. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.938540 · Cited by 13
Wang, Y., Lan, B., Zhang, S., Mu, Y., Mi, J., Yu, J., Zhan, Q., Luo, B., Lin, F., Teng, J., et al. (2025) · Diabetology & Metabolic Syndrome · doi:10.1186/s13098-025-01877-4 · Cited by 8
“I² statistic was used to assess statistical heterogeneity. τ 2 statistic was used to quantify the degree of heterogeneity [25, 26].”
Metadata
- Authors
- Lin, L.
- Venue
- Journal of Evaluation in Clinical Practice
- Publisher
- Wiley
- Published
- 2019-06-24
- Volume
- 26
- Issue
- 1
- Pages
- 376-384
- ISSN
- 1356-1294 (ISSN-L)
- 1365-2753
- OpenAlex
- W2955544044
- Cited by
- 184
- References
- 17
- FWCI
- 4.51
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Comparison of four heterogeneity measures for meta‐analysis. https://app.cochat.ai/papers/10.1111/jep.13159
- The paper
- Lin, L. (2020). Comparison of four heterogeneity measures for meta‐analysis. Journal of Evaluation in Clinical Practice. https://doi.org/10.1111/jep.13159
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.