Heterogeneous views on heterogeneity
Open accessAbstract
The insightful and stimulating commentary by Julian Higgins1 on our paper2 raises several important issues that need to be clarified. First, we need to agree on nomenclature. The heterogeneity literature has been plagued by inconsistent terminology. Terms like ‘heterogeneity’, ‘inconsistency’, ‘variation’, ‘diversity’, ‘between-study variance’, ‘variability’, etc. are used interchangeably. While Higgins prefers the term ‘inconsistency’ for I2, in other writings he has used the words ‘variability’ and ‘heterogeneity’ in association with this measure.3 We believe that the term ‘heterogeneity’ is a nice word with roots going back to ETEPOΓENHΣ of Aristotle and ETEPOΓENΩΣ of Sextus Empiricus. It can be applied to any of the popular metrics and tests, but then one simply has to specify which metric or test is exactly alluded to. ‘Inconsistency’ is also a nice, more recent word, but again we need to clarify what it refers to each time. Higgins worries about ‘the post hoc hypotheses that need to be thought up to explain why the excluded studies might be outlying or influential’. We were clear cut in our paper that this is indeed not an easy task. We believe that sensitivity analyses, as currently performed, are usually an invitation to post hoc data dredging with few or no rules in the game. This reduces their inferential reliability. However, this is a major reason why our proposed algorithms may offer one way to improve this free-lunch situation. There are two components to any sensitivity analysis. The first component is how it is done. The second component is how the results are interpreted. We argue that our method takes away much of the subjectivity in the first component. We do not wish to diminish the uncertainty that arises in the second component, and we wish that all meta-analysts recognize and acknowledge this uncertainty properly. Higgins questions whether it is sensible to define a ‘desired’ threshold in terms of I2 statistic. Although we agree that indeed ‘(some) heterogeneity is to be expected in (almost any) meta-analysis’ and ‘any amount of heterogeneity is acceptable, providing both that the predefined eligibility criteria for the meta-analysis are sound and that the data are correct’, we believe that using thresholds to describe heterogeneity is an unavoidable consequence of the effort to translate statistical terms into real life. Higgins and colleagues have faced this problem, similarly recommending categorization of values for I2 and assigning adjectives of low, moderate, and high heterogeneity or inconsistency.4,5 In our article we have used these values of 50% and 25% for I2, as traditional thresholds for large and moderate heterogeneity, respectively. This does not negate the need to recognize the major uncertainty in heterogeneity estimates,6 but provides a standardized approach that can be applied consistently across meta-analyses. Higgins argues in favour of using τ2, the estimate of between-study variance, rather than I2 in our paper, because I2 depends also on the within-study precisions. Actually I2 has become popular as a measure primarily due to the groundbreaking work of Higgins.3,4I2 is one of the most commonly reported heterogeneity (or inconsistency) metrics, while the between-study variance τ2 is rarely reported in the medical literature. I2 has an intuitive interpretation, and it is comparable across meta-analyses with different numbers of studies or different types of effect metrics, whereas τ2 is difficult both to understand and compare, according to Higgins’ writings.2 Therefore, we focused on I2 in our paper. However, the algorithms that we have proposed are not applicable only to I2. These are general methods that can be used with any kind of metric, e.g. τ2. If another metric may be useful to apply more widely, we would rather suggest h, the ratio of τ over the absolute value of the normalized summary effect (e.g. log odds ratio).7 This h is not to be confused with yet another heterogeneity metric, H capital, which is the square root of the chi-squared heterogeneity statistic divided by its degrees of freedom.2 The major problem with τ2 for an epidemiologist is that it means almost nothing when seen in isolation. The same τ2 value could be huge or negligible depending on what the summary effect is, and what impact that between-study variance has in shaping the upper and lower bounds of the summary effect confidence interval. We are in the process of implementing the sensitivity analysis algorithm on other heterogeneity metrics and we will release the new software module when it is properly beta-tested. Regardless, at a practical level, τ2 and I2 tend to be largely concordant, when examined across many meta-analyses. In Figure 1 we illustrate the correlation between I2 and τ2 in the Cochrane meta-analyses database (n = 1011 meta-analyses) used in our article: the rank correlation coefficient is as high as 0.93. For comparison, the correlation coefficients for h against
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Fixed-effects models may be inaccurate when heterogeneity is large.
“In addition, a fixed-effects models were used to estimate ESs, which may be inaccurate when heterogeneity is large (27).”
From Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies (2017) · cites this paper as [27]
That meta-analysis included relatively few studies and estimated effect sizes through fixed-effects modelling, which can provide unreliable results when heterogeneity across studies is high.
“However, this meta-analysis included relatively few studies, and estimated ES through fixed-effects modelling, which can provide unreliable results when heterogeneity across studies is high [32,21].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139 · see citing works
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763 · see citing works
Uncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254 · see citing works
Self-citationPatsopoulos, N. A., Εvangelou, Ε., & Ioannidis, J. P. A. (2008) · International Journal of Epidemiology · doi:10.1093/ije/dyn065 · Cited by 1,147 · see citing works
Self-citationCommentary: Heterogeneity in meta-analysis should be expected and appropriately quantified
Higgins, J. P. T. (2008) · International Journal of Epidemiology · doi:10.1093/ije/dyn204 · Cited by 995 · see citing works
Required sample size and nonreplicability thresholds for heterogeneous genetic associations
Moonesinghe, R., Khoury, M. J., Liu, T., & Ioannidis, J. P. A. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0705554105 · Cited by 106 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Peripheral cytokine and chemokine alterations in depression: a meta‐analysis of 82 studies
Köhler, C. A., Freitas, T. H., Maes, M., de Andrade, N. Q., Liu, C. S., Fernandes, B. S., Stubbs, B., Solmi, M., Veronese, N., Herrmann, N., et al. (2017) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12698 · Cited by 1,443 · see citing works
Socioeconomic Status and Child Development
Letourneau, N. L., Duffett‐Leger, L., Levac, L., Watson, B. C., & Young-Morris, C. (2011) · Journal of Emotional and Behavioral Disorders · doi:10.1177/1063426611421007 · Cited by 458 · see citing works
Köhler, C. A., Freitas, T. H., Stubbs, B., Maes, M., Solmi, M., Veronese, N., de Andrade, N. Q., Morris, G., Fernandes, B. S., Brunoni, A. R., et al. (2017) · Molecular Neurobiology · doi:10.1007/s12035-017-0632-1 · Cited by 449 · see citing works
Köhler, C. A., Evangelou, E., Stubbs, B., Solmi, M., Veronese, N., Belbasis, L., Bortolato, B., Melo, M. C. A., Coelho, C. A., Fernandes, B. S., et al. (2018) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2018.05.020 · Cited by 302 · see citing works
Self-citationGrading the strength of a body of evidence when assessing health care interventions: an EPC update
Berkman, N. D., Lohr, K. N., Ansari, M. T., Balk, E. M., Kane, R., McDonagh, M., Morton, S. C., Viswanathan, M., Bass, E. B., Butler, M., et al. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.023 · Cited by 279 · see citing works
Pereira, T., Patsopoulos, N. A., Salanti, G., & Ioannidis, J. P. A. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.13 · Cited by 161 · see citing works
Self-citationDepression and Anxiety in Adults With Hidradenitis Suppurativa
Machado, M. O., Stergiopoulos, V., Maes, M. H. J., Kurdyak, P. A., Lin, P.-Y., Wang, L., Shyu, Y., Firth, J., Koyanagi, A., Solmi, M., et al. (2019) · JAMA Dermatology · doi:10.1001/jamadermatol.2019.0759 · Cited by 153 · see citing works
Berkman, N. D., Lohr, K. N., Ansari, M., McDonagh, M. S., Balk, E. M., Whitlock, E. P., Reston, J. T., Bass, E. B., Butler, M. E., Gartlehner, G., et al. (2013) · Europe PMC (PubMed Central) · Cited by 115 · see citing works
Cognitive Impairment in Euthymic Pediatric Bipolar Disorder: A Systematic Review and Meta-Analysis
Elias, L. R., Miskowiak, K. W., Vale, A. M. O., Köhler, C. A., Kjærstad, H. L., Stubbs, B., Kessing, L. V., Vieta, E., Maes, M., Goldstein, B. I., et al. (2017) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2017.01.008 · Cited by 104 · see citing works
Andrés-Rodríguez, L., Borràs, X., Feliu-Soler, A., Pérez-Aranda, A., Angarita-Osorio, N., Moreno-Peral, P., Montero-Marin, J., García-Campayo, J., Carvalho, A. F., Maes, M., et al. (2020) · Brain Behavior and Immunity · doi:10.1016/j.bbi.2019.12.020 · Cited by 95 · see citing works
Vasupanrajit, A., Jirakran, K., Tunvirachaisakul, C., & Maes, M. H. J. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.08.015 · Cited by 69 · see citing works
Immune Aberrations in Obsessive-Compulsive Disorder: a Systematic Review and Meta-analysis
Cosco, T. D., Pillinger, T., Emam, H., Solmi, M., Budhdeo, S., Prina, M. A., Maes, M. H. J., Stein, D. J., Stubbs, B., & Carvalho, A. F. (2018) · Molecular Neurobiology · doi:10.1007/s12035-018-1409-x · Cited by 60 · see citing works
Bias in Peripheral Depression Biomarkers
Carvalho, A. F., Köhler, C. A., Brunoni, A. R., Miskowiak, K. W., Herrmann, N., Lanctôt, K. L., Hyphantis, T. N., Quevedo, J., Fernandes, B. S., & Berk, M. (2016) · Psychotherapy and Psychosomatics · doi:10.1159/000441457 · Cited by 52 · see citing works
Prognostic Biomarkers for Gastric Cancer: An Umbrella Review of the Evidence
Zhou, C., Zhong, X. Z., Song, Y., Shi, J., Wu, Z., Guo, Z., Sun, J., & Wang, Z. (2019) · Frontiers in Oncology · doi:10.3389/fonc.2019.01321 · Cited by 32 · see citing works
Prognostic Biomarkers for Pancreatic Ductal Adenocarcinoma: An Umbrella Review
Wang, Y., Zhong, X. Z., Zhou, L., Lü, J., Jiang, B., Liu, C., & Guo, J. (2020) · Frontiers in Oncology · doi:10.3389/fonc.2020.01466 · Cited by 27 · see citing works
Assessing and Exploring Heterogeneity
Schulzke, S. M. (2021) · Springer eBooks · doi:10.1007/978-3-030-71921-0_3 · Cited by 27 · see citing works
dos Santos, E. C., Orbegozo, D., Mongkolpun, W., Galfo, V., Wang, N., Bogossian, E. G., Taccone, F. S., Vallet, B., Créteur, J., & Vincent, J. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000004115 · Cited by 24 · see citing works
Imaging genetics paradigms in depression research: Systematic review and meta-analysis
Pereira, L. P., Köhler, C. A., Stubbs, B., Miskowiak, K. W., Morris, G., de Freitas, B. P., Thompson, T., Fernandes, B. S., Brunoni, A. R., Maes, M. H. J., et al. (2018) · Progress in Neuro-Psychopharmacology and Biological Psychiatry · doi:10.1016/j.pnpbp.2018.05.012 · Cited by 24 · see citing works
Epigenetic Clock Analysis in Children With Fetal Alcohol Spectrum Disorder
Okazaki, S., Otsuka, I., Shinko, Y., Horai, T., Hirata, T., Yamaki, N., Sora, I., & Hishimoto, A. (2020) · Alcoholism Clinical and Experimental Research · doi:10.1111/acer.14532 · Cited by 21 · see citing works
Sajjaboontawee, N., Supasitthumrong, T., Tunvirachaisakul, C., Nantachai, K., Snabboon, T., Reiche, E. M. V., Simão, A. N. C., & Maes, M. H. J. (2020) · The Aging Male · doi:10.1080/13685538.2020.1858048 · Cited by 15 · see citing works
Bian, C., Lu, D., Su, J., Zhou, X., Zhuge, H.-X., & Lamberton, P. H. L. (2015) · PLoS ONE · doi:10.1371/journal.pone.0128896 · Cited by 9 · see citing works
Exposure to Commonly Used Drugs and the Risk of Gastric Cancer: An Umbrella Review of Meta-Analyses
Bai, X., Ding, S., Zhang, X., Han, M.-H., & Dai, D. (2023) · Cancers · doi:10.3390/cancers15020372 · Cited by 8 · see citing works
Lee, S. A. W., & Gates, K. M. (2023) · Multivariate Behavioral Research · doi:10.1080/00273171.2023.2229310 · Cited by 9 · see citing works
Bai, X., Guo, Y., Zhu, X.-M., & Dai, D. (2025) · International Journal of Surgery · doi:10.1097/js9.0000000000002219 · Cited by 7 · see citing works
Wagh, D., Pawale, D., Tan, J. K. G., & Rao, S. C. (2022) · Archives of Disease in Childhood · doi:10.1136/archdischild-2021-323757 · Cited by 7 · see citing works
Su, J., Lu, D., Zhou, X., Wang, S., & Zhuge, H.-X. (2013) · PLoS ONE · doi:10.1371/journal.pone.0078509 · Cited by 7 · see citing works
Association of Essential Tremor With Dementia and Affective Disorders: A Meta-Analysis
Shang, Y., Chen, X., Ai, M., Gao, X., Dai, S., Zhao, M., Yang, C., Wang, L., Zhang, J., Zhong, L., et al. (2022) · Frontiers in Neurology · doi:10.3389/fneur.2022.842732 · Cited by 6 · see citing works
Mohammadi, S., Ziaei, S., Morvaridi, M., Hasani, M., Mirtaheri, E., Farsi, F., Ebrahimi, S., Daneshzad, E., & Heshmati, J. (2025) · Health Science Reports · doi:10.1002/hsr2.70525 · Cited by 5 · see citing works
A ‘reluctant’ critical review: ‘Manual for evidence‐based clinical practice (2015)’
Seshia, S. S. (2015) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12509 · Cited by 3 · see citing works
Lee, S. A. W., & Gates, K. M. (2022) · doi:10.31234/osf.io/u8mcp · Cited by 3 · see citing works
Sajjaboontawee, N., Supasitthumrong, T., Tunvirachaisakul, C., Nantachai, K., Snabboon, T., Reiche, E. M. V., Simão, A. N. C., & Maes, M. H. J. (2020) · Preprints.org · doi:10.20944/preprints202006.0003.v1 · Cited by 2 · see citing works
Sánchez‐Ortí, J. V., Forés-Martos, J., Doan, V., Vicente-Martínez, P., Saint-Gerons, D. M., Flores-Rodero, M., Sánchez-Valle, J., Correa‐Ghisays, P., Balanzà, V., Soldevila-Matías, P., et al. (2025) · JMIR Research Protocols · doi:10.2196/78596 · Cited by 1 · see citing works
Vasupanrajit, A., Jirakran, K., Tunvirachaisakul, C., & Maes, M. H. J. (2021) · Preprints.org · doi:10.20944/preprints202104.0479.v1 · Cited by 1 · see citing works
Liu, H., Bai, X., Wang, Y., & 戴冬秋 (2026) · BMC Oral Health · doi:10.1186/s12903-026-09402-3
Gentile, E., Guglietti, G. V., Haward, B., Borokhovski, E., Plante, I., Fitzcharles, M., Vachon‐Presseau, É., & Roy, M. (2026) · Pain · doi:10.1097/j.pain.0000000000004085
- Authors
- Patsopoulos, N. A., Evangelou, E., & Ioannidis, J. P.
- Affiliations
- University of Ioannina (GR)
- Corresponding author
- Ioannidis, J. P.
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2008-10-21
- Volume
- 38
- Issue
- 6
- Pages
- 1740-1742
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- PMC
- PMC4719167
- OpenAlex
- W2101347931
- Cited by
- 35
- References
- 6
- Citations by year
- 2026: 2 · 2025: 3 · 2023: 2 · 2022: 3 · 2021: 3 · 2020: 4 · 2019: 4 · 2018: 3 · 2017: 3 · 2016: 1 · 2015: 2 · 2014: 1 · 2013: 2
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- UCLA - Biblioteca de Ciencias de la Salud (submittedVersion)
- Funders
- National Heart, Lung, and Blood Institute (T32HL007034)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- heterogeneity, inconsistency, I2 statistic, between-study variability, sensitivity analysis
- MeSH terms
- Data Interpretation, Statistical, Humans, Terminology as Topic, Meta-Analysis as Topic, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
