Commentary: Heterogeneity in meta-analysis should be expected and appropriately quantified
Abstract
It is generally accepted that meta-analyses should assess heterogeneity, which may be defined as the presence of variation in true effect sizes underlying the different studies. This assessment might be achieved by performing a statistical test for heterogeneity, by quantifying its magnitude, by quantifying its impact or by a combination of these. Patsopoulos, Evangelou and Ioannidis propose methods for examining the effect of excluding studies (or groups of studies) on an assessment of heterogeneity.1 Their methods offer benefits over the sometimes practiced ‘leave one out’ approach to sensitivity analysis, by recognizing that the overall effect (against which heterogeneity is measured) changes each time an influential study is excluded. The authors offer a sequential approach (in which the overall effect and heterogeneity measure are re-estimated after the most influential study is removed at each iteration), and a combinatorial approach (in which groups of studies are removed). I suspect the sequential approach may often be performed informally in practice, whereby an obvious outlier is excluded, but another study then appears to be an outlier compared with the remaining studies and is in turn excluded. Of course, if heterogeneity permeates the entire set of studies, one might be tempted continue excluding studies to reduce heterogeneity until a single study remains. A predefined stopping rule (a ‘desired heterogeneity threshold’, in the authors’ terminology) may therefore appear to offer a useful way forward.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Heterogeneity was assessed by the Cochran Q test and I2 statistic, with low, moderate, and high I2 values considered to be 25%, 50%, and 75%, respectively.
“Heterogeneity was assessed by the Cochran Q test and I2 statistic; low, moderate, and high I2 values were considered to be 25%, 50%, and 75%, respectively.32, 33”
From Gut Microbiota Metabolites and Risk of Major Adverse Cardiovascular Disease Events and Death: A Systematic Review and Meta‐Analysis of Prospective Studies (2017) · cites this paper as [32]
Cites
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139 · see citing works
Self-citationQuantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763 · see citing works
Self-citationA Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569 · see citing works
Self-citationUncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254 · see citing works
Patsopoulos, N. A., Εvangelou, Ε., & Ioannidis, J. P. A. (2008) · International Journal of Epidemiology · doi:10.1093/ije/dyn065 · Cited by 1,147 · see citing works
Confidence intervals for the amount of heterogeneity in meta‐analysis
Viechtbauer, W. (2006) · Statistics in Medicine · doi:10.1002/sim.2514 · Cited by 399 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The global prevalence of common mental disorders: a systematic review and meta-analysis 1980–2013
Steel, Z., Marnane, C., Iranpour, C., Chey, T., Jackson, J. W., Patel, V., & Silove, D. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyu038 · Cited by 2,846 · see citing works
Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies
Higgins, J. P. T., Jackson, D., Barrett, J. K., Lu, G., Ades, A. E., & White, I. R. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.1044 · Cited by 2,098 · see citing works
Self-citationBasics of meta‐analysis: <i>I</i> <sup>2</sup> is not an absolute measure of heterogeneity
Borenstein, M., Higgins, J. P. T., Hedges, L. V., & Rothstein, H. R. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1230 · Cited by 1,848 · see citing works
Self-citationPrevalence of intellectual disability: a meta-analysis of population-based studies
Maulik, P. K., Mascarenhas, M. N., Mathers, C. D., Dua, T., & Saxena, S. (2011) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2010.12.018 · Cited by 1,770 · see citing works
Undue reliance on I 2 in assessing heterogeneity may mislead
Rücker, G., Schwarzer, G., Carpenter, J. R., & Schumacher, M. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-79 · Cited by 1,287 · see citing works
Objectively measured physical capability levels and mortality: systematic review and meta-analysis
Cooper, R., Kuh, D., Hardy, R., Mortality Review Group, & teams, O. B. O. T. F. A. H. S. (2010) · BMJ · doi:10.1136/bmj.c4467 · Cited by 1,166 · see citing works
“We have presented summary, overall estimates from meta-analyses despite this, as most effect estimates from individual studies were in the same direction and doing meta-analyses has been argued to still be appropriate.”
MethodSevere mental illness in 33 588 prisoners worldwide: systematic review and meta-regression analysis
Fazel, S., & Seewald, K. (2012) · British Journal of Psychiatry · doi:10.1192/bjp.bp.111.096370 · Cited by 985 · see citing works
Bowden, J., Spiller, W., Del Greco M, F., Sheehan, N., Thompson, J., Minelli, C., & Davey Smith, G. (2018) · International Journal of Epidemiology · doi:10.1093/ije/dyy101 · Cited by 880 · see citing works
Shi, L., & Lin, L. (2019) · Medicine · doi:10.1097/md.0000000000015987 · Cited by 818 · see citing works
Risk factors for type 2 diabetes mellitus: An exposure-wide umbrella review of meta-analyses
Bellou, V., Belbasis, L., Tzoulaki, I., & Evangelou, E. (2018) · PLOS ONE · doi:10.1371/journal.pone.0194127 · Cited by 713 · see citing works
Carroll, R., Metcalfe, C., & Gunnell, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0089944 · Cited by 586 · see citing works
“Heterogeneity in itself should not deter one from a meta-analysis [19].”
BackgroundHeianza, Y., Ma, W., Manson, J. E., Rexrode, K. M., & Qi, L. (2017) · Journal of the American Heart Association · doi:10.1161/jaha.116.004947 · Cited by 544 · see citing works
Schoenaker, D. A., Jackson, C. A., Rowlands, J. V., & Mishra, G. D. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyu094 · Cited by 550 · see citing works
Depression and the risk of coronary heart disease: a meta-analysis of prospective cohort studies
Gan, Y., Gong, Y., Tong, X., Sun, H., Cong, Y., Dong, X., Wang, Y., Xu, X., Yin, X., Deng, J., et al. (2014) · BMC Psychiatry · doi:10.1186/s12888-014-0371-z · Cited by 502 · see citing works
Lehrer, P., Kaur, K., Sharma, A., Shah, K., Huseby, R., Bhavsar, J., Sgobba, P., & Zhang, Y. (2020) · Applied Psychophysiology and Biofeedback · doi:10.1007/s10484-020-09466-z · Cited by 469 · see citing works
Giles, E. L., Robalino, S., McColl, E., Sniehotta, F. F., & Adams, J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0090347 · Cited by 470 · see citing works
Weisman, A., Bai, J.-W., Cardinez, M., Kramer, C. K., & Perkins, B. A. (2017) · The Lancet Diabetes & Endocrinology · doi:10.1016/s2213-8587(17)30167-5 · Cited by 463 · see citing works
The association of telomere length and cancer: a meta-analysis
Wentzensen, I. M., Mirabello, L., Pfeiffer, R. M., & Savage, S. A. (2011) · Cancer Epidemiology, Biomarkers & Prevention · doi:10.1158/1055-9965.epi-11-0005 · Cited by 458 · see citing works
Oliveira, A. S., Weinberg, Z. G., Ogunade, I. M., Cervantes, A. A. P., Arriola, K. G., Jiang, Y., Kim, D., Li, X., Gonçalves, M. C. M., Vyas, D., et al. (2017) · Journal of Dairy Science · doi:10.3168/jds.2016-11815 · Cited by 409 · see citing works
Grgic, J., Schoenfeld, B. J., Davies, T. B., Lazinica, B., Krieger, J. W., & Pedisic, Z. (2018) · Sports Medicine · doi:10.1007/s40279-018-0872-x · Cited by 365 · see citing works
Rong, Y., Chen, L., Zhu, T., Song, Y., Yu, M., Shan, Z., Sands, A., Hu, F. B., & Liu, L. (2013) · BMJ · doi:10.1136/bmj.e8539 · Cited by 364 · see citing works
Locher, C., Koechlin, H., Zion, S. R., Werner, C., Pine, D. S., Kirsch, I., Kessler, R. C., & Kossowsky, J. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.2432 · Cited by 360 · see citing works
Madigan, S., Racine, N., Vaillancourt, T., Korczak, D. J., Hewitt, J. M. A., Pador, P., Park, J. L., McArthur, B. A., Holy, C., & Neville, R. D. (2023) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2023.0846 · Cited by 354 · see citing works
Busch, T., & Friede, G. (2018) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.1480 · Cited by 345 · see citing works
A meta-analysis and review of holistic face processing.
Richler, J. J., & Gauthier, I. (2014) · Psychological Bulletin · doi:10.1037/a0037004 · Cited by 340 · see citing works
Wong, M. C. S., Huang, J., Wang, J., Chan, P. S. F., Lok, V., Chen, X., Leung, C., Wang, H. H. X., Lao, X. Q., & Zheng, Z.-J. (2020) · European Journal of Epidemiology · doi:10.1007/s10654-020-00650-3 · Cited by 327 · see citing works
Jabre, P., Roger, V. L., Murad, M. H., Chamberlain, A. M., Prokop, L., Adnet, F., & Jouven, X. (2011) · Circulation · doi:10.1161/circulationaha.110.986661 · Cited by 314 · see citing works
Huo, R., Du, T., Xu, Y., Xu, W., Chen, X., Sun, K., & Yu, X. (2015) · European Journal of Clinical Nutrition · doi:10.1038/ejcn.2014.243 · Cited by 296 · see citing works
Fernandes, B. S., Molendijk, M. L., Köhler, C. A., Soares, J. C., Leite, C. M. G. S., Machado-Vieira, R., Ribeiro, T. L., Silva, J. C., Sales, P. M. G., Quevedo, J., et al. (2015) · BMC Medicine · doi:10.1186/s12916-015-0529-7 · Cited by 291 · see citing works
Grading the strength of a body of evidence when assessing health care interventions: an EPC update
Berkman, N. D., Lohr, K. N., Ansari, M. T., Balk, E. M., Kane, R., McDonagh, M., Morton, S. C., Viswanathan, M., Bass, E. B., Butler, M., et al. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.023 · Cited by 279 · see citing works
Wu, Y., Wang, W., Liu, T., & Zhang, D. (2017) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2017.03.011 · Cited by 276 · see citing works
Hartmann, M., Bäzner, E., Wild, B., Eisler, I., & Herzog, W. (2010) · Psychotherapy and Psychosomatics · doi:10.1159/000286958 · Cited by 261 · see citing works
Adverse childhood experiences and risk of type 2 diabetes: A systematic review and meta-analysis
Huang, H., Yan, P., Shan, Z., Chen, S., Li, M., Luo, C., Gao, H., Hao, L., & Liu, L. (2015) · Metabolism · doi:10.1016/j.metabol.2015.08.019 · Cited by 239 · see citing works
Belbasis, L., Savvidou, M. D., Kanu, C., Evangelou, E., & Tzoulaki, I. (2016) · BMC Medicine · doi:10.1186/s12916-016-0692-5 · Cited by 237 · see citing works
Kok, B. C., Herrell, R. K., Thomas, J. L., & Hoge, C. W. (2012) · Journal of Nervous & Mental Disease · doi:10.1097/nmd.0b013e3182532312 · Cited by 238 · see citing works
El Shayeb, M., Topfer, L.-A., Stafinski, T., Pawluk, L., & Menon, D. (2014) · Canadian Medical Association Journal · doi:10.1503/cmaj.130952 · Cited by 237 · see citing works
Karver, M. S., De Nadai, A. S., Monahan, M., & Shirk, S. R. (2018) · Psychotherapy · doi:10.1037/pst0000176 · Cited by 234 · see citing works
Walsh, K., Zwi, K., Woolfenden, S., & Shlonsky, A. (2018) · Research on Social Work Practice · doi:10.1177/1049731515619705 · Cited by 239 · see citing works
Asogwa, O. A., Boateng, D., Marzà-Florensa, A., Peters, S., Levitt, N., van Olmen, J., & Klipstein-Grobusch, K. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-049133 · Cited by 221 · see citing works
Burgess, E., Hassmén, P., Welvaert, M., & Pumpa, K. L. (2017) · Clinical Obesity · doi:10.1111/cob.12180 · Cited by 214 · see citing works
Intimate partner violence victimization and parenting: A systematic review.
Chiesa, A. E., Kallechey, L., Harlaar, N., Rashaan Ford, C., Garrido, E. F., Betts, W. R., & Maguire, S. (2018) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2018.03.028 · Cited by 198 · see citing works
Bright light therapy for nonseasonal depression: meta-analysis of clinical trials
Al-Karawi, D., & Jubair, L. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.03.016 · Cited by 189 · see citing works
The efficacy of manual toothbrushes following a brushing exercise: a systematic review
Slot, D., Wiggelinkhuizen, L., Rosema, N., & Van der Weijden, G. (2012) · International Journal of Dental Hygiene · doi:10.1111/j.1601-5037.2012.00557.x · Cited by 182 · see citing works
Hilgard, J., Engelhardt, C. R., & Rouder, J. N. (2017) · Psychological Bulletin · doi:10.1037/bul0000074 · Cited by 162 · see citing works
Rubio-Arias, J. Á., Marín-Cascales, E., Ramos-Campo, D. J., Hernandez, A. V., & Pérez-López, F. R. (2017) · Maturitas · doi:10.1016/j.maturitas.2017.04.003 · Cited by 159 · see citing works
Langan, D., Higgins, J. P. T., & Simmonds, M. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1198 · Cited by 149 · see citing works
Self-citationFletcher, R., Freeman, E., & Matthey, S. (2011) · Fathering: A Journal of Theory, Research, and Practice about Men as Fathers · doi:10.3149/fth.0903.291 · Cited by 145 · see citing works
High statistical heterogeneity is more frequent in meta-analysis of continuous than binary outcomes
Alba, A. C., Alexander, P. E., Chang, J., MacIsaac, J., DeFry, S., & Guyatt, G. H. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.09.005 · Cited by 129 · see citing works
Height at diagnosis and birth-weight as risk factors for osteosarcoma
Mirabello, L., Pfeiffer, R., Murphy, G., Daw, N. C., Patiño-Garcia, A., Troisi, R. J., Hoover, R. N., Douglass, C., Schüz, J., Craft, A. W., et al. (2011) · Cancer Causes & Control · doi:10.1007/s10552-011-9763-2 · Cited by 116 · see citing works
Facial mimicry, empathy, and emotion recognition: a meta-analysis of correlations
Holland, A. C., O’Connell, G., & Dziobek, I. (2021) · Cognition and Emotion · doi:10.1080/02699931.2020.1815655 · Cited by 114 · see citing works
Pérez-López, F. R., Martínez-Domínguez, S. J., Lajusticia, H., & Chedraui, P. (2017) · Maturitas · doi:10.1016/j.maturitas.2017.09.001 · Cited by 108 · see citing works
Tea consumption and the risk of depression: a meta-analysis of observational studies
Dong, X., Yang, C., Cao, S., Gan, Y., Sun, H., Gong, Y., Yang, H., Yin, X., & Lu, Z. (2015) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867414567759 · Cited by 102 · see citing works
Grgic, J., Lazinica, B., Garofolini, A., Schoenfeld, B. J., Saner, N. J., & Mikulic, P. (2019) · Chronobiology International · doi:10.1080/07420528.2019.1567524 · Cited by 83 · see citing works
Hita-Contreras, F., Bueno-Notivol, J., Martínez-Amat, A., Cruz-Díaz, D., Hernandez, A. V., & Pérez-López, F. R. (2018) · Maturitas · doi:10.1016/j.maturitas.2018.07.007 · Cited by 86 · see citing works
Silva Pinto, B. G., Marques Soares, T. K., Azevedo Linhares, M., & Castilhos Ghisi, N. (2020) · Science of The Total Environment · doi:10.1016/j.scitotenv.2020.141382 · Cited by 72 · see citing works
Rodríguez-Rivas, M. E., Cangas, A. J., Cariola, L. A., Varela, J. J., & Valdebenito, S. (2022) · JMIR Serious Games · doi:10.2196/35099 · Cited by 68 · see citing works
Marshall, J., Bishop, C., Turner, A., & Haff, G. G. (2021) · Sports Medicine · doi:10.1007/s40279-021-01430-z · Cited by 64 · see citing works
Bias in emerging biomarkers for bipolar disorder
Carvalho, A. F., Köhler, C. A., Fernandes, B. S., Quevedo, J., Miskowiak, K. W., Brunoni, A. R., Machado-Vieira, R., Maes, M., Vieta, E., & Berk, M. (2016) · Psychological Medicine · doi:10.1017/s0033291716000957 · Cited by 63 · see citing works
Bias in Peripheral Depression Biomarkers
Carvalho, A. F., Köhler, C. A., Brunoni, A. R., Miskowiak, K. W., Herrmann, N., Lanctôt, K. L., Hyphantis, T. N., Quevedo, J., Fernandes, B. S., & Berk, M. (2016) · Psychotherapy and Psychosomatics · doi:10.1159/000441457 · Cited by 52 · see citing works
Perez-Lopez, F. R., Pasupuleti, V., Gianuzzi, X., Palma-Ardiles, G., Hernandez-Fernandez, W., & Hernandez, A. V. (2017) · Maturitas · doi:10.1016/j.maturitas.2017.04.001 · Cited by 45 · see citing works
Qu, H., Guo, M., Zhang, Y., & Shi, D.-Z. (2018) · Sleep and Breathing · doi:10.1007/s11325-017-1503-8 · Cited by 36 · see citing works
Heterogeneous views on heterogeneity
Patsopoulos, N. A., Evangelou, E., & Ioannidis, J. P. (2009) · International Journal of Epidemiology · doi:10.1093/ije/dyn235 · Cited by 35 · see citing works
Lan, C.-W., Scott-Sheldon, L. A. J., Carey, K. B., Johnson, B. T., & Carey, M. P. (2017) · International Journal of Behavioral Medicine · doi:10.1007/s12529-016-9596-1 · Cited by 33 · see citing works
Gerolymos, C., Barazer, R., Yon, D. K., Loundou, A., Boyer, L., & Fond, G. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.1527 · Cited by 28 · see citing works
Bakker, C., Boughton, S., Faggion, C. M., Fanelli, D., Kaiser, K., & Schneider, J. (2024) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2022-111921 · Cited by 28 · see citing works
Bayesian multivariate meta-analysis of multiple factors
Lin, L., & Chu, H. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1293 · Cited by 19 · see citing works
- Authors
- Higgins, J. P. T.
- Affiliations
- MRC Biostatistics Unit (GB)
- Corresponding author
- Higgins, J. P T
- ORCID
- Higgins, J. P T
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2008-10-01
- Volume
- 37
- Issue
- 5
- Pages
- 1158-1160
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2153287976
- Cited by
- 995
- References
- 6
- Citations by year
- 2026: 50 · 2025: 76 · 2024: 84 · 2023: 89 · 2022: 74 · 2021: 97 · 2020: 96 · 2019: 63 · 2018: 71 · 2017: 68 · 2016: 62 · 2015: 32 · 2014: 41 · 2013: 30 · 2012: 23
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- meta-analysis heterogeneity, sensitivity analysis, leave-one-out analysis, stopping rules
- MeSH terms
- Data Interpretation, Statistical, Humans, Meta-Analysis as Topic, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
