The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
The heterogeneity statistic I2 can be biased in small meta-analyses
Open accessAbstract
BACKGROUND: Estimated effects vary across studies, partly because of random sampling error and partly because of heterogeneity. In meta-analysis, the fraction of variance that is due to heterogeneity is estimated by the statistic I(2). We calculate the bias of I(2), focusing on the situation where the number of studies in the meta-analysis is small. Small meta-analyses are common; in the Cochrane Library, the median number of studies per meta-analysis is 7 or fewer. METHODS: We use Mathematica software to calculate the expectation and bias of I(2). RESULTS: I(2) has a substantial bias when the number of studies is small. The bias is positive when the true fraction of heterogeneity is small, but the bias is typically negative when the true fraction of heterogeneity is large. For example, with 7 studies and no true heterogeneity, I(2) will overestimate heterogeneity by an average of 12 percentage points, but with 7 studies and 80 percent true heterogeneity, I(2) can underestimate heterogeneity by an average of 28 percentage points. Biases of 12-28 percentage points are not trivial when one considers that, in the Cochrane Library, the median I(2) estimate is 21 percent. CONCLUSIONS: The point estimate I(2) should be interpreted cautiously when a meta-analysis has few studies. In small meta-analyses, confidence intervals should supplement or replace the biased point estimate I(2).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The I2 statistic in analyses of a few studies is considered to be uninformative90,91”
From Effects of Tea (Camellia sinensis) or its Bioactive Compounds l-Theanine or l-Theanine plus Caffeine on Cognition, Sleep, and Mood in Healthy Participants: A Systematic Review and Meta-Analysis of Randomized Controlled Trials (2025) · cites this paper as [91]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The heterogeneity estimates for professional burnout, somatisation, positive affect, negative affect, and resilience may not be reliable given the few comparisons contributing to the effect size (von Hippel, 2015).”
From Intervention Programs Targeting the Mental Health, Professional Burnout, and/or Wellbeing of School Teachers: Systematic Review and Meta-Analyses (2023) · cites this paper as von Hippel, 2015
In the citing paper
“Variation in effect sizes due to heterogeneity (rather than chance) was assessed with I2.54”
From Birth outcomes following intrauterine exposure to electronic nicotine delivery systems: a systematic review and meta-analysis (2026) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
The combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
A Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569
Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model
Viechtbauer, W. (2005) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986030003261 · Cited by 1,699
Undue reliance on I 2 in assessing heterogeneity may mislead
Rücker, G., Schwarzer, G., Carpenter, J. R., & Schumacher, M. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-79 · Cited by 1,287
Uncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254
Turner, R. M., Davey, J., Clarke, M. J., Thompson, S. G., & Higgins, J. P. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dys041 · Cited by 837
Melsen, W. G., Bootsma, M. C. J., Rovers, M. M., & Bonten, M. J. M. (2014) · Clinical Microbiology and Infection · doi:10.1111/1469-0691.12494 · Cited by 776
The power of statistical tests in meta-analysis
Hedges, L. V., & Pigott, T. D. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.3.203 · Cited by 716
Research works printed but not linked: 2
- von Hippel PT, Osborne C, Lincove A, Bellows L, Mills N. The challenges of seeking exceptional teacher preparation programs among many noisy estimates. Rochester, NY: Social Science Research Network; 2014. SSRN Scholarly Paper ID 2506935.
- Cochrane Collaborative, “Cochrane Library”, 2015. [Online]. Available: http://www.cochranelibrary.com/ . [Accessed: 25-Feb-2015].
Cited by
Prevalence of depression during the COVID-19 outbreak: A meta-analysis of community-based studies
Bueno-Notivol, J., Gracia-García, P., Olaya, B., Lasheras, I., López-Antón, R., & Santabárbara, J. (2021) · International Journal of Clinical and Health Psychology · doi:10.1016/j.ijchp.2020.07.007 · Cited by 820
Correll, C. U., Solmi, M., Croatto, G., Schneider, L. K., Rohani‐Montez, S. C., Fairley, L., Smith, N., Bitter, I., Gorwood, P., Taipale, H., et al. (2022) · World Psychiatry · doi:10.1002/wps.20994 · Cited by 686
Braithwaite, I., Zhang, S., Kirkbride, J. B., Osborn, D. P. J., & Hayes, J. F. (2019) · Environmental Health Perspectives · doi:10.1289/ehp4595 · Cited by 680
Meta‐analysis of prevalence: I 2 statistic and how to deal with heterogeneity
Migliavaca, C. B., Stein, C., Colpani, V., Barker, T. H., Ziegelmann, P. K., Munn, Z., Falavigna, M., & Prevalence Estimates Reviews—Systematic Review Methodology Group (PERSyst) (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1547 · Cited by 577
Goldberg, S. B., Riordan, K. M., Sun, S., & Davidson, R. J. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620968771 · Cited by 520
“A third limitation was the moderate degree of heterogeneity within the included studies (compounded by uncertainty in the I 2 estimates themselves given the 126 Goldberg et al. small number of studies for some effect sizes; von Hippel, 2015).”
Background
Metadata
- Authors
- von Hippel, P. T.
- Venue
- BMC Medical Research Methodology
- Published
- 2015-04-14
- Volume
- 15
- Issue
- 1
- Pages
- 35-35
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC4410499
- OpenAlex
- W2021593215
- Cited by
- 1,229
- References
- 26
- Citations by year
- 2026: 106 · 2025: 185 · 2024: 216 · 2023: 195 · 2022: 173 · 2021: 131 · 2020: 71 · 2019: 63 · 2018: 51 · 2017: 24 · 2016: 9 · 2015: 3
- FWCI
- 26.46
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- I2 statistic, heterogeneity, estimation bias, between-study variance, confidence intervals, Cochrane Library
- MeSH terms
- Algorithms, Biometry, Humans, Mathematical Computing, Research, Research Design, Software, Meta-Analysis as Topic, Reproducibility of Results, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The heterogeneity statistic I2 can be biased in small meta-analyses. https://app.cochat.ai/papers/10.1186/s12874-015-0024-z
- The paper
- von Hippel, P. T. (2015). The heterogeneity statistic I2 can be biased in small meta-analyses. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-015-0024-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.