The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Interpretation of tests of heterogeneity and bias in meta-analysis
Abstract
Statistical tests of heterogeneity and bias, in particular publication bias, are very popular in meta-analyses. These tests use statistical approaches whose limitations are often not recognized. Moreover, it is often implied with inappropriate confidence that these tests can provide reliable answers to questions that in essence are not of statistical nature. Statistical heterogeneity is only a correlate of clinical and pragmatic heterogeneity and the correlation may sometimes be weak. Similarly, statistical signals may hint to bias, but seen in isolation they cannot fully prove or disprove bias in general, let alone specific causes of bias, such as publication bias in particular. Both false-positive and false-negative signals of heterogeneity and bias can be common and their prevalence may be anticipated based on some rational considerations. Here I discuss the major common challenges and flaws that emerge in using and interpreting statistical tests of heterogeneity and bias in meta-analyses. I discuss misinterpretations that can occur at the level of statistical inference, clinical/pragmatic inference and specific cause attribution. Suggestions are made on how to avoid these flaws, use these tests properly and learn from them.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These limitations are common in meta-analyses of cross-sectional studies (Ioannidis, 2008).”
From Decision making capacity for treatment in psychiatric inpatients: a systematic review and meta-analysis (2024) · cites this paper as Ioannidis, 2008
Citations
Cites
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationThe combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Self-citationAssessing heterogeneity in meta-analysis: Q statistic or I² index?
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.2.193 · Cited by 4,256
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
How should meta‐regression analyses be undertaken and interpreted?
Thompson, S. G., & Higgins, J. P. T. (2002) · Statistics in Medicine · doi:10.1002/sim.1187 · Cited by 3,026
A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955
Quantitative Synthesis in Systematic Reviews
Lau, J., Ioannidis, J. P. A., & Schmid, C. H. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-9-199711010-00008 · Cited by 2,517
Self-citationPublication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Comparison of two methods to detect publication bias in meta-analysis.
Peters, J. L. (2006) · JAMA · doi:10.1001/jama.295.6.676 · Cited by 2,278
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
(2000) · Queen Mary Research Online (Queen Mary University of London) · Cited by 1,795
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766
Self-citationPublication bias in meta-analysis: Prevention, assessment and adjustments
Viechtbauer, W. (2007) · Psychometrika · doi:10.1007/s11336-006-1450-y · Cited by 1,508
Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417
A comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286
Controlling the risk of spurious findings from meta-regression
Higgins, J. P. T., & Thompson, S. G. (2004) · Statistics in Medicine · doi:10.1002/sim.1752 · Cited by 1,267
Uncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254
Self-citationPatsopoulos, N. A., Εvangelou, Ε., & Ioannidis, J. P. A. (2008) · International Journal of Epidemiology · doi:10.1093/ije/dyn065 · Cited by 1,157
Self-citationThe appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052
Self-citationA consumer's guide to subgroup analyses
Oxman, A. D., & Guyatt, G. H. (1992) · Annals of Internal Medicine · doi:10.7326/0003-4819-116-1-78 · Cited by 914
BAUJAT, B., Mahé, C., Pignon, J., & Hill, C. (2002) · Statistics in Medicine · doi:10.1002/sim.1221 · Cited by 798
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751
Self-citationA note on graphical presentation of estimated odds ratios from several clinical trials
Galbraith, R. F. (1988) · Statistics in Medicine · doi:10.1002/sim.4780070807 · Cited by 663
Summing up evidence: one answer is not always enough
Lau, J., Ioannidis, J. P. A., & Schmid, C. H. (1998) · The Lancet · doi:10.1016/s0140-6736(97)08468-7 · Cited by 655
Self-citationBias in clinical intervention research.
Gluud, L. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj069 · Cited by 636
Chan, A.-W., & Altman, D. G. (2005) · BMJ · doi:10.1136/bmj.38356.424606.8f · Cited by 613
Simmonds, M. C., Higgins, J. P. T., Stewart, L. A., Tierney, J. F., Clarke, M. J., & Thompson, S. G. (2005) · Clinical Trials · doi:10.1191/1740774505cn087oa · Cited by 554
Limitations of Applying Summary Results of Clinical Trials to Individual Patients
Kent, D. M., & Hayward, R. A. (2007) · JAMA · doi:10.1001/jama.298.10.1209 · Cited by 502
Terrin, N. C., Schmid, C. H., & Lau, J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.01.006 · Cited by 502
Relative Citation Impact of Various Study Designs in the Health Sciences
Patsopoulos, N. A. (2005) · JAMA · doi:10.1001/jama.293.19.2362 · Cited by 498
Misleading funnel plot for detection of bias in meta-analysis
Tang, J.-L., & Liu, J. L. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00204-8 · Cited by 420
Lambert, P. C., Sutton, A. J., Abrams, K. R., & Jones, D. R. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00414-0 · Cited by 398
Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care
Lau, J., Schmid, C. H., & Chalmers, T. C. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00106-z · Cited by 350
Schmid, C. H., Stark, P. C., Berlin, J. A., Landais, P., & Lau, J. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.12.001 · Cited by 245
Evolution of treatment effects over time: Empirical insight from recursive cumulative metaanalyses
Ioannidis, J. P. A., & Lau, J. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.98.3.831 · Cited by 190
Self-citationA COMPARISON OF STATISTICAL METHODS FOR CLUSTERED DATA ANALYSIS WITH GAUSSIAN ERROR
Feng, Z., McLerran, D., & Grizzle, J. E. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19960830)15:16 · Cited by 158
Claims of Sex Differences: An Empirical Assessment in Genetic Associations
Patsopoulos, N. A., Tatsioni, A., & Ioannidis, J. P. A. (2007) · JAMA · doi:10.1001/jama.298.8.880 · Cited by 150
Self-citationA simulation study comparing properties of heterogeneity measures in meta‐analyses
Mittlböck, M., & Heinzl, H. (2006) · Statistics in Medicine · doi:10.1002/sim.2692 · Cited by 143
Commentary: Meta-analysis of Individual Participants' Data in Genetic Epidemiology
Ioannidis, J. P. A. (2002) · American Journal of Epidemiology · doi:10.1093/aje/kwf031 · Cited by 117
Self-citationSelection in Reported Epidemiological Risks: An Empirical Assessment
Kavvoura, F. K., Liberopoulos, G., & Ioannidis, J. P. A. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040079 · Cited by 115
Self-citationThe impact of high-risk patients on the results of clinical trials
Ioannidis, J. P. A., & Lau, J. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00149-2 · Cited by 108
Self-citationClinical trial registration: looking back and moving ahead
Laine, C. M., Horton, R. C., DeAngelis, C. D., Drazen, J. M., Frizelle, F., Godlee, F., Haug, C. J., Hébert, P. C., Kotzin, S., Marušić, A., et al. (2007) · The Lancet · doi:10.1016/s0140-6736(07)60894-0 · Cited by 97
Recursive Cumulative Meta-analysis
Ioannidis, J. P. A., Contopoulos‐Ioannidis, D. G., & Lau, J. (1999) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(98)00159-0 · Cited by 84
Self-citationSchwarzer, G., Antes, G., & Schumacher, M. C. (2002) · Statistics in Medicine · doi:10.1002/sim.1224 · Cited by 81
When Should Potentially False Research Findings Be Considered Acceptable?
Djulbegović, B., & Hozo, I. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040026 · Cited by 77
loannidis, J. P. A., & Lau, J. (1998) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009590 · Cited by 72
Extreme between-study homogeneity in meta-analyses could offer useful insights
Ioannidis, J. P. A., Trikalinos, T. A., & Zintzaras, E. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.02.013 · Cited by 58
Self-citationIoannidis, J. P. A., Cappelleri, J. C., Sacks, H. S., & Lau, J. (1997) · Controlled Clinical Trials · doi:10.1016/s0197-2456(97)00097-4 · Cited by 43
Self-citation
Cited by
Sorted by citations
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709
Self-citationGloy, V. L., Briel, M., Bhatt, D. L., Kashyap, S. R., Schauer, P. R., Mingrone, G., Bucher, H. C., & Nordmann, A. J. (2013) · BMJ · doi:10.1136/bmj.f5934 · Cited by 1,351
Sonuga-Barke, E. J. S., Brandeis, D., Cortese, S., Daley, D., Ferrin, M., Holtmann, M., Stevenson, J., Danckaerts, M., van der Oord, S., Döpfner, M., et al. (2013) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2012.12070991 · Cited by 1,167
“Given the relatively small number of methodologically sound studies, the field is not yet mature enough for the investigation of publication bias using funnel plots—the interpretation of which, moreover, is equivocal when based on a small number of studies (29).”
BackgroundA systematic review of personality trait change through intervention.
Roberts, B. W., Luo, J., Briley, D. A., Chow, P. I., Su, R., & Hill, P. L. (2017) · Psychological Bulletin · doi:10.1037/bul0000088 · Cited by 1,047
“Most critically, Ioannidis (2008) argued that when effect sizes are even moderately heterogeneous, regression-based tests are “inappropriate, meaningless, or both” (p. 954).”
BackgroundZachariae, R., Lyby, M. S., Ritterband, L. M., & O'Toole, M. S. (2016) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2015.10.004 · Cited by 704
Metadata
- Authors
- Ioannidis, J. P. A.
- Affiliations
- University of Ioannina (GR)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- Journal of Evaluation in Clinical Practice
- Publisher
- Wiley
- Published
- 2008-10
- Volume
- 14
- Issue
- 5
- Pages
- 951-957
- ISSN
- 1356-1294 (ISSN-L)
- 1365-2753
- OpenAlex
- W2144112749
- Cited by
- 696
- References
- 57
- Citations by year
- 2026: 52 · 2025: 69 · 2024: 98 · 2023: 72 · 2022: 50 · 2021: 64 · 2020: 49 · 2019: 34 · 2018: 30 · 2017: 23 · 2016: 23 · 2015: 26 · 2014: 27 · 2013: 22 · 2012: 15
- FWCI
- 4.52
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Evaluation in Clinical Practice (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- heterogeneity testing, publication bias, meta-analysis, statistical heterogeneity, clinical heterogeneity, false positive signals, bias detection
- MeSH terms
- Data Interpretation, Statistical, Humans, Regression Analysis, Research Design, Risk, Sensitivity and Specificity, Meta-Analysis as Topic, Reproducibility of Results, Bias, Confidence Intervals, Publication Bias, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
