Empirical evidence about inconsistency among studies in a pair‐wise meta‐analysis
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
This paper investigates how inconsistency (as measured by the I 2 statistic) among studies in a meta‐analysis may differ, according to the type of outcome data and effect measure. We used hierarchical models to analyse data from 3873 binary, 5132 continuous and 880 mixed outcome meta‐analyses within the Cochrane Database of Systematic Reviews. Predictive distributions for inconsistency expected in future meta‐analyses were obtained, which can inform priors for between‐study variance. Inconsistency estimates were highest on average for binary outcome meta‐analyses of risk differences and continuous outcome meta‐analyses. For a planned binary outcome meta‐analysis in a general research setting, the predictive distribution for inconsistency among log odds ratios had median 22% and 95% CI: 12% to 39%. For a continuous outcome meta‐analysis, the predictive distribution for inconsistency among standardized mean differences had median 40% and 95% CI: 15% to 73%. Levels of inconsistency were similar for binary data measured by log odds ratios and log relative risks. Fitted distributions for inconsistency expected in continuous outcome meta‐analyses using mean differences were almost identical to those using standardized mean differences. The empirical evidence on inconsistency gives guidance on which outcome measures are most likely to be consistent in particular circumstances and facilitates Bayesian meta‐analysis with an informative prior for heterogeneity. © 2015 The Authors. Research Synthesis Methods published by John Wiley & Sons, Ltd. © 2015 The Authors. Research Synthesis Methods published by John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
General methods for monitoring convergence of iterative simulations
Brooks, S. P., & Gelman, A. (1998) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1998.10474787 · Cited by 6,045
A Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569 · cited 1×
“We can allow for unexplained heterogeneity in a random-effects meta-analysis, estimating a between-study variance and a summary effect (Higgins et al , 2009).”
Self-citationImproved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075 · cited 1×
“The Knapp and Hartung adjustment (Knapp and Hartung, 2003) and Bayesian modelling provide approaches that better capture the uncertainty associated with statistical heterogeneity.”
Undue reliance on I 2 in assessing heterogeneity may mislead
Rücker, G., Schwarzer, G., Carpenter, J. R., & Schumacher, M. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-79 · Cited by 1,287
Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & the Reporting Bias Group (2013) · PLoS ONE · doi:10.1371/journal.pone.0066844 · Cited by 1,127 · cited 1×
“…and there is inconsistency in the direction of effect, it would be misleading to give an average estimate for the intervention effect, particularly if study designs, within-study biases, and reporting biases are not taken into account (Turner et al , 2009; Welton et al , 2009; Dwan et al , 2013).”
Turner, R. M., Davey, J., Clarke, M. J., Thompson, S. G., & Higgins, J. P. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dys041 · Cited by 837 · cited 8×
“The classi fi cations of outcome types, types of intervention comparison and therapeutic area enabled Turner et al (2012) and Rhodes et al (2015) to explore how meta-analysis characteristics in fl uence the extent of between-study heterogeneity in meta-analyses of binary and continuous outcome data.”
InfluentialSelf-citationDeeks, J. J. (2002) · Statistics in Medicine · doi:10.1002/sim.1188 · Cited by 660 · cited 4×
“Engels et al (2000) compared analyses of odds ratios and risk differences in 125 meta-analyses, and Deeks (2002) additionally explored heterogeneity among relative risks in 551 meta-analyses.”
InfluentialA comparison of statistical methods for meta‐analysis
Brockwell, S. E., & Gordon, I. R. (2001) · Statistics in Medicine · doi:10.1002/sim.650 · Cited by 573 · cited 1×
“…procedure is likely to underestimate the between-study variance, particularly when the number of studies is small and there is substantial heterogeneity between studies (Brockwell and Gordon, Brockwell and Gordon, 2007; Sidik and Jonkman, 2005; Sidik and Jonkman, 2007; Hartung and Makambi, 2003).”
Bayesian approaches to random-effects meta-analysis: a comparative study
Smith, T. C., Spiegelhalter, D. J., & Thomas, A. (1995) · Statistics in Medicine · doi:10.1002/sim.4780142408 · Cited by 568 · cited 1×
“The Bayesian random-effects model of Smith et al (1995) was used for analyses of log odds ratios.”
Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change
Cornell, J. E., Mulrow, C. D., Localio, R., Stack, C. B., Meibohm, A. R., Guallar, E., & Goodman, S. N. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2886 · Cited by 453 · cited 1×
“When it is appropriate to combine studies whose estimates differ considerably, methods that better account for uncertainty than the DL procedure are recommended (Cornell et al , 2014).”
Cited by
Qin, P., Li, Q., Zhao, Y., Chen, Q., Sun, X., Liu, Y., Li, H., Wang, T., Chen, X., Zhou, Q., et al. (2020) · European Journal of Epidemiology · doi:10.1007/s10654-020-00655-y · Cited by 323
Leucht, S., Schneider‐Thoma, J., Burschinski, A., Peter, N., Wang, D., Dong, S., Huhn, M., Nikolakopoulou, A., Salanti, G., & Davis, J. M. (2023) · World Psychiatry · doi:10.1002/wps.21089 · Cited by 78
Wu, A.-X., Wang, X.-J., Zhao, C., Hu, J.-Q., Li, J.-W., Xu, Y., Li, Y., & Liu, S. (2026) · Pharmacological Research · doi:10.1016/j.phrs.2026.108178 · Cited by 2
Salanti, G., & Higgins, J. P. T. (2022) · Systematic Reviews in Health Research · doi:10.1002/9781119099369.ch13
Self-citation
Metadata
- Authors
- Rhodes, K. M., Turner, R. M., & Higgins, J. P. T.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2015-12-17
- Volume
- 7
- Issue
- 4
- Pages
- 346-370
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2218942987
- Cited by
- 66
- References
- 13
- FWCI
- 2.39
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Empirical evidence about inconsistency among studies in a pair‐wise meta‐analysis. https://app.cochat.ai/papers/10.1002/jrsm.1193
- The paper
- Rhodes, K. M., Turner, R. M., & Higgins, J. P. T. (2016). Empirical evidence about inconsistency among studies in a pair‐wise meta‐analysis. Research Synthesis Methods. https://doi.org/10.1002/jrsm.1193
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.