The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Statistical heterogeneity in systematic reviews of clinical trials: a critical appraisal of guidelines and practice
Abstract
Objective: Heterogeneity between study results can be a problem in any systematic review or meta-analysis of clinical trials. Identifying its presence, investigating its cause and correctly accounting for it in analyses all involve difficult decisions for the researcher. Our objectives were: to collate recommendations on the subject of dealing with heterogeneity in systematic reviews of clinical trials; to investigate current practice in addressing heterogeneity in Cochrane reviews; and to compare current practice with recommendations. Methods: We review guidelines for those undertaking systematic reviews and examine how heterogeneity is addressed in practice in a sample of systematic reviews, and their protocols, from the Cochrane Database of Systematic Reviews. Results: Advice to reviewers is on the whole consistent and sensible. However, examination of a sample of Cochrane protocols and reviews demonstrates that the advice is difficult to follow given the small numbers of studies identified in many systematic reviews, the difficulty of pre-specifying important effect modifiers for subgroup analysis or meta-regression and the unresolved debate concerning fixed versus random effects metaanalyses. There was disagreement between protocols and reviews, often either regarding choice of important potential effect modifiers or due to the review identifying too few studies to perform planned analyses. Conclusion: Guidelines that address practical issues are required to reduce the risk of spurious findings from investigations of heterogeneity. This may involve discouraging statistical investigations such as subgroup analyses and meta-regression, rather than simply adopting a cautious approach to their interpretation, unless a large number of studies is available. The notion of a priori specification of potential effect modifiers for a retrospective review of studies is ill-defined, and the appropriateness of using a statistical test for heterogeneity to decide between analysis strategies is suspect.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Systematic Review: Why sources of heterogeneity in meta-analysis should be investigated
Thompson, S. G. (1994) · BMJ · doi:10.1136/bmj.309.6965.1351 · Cited by 1,017 · cited 1×
“Reasons for heterogeneity should be explored in order to increase scientific understanding and clinical relevance.(3) But despite the inevitability of heterogeneity, how it should be handled in systematic reviews is far from clear.”
BackgroundHeterogeneity in Meta-analysis of Data from Epidemiologic Studies: A Commentary
Colditz, G. A., Burdick, E., & Mosteller, F. (1995) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117644 · Cited by 245
Understanding research synthesis (meta-analysis).
Mosteller, F., & Colditz, G. A. (1996) · Annual Review of Public Health · doi:10.1146/annurev.pu.17.050196.000245 · Cited by 228
Invited commentary: benefits of heterogeneity in meta-analysis of data from epidemiologic studies
Berlin, J. A. (1995) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117645 · Cited by 191 · cited 1×
“This is despite the strong and, some would say, untenable assumption of equal underlying treatment effects on which this method is based.(6;26)”
ResultMeta-analysis. Beyond the grand mean?
Smith, G. D., Egger, M., & Phillips, A. N. (1997) · BMJ · doi:10.1136/bmj.315.7122.1610 · Cited by 174
Cited by
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Self-citationHow should meta‐regression analyses be undertaken and interpreted?
Thompson, S. G., & Higgins, J. P. T. (2002) · Statistics in Medicine · doi:10.1002/sim.1187 · Cited by 3,026
Self-citationInterpretation of random effects meta-analyses
Riley, R. D., Higgins, J. P. T., & Deeks, J. J. (2011) · BMJ · doi:10.1136/bmj.d549 · Cited by 2,884
Self-citationA Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569
“Although sources of heterogeneity should ideally be investigated (Thompson, 1994), selection of covariates to explain heterogeneity can be difficult, both practically (Higgins et al., 2002) and theoretically (Thompson and Higgins, 2002).”
BackgroundSelf-citationChan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
Self-citationControlling the risk of spurious findings from meta-regression
Higgins, J. P. T., & Thompson, S. G. (2004) · Statistics in Medicine · doi:10.1002/sim.1752 · Cited by 1,267
Self-citationEffect of lower sodium intake on health: systematic review and meta-analyses
Aburto, N. J., Ziolkovska, A., Hooper, L., Elliott, P., Cappuccio, F. P., & Meerpohl, J. J. (2013) · BMJ · doi:10.1136/bmj.f1326 · Cited by 1,242
Opioids in chronic non-cancer pain: systematic review of efficacy and safety
Kalso, E., Edwards, J. E., Moore, A. R., & McQuay, H. J. (2004) · Pain · doi:10.1016/j.pain.2004.09.019 · Cited by 1,216
Evaluation of networks of randomized trials
Salanti, G., Higgins, J. P., Ades, A., & Ioannidis, J. P. (2008) · Statistical Methods in Medical Research · doi:10.1177/0962280207080643 · Cited by 1,078
Self-citationHarbord, R. M., & Higgins, J. P. T. (2008) · The Stata Journal: Promoting communications on statistics and Stata · doi:10.1177/1536867x0800800403 · Cited by 897
Self-citationEarly palliative care for adults with advanced cancer
Haun, M. W., Estel, S., Rücker, G., Friederich, H.-C., Villalobos, M., Thomas, M., & Hartmann, M. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011129.pub2 · Cited by 757
Assessing Evidence Inconsistency in Mixed Treatment Comparisons
Lu, G., & Ades, A. E. (2006) · Journal of the American Statistical Association · doi:10.1198/016214505000001302 · Cited by 756
Jakobsen, J. C., Wetterslev, J., Winkel, P., Lange, T., & Gluud, C. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-120 · Cited by 704
Liu, W., Zhang, J., Liu, H., Guo, X., Zhang, X., Yao, X., Cao, Z., & Zhang, T. (2021) · Environment International · doi:10.1016/j.envint.2020.106277 · Cited by 688
The effect of sperm DNA fragmentation on miscarriage rates: a systematic review and meta-analysis
Robinson, L., Gallos, I. D., Conner, S. J., Rajkhowa, M., Miller, D., Lewis, S., Kirkman-Brown, J., & Coomarasamy, A. (2012) · Human Reproduction · doi:10.1093/humrep/des261 · Cited by 629
Aspirin for the prevention of preterm and term preeclampsia: systematic review and metaanalysis
Roberge, S., Bujold, E., & Nicolaides, K. H. (2018) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2017.11.561 · Cited by 628
Understanding heterogeneity in meta-analysis: the role of meta-regression
Baker, W. L., Michael White, C., Cappelleri, J. C., Kluger, J., Coleman, C. I., & From the Health Outcomes, P. (2009) · International Journal of Clinical Practice · doi:10.1111/j.1742-1241.2009.02168.x · Cited by 492
Cytokines in bipolar disorder vs. healthy control subjects: A systematic review and meta-analysis
Munkholm, K., Braüner, J. V., Kessing, L. V., & Vinberg, M. (2013) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2013.05.018 · Cited by 402
Bronsard, G., Alessandrini, M., Fond, G., Loundou, A., Auquier, P., Tordjman, S., & Boyer, L. (2016) · Medicine · doi:10.1097/md.0000000000002622 · Cited by 346
Limitations of meta-analyses of studies with high heterogeneity
Imrey, P. B. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.19325 · Cited by 319
Head injury and risk of Parkinson disease: A systematic review and meta‐analysis
Jafari, S., Etminan, M., Aminzadeh, F., & Samii, A. (2013) · Movement Disorders · doi:10.1002/mds.25458 · Cited by 282
Social support and subjective burden in caregivers of adults and older adults: A meta-analysis
del-Pino-Casado, R., Frías-Osuna, A., Palomino-Moral, P. A., Ruzafa-Martínez, M., & Ramos-Morcillo, A. J. (2018) · PLOS ONE · doi:10.1371/journal.pone.0189874 · Cited by 272
A meta-analytic review of the psychosocial adjustment of youth with inflammatory bowel disease
Greenley, R. N., Hommel, K. A., Nebel, J., Raboin, T., Li, S.-H., Simpson, P., & Mackner, L. (2010) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsp120 · Cited by 259
Bärnighausen, T., Chaiyachati, K., Chimbindi, N., Peoples, A., Haberer, J., & Newell, M.-L. (2011) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(11)70181-5 · Cited by 246
Oldershaw, A., Lavender, T., Sallis, H., Stahl, D., & Schmidt, U. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.04.005 · Cited by 222
Gagnier, J. J., Moher, D., Boon, H., Beyene, J., & Bombardier, C. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-111 · Cited by 222
Obstacles and opportunities in meta-analysis of genetic association studies
Salanti, G., Sanderson, S., & Higgins, J. P. T. (2005) · Genetics in Medicine · doi:10.1097/01.gim.0000151839.12032.1a · Cited by 196
Self-citationPsychological interventions for treatment of inflammatory bowel disease
Timmer, A., Preiss, J. C., Motschall, E., Rücker, G., Jantschek, G., & Moser, G. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006913.pub2 · Cited by 187
Can meta-analysis help target interventions at individuals most likely to benefit?
Thompson, S. G., & Higgins, J. P. (2005) · The Lancet · doi:10.1016/s0140-6736(05)70200-2 · Cited by 174
Self-citationA critical review and meta-analysis of the association between overt hyperthyroidism and mortality
Brandt, F., Green, A., Hegedüs, L., & Brix, T. H. (2011) · European Journal of Endocrinology · doi:10.1530/eje-11-0299 · Cited by 170
Casonatto, J., Goessler, K. F., Cornelissen, V. A., Cardoso, J. R., & Polito, M. D. (2016) · European Journal of Preventive Cardiology · doi:10.1177/2047487316664147 · Cited by 169
Bias Due to Changes in Specified Outcomes during the Systematic Review Process
Kirkham, J. J., Altman, D. G., & Williamson, P. R. (2010) · PLoS ONE · doi:10.1371/journal.pone.0009810 · Cited by 164
Self-citationBurnout in French physicians: A systematic review and meta-analysis
Kansoun, Z., Boyer, L., Hodgkinson, M., Villes, V., Lançon, C., & Fond, G. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.12.056 · Cited by 167
Ebrahim, S., Akl, E. A., Mustafa, R. A., Sun, X., Walter, S. D., Heels-Ansdell, D., Alonso-Coello, P., Johnston, B. C., & Guyatt, G. H. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.03.014 · Cited by 141
Bortolotti, B., Menchetti, M., Bellini, F., Montaguti, M. B., & Berardi, D. (2008) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2008.04.001 · Cited by 136
Serban, C., Sahebkar, A., Ursoniu, S., Andrica, F., & Banach, M. (2015) · Journal of Hypertension · doi:10.1097/hjh.0000000000000585 · Cited by 133
Single dose oral paracetamol (acetaminophen) for postoperative pain
Barden, J., Edwards, J., Moore, A., & McQuay, H. (2004) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004602 · Cited by 113
Atrioventricular Nodal Ablation in Atrial Fibrillation: A Meta-Analysis and Systematic Review
Chatterjee, N. A., Upadhyay, G. A., Ellenbogen, K. A., McAlister, F. A., Choudhry, N. K., & Singh, J. P. (2012) · Circulation Arrhythmia and Electrophysiology · doi:10.1161/circep.111.967810 · Cited by 109
Yang, W., Cai, X., Wu, H., & Ji, L. (2019) · Journal of Diabetes · doi:10.1111/1753-0407.12900 · Cited by 110
Systematic reviews and meta-analysis for the surgeon scientist
Mahid, S. S., Hornung, C. A., Minor, K. S., Turina, M., & Galandiuk, S. (2006) · British Journal of Surgery · doi:10.1002/bjs.5596 · Cited by 109
Álvarez-García, C., & Yaban, Z. Ş. (2020) · Complementary Therapies in Clinical Practice · doi:10.1016/j.ctcp.2019.101077 · Cited by 94
Poggio, R., Gutierrez, L., Matta, M. G., Elorriaga, N., Irazola, V., & Rubinstein, A. (2015) · Public Health Nutrition · doi:10.1017/s1368980014000949 · Cited by 92
Katsaras, G. N., Vouloumanou, E. K., Kourlaba, G., Kyritsi, E., Evagelou, E., & Bakoula, C. (2018) · Adolescent Research Review · doi:10.1007/s40894-018-0081-8 · Cited by 75
The Effect of Cognitive Behavioral Group Therapy for Depression: A Meta‐Analysis 2000–2010
Feng, C., Chu, H., Chen, C., Chang, Y., Chen, T., Chou, Y., Chang, Y., & Chou, K. (2012) · Worldviews on Evidence-Based Nursing · doi:10.1111/j.1741-6787.2011.00229.x · Cited by 66
Liu, H., Yang, M., Li, G.-M., Qiu, Y., Zheng, J., Du, X., Wang, J.-L., & Liu, R.-W. (2010) · Journal of the Neurological Sciences · doi:10.1016/j.jns.2010.04.001 · Cited by 51
Tsipa, A., O’Connor, D. B., Branley-Bell, D., Day, F., Hall, L. H., Sykes-Muskett, B., Wilding, S., Taylor, N., & Conner, M. (2021) · Health Psychology Review · doi:10.1080/17437199.2020.1760726 · Cited by 46
Montoro-Cárdenas, D., Cortés-Pérez, I., Ibancos-Losada, M., Zagalaz-Anula, N., Obrero-Gaitán, E., & Osuna-Pérez, M. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph191912343 · Cited by 46
The correlation between paper length and citations: a meta-analysis
Xie, J., Gong, K., Cheng, Y., & Ke, Q. (2019) · Scientometrics · doi:10.1007/s11192-019-03015-0 · Cited by 40
Messina, C., Giunta, E. F., Signori, A., Rebuzzi, S. E., Banna, G. L., Maniam, A., Buti, S., Cattrini, C., Fornarini, G., Bauckneht, M., et al. (2024) · European Urology Oncology · doi:10.1016/j.euo.2023.07.013 · Cited by 39
Wang, Q., & Mousavi, A. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13282 · Cited by 38
LIN, G. S. S., CHAN, D. Z. K., LEE, H. Y., LOW, T. T., LAER, T. S., PILLAI, M. P. M., YEW, Y. Q., & WAFA, S. W. W. S. S. T. (2022) · Journal of Evidence-Based Dental Practice · doi:10.1016/j.jebdp.2022.101723 · Cited by 33
Foodborne hazards in food in Burkina Faso, 1990–2019: a systematic review and meta-analysis
Dinede, G., Amenu, K., Alonso, S., Gazu, L., Mutua, F., Roesel, K., Lindahl, J. F., Sousa, F. M., Ulrich, P., Guadu, T., et al. (2023) · Frontiers in Sustainable Food Systems · doi:10.3389/fsufs.2023.1232992 · Cited by 12
Systematic reviews and meta-analysis in rheumatology: a gentle introduction for clinicians
Kelley, G. A., & Kelley, K. S. (2019) · Clinical Rheumatology · doi:10.1007/s10067-019-04590-6 · Cited by 8
Metadata
- Authors
- Higgins, J., Thompson, S., Deeks, J., & Altman, D.
- Venue
- Journal of Health Services Research & Policy
- Publisher
- SAGE Publications
- Published
- 2002-01-01
- Volume
- 7
- Issue
- 1
- Pages
- 51-61
- ISSN
- 1355-8196 (ISSN-L)
- 1758-1060
- OpenAlex
- W2081320515
- Cited by
- 755
- References
- 6
- FWCI
- 22.54
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
