Dealing with clinical heterogeneity in meta‐analysis. Assumptions, methods, interpretation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Objective: There is an ongoing debate how to interpret findings of meta‐analyses when substantial clinical heterogeneity is present among included trials. The aim of the present study was to demonstrate various ways of dealing with clinical heterogeneity along with underlying assumptions and interpretation. A recent meta‐analysis on long‐term psychodynamic psychotherapy (LTPP) was used as an illustrative example. Method: Re‐analysis of published data including calculation of a prediction interval, heterogeneity tests, Bayesian meta‐analysis, meta‐regression, and subgroup analysis to explore and interpret summary estimates in clinically heterogeneous studies. Results: Meta‐analytic results and their implications varied considerably depending on whether and how clinical heterogeneity was addressed. Conclusions: Whether or not to trust summary estimates in meta‐analysis depends largely on the subjective relevance of clinical heterogeneity present. No single analysis and interpretation strategy can be valid in every context or paradigm, thus, reflection of own beliefs on the role of heterogeneity is needed. Copyright © 2013 John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380 · cited 1×
“Whether heterogeneity is present is commonly tested with Cochran’s Q test statistic that examines the null hypothesis that all studies estimate the same effect (Cochran, 1954).”
BackgroundRealist review - a new method of systematic review designed for complex policy interventions
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308530 · Cited by 3,190 · cited 2×
“…treatments are likely to have at least as much in common with complex health services interventions (Craig et al., 2008) as with pharmacological clinical trials, the so called “realist review” methodology may also present an alternative to mechanic meta-analysis (Pawson et al., 2005).”
BackgroundInterpretation of random effects meta-analyses
Riley, R. D., Higgins, J. P. T., & Deeks, J. J. (2011) · BMJ · doi:10.1136/bmj.d549 · Cited by 2,884
A Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569 · cited 1×
“…true treatment effect, while random-effects models assume that no single true treatment effect exists, but each trial estimates a trial-specific true treatment effect, with these effects following a certain (commonly normal) distribution (Lau et al., 1998; Higgins et al., 2009; Riley et al., 2011).”
BackgroundRandom-effects model for meta-analysis of clinical trials: An update
DerSimonian, R., & Kacker, R. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.04.004 · Cited by 2,374
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · cited 1×
“Several authors were interested in the overall effectiveness of psychotherapeutic treatments and contributed to the development of the methodological-statistical foundations of meta-analysis (Smith et al., 1980; Lipsey and Wilson, 1993).”
BackgroundStatistics notes: Absence of evidence is not evidence of absence
Altman, D. G., & Bland, J. M. (1995) · BMJ · doi:10.1136/bmj.311.7003.485 · Cited by 1,946
Systematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693
Systematic Review: Why sources of heterogeneity in meta-analysis should be investigated
Thompson, S. G. (1994) · BMJ · doi:10.1136/bmj.309.6965.1351 · Cited by 1,017 · cited 5×
“…heterogeneity description should be given priority over inference, not only because an “absence of evidence is not evidence of absence” (Altman and Bland, 1995), but also because heterogeneity tests frequently lack power to detect “even a moderate degree of genuine heterogeneity” (Thompson, 1994).”
BackgroundMethodInfluentialYusuf, S. (1991) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1991.03470010097038 · Cited by 1,014 · cited 1×
“On the other hand, one may prefer to look at certain subgroups in clinical trials (Yusuf et al., 1991).”
BackgroundEvaluating three treatments for borderline personality disorder: a multiwave study
Clarkin, J. F., Levy, K. N., Lenzenweger, M. F., & Kernberg, O. F. (2007) · American Journal of Psychiatry · doi:10.1176/ajp.2007.164.6.922 · Cited by 997 · cited 1×
“For example, both in the only trial performed in Cluster C personality disorders (Svartberg et al., 2004) and the only trial comparing LTPP with dialectic behavioural therapy (Clarkin et al., 2007) the comparator treatments were similar in duration to the administered LTPP.”
ResultIoannidis, J. P. A. (2009) · Canadian Medical Association Journal · doi:10.1503/cmaj.081086 · Cited by 716 · cited 1×
“Further lumping is possible in the form of overviews of reviews or so called “umbrella” reviews (Whitlock et al., 2008; Ioannidis, 2009; Caldwell et al., 2010; Higgins and Green, 2011).”
BackgroundBayesian approaches to random-effects meta-analysis: a comparative study
Smith, T. C., Spiegelhalter, D. J., & Thomas, A. (1995) · Statistics in Medicine · doi:10.1002/sim.4780142408 · Cited by 568
Invited commentary: a critical look at some popular meta-analytic methods
Greenland, S. (1994) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117248 · Cited by 511
Formulating Questions and Locating Primary Studies for Inclusion in Systematic Reviews
Counsell, C. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-5-199709010-00008 · Cited by 505 · cited 1×
“On the contrary, “splitters” prefer formulating narrow research questions and focus their analyses on clinically homogeneous groups of trials (Counsell, 1997).”
BackgroundSvartberg, M., Stiles, T. C., & Seltzer, M. H. (2004) · American Journal of Psychiatry · doi:10.1176/appi.ajp.161.5.810 · Cited by 398 · cited 1×
“For example, both in the only trial performed in Cluster C personality disorders (Svartberg et al., 2004) and the only trial comparing LTPP with dialectic behavioural therapy (Clarkin et al., 2007) the comparator treatments were similar in duration to the administered LTPP.”
ResultMeta-analysis. Beyond the grand mean?
Smith, G. D., Egger, M., & Phillips, A. N. (1997) · BMJ · doi:10.1136/bmj.315.7122.1610 · Cited by 174
Using Existing Systematic Reviews in Complex Systematic Reviews
Whitlock, E. P., Lin, J. S., Chou, R., Shekelle, P., & Robinson, K. A. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-148-10-200805200-00010 · Cited by 144
Pladevall-Vila, M., Delclos, G. L., Varas, C., Guyer, H., BruguEs-Tarradellas, J., & Anglada-Arisa, A. (1996) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a008846 · Cited by 74
Effectiveness of Long-term Psychodynamic Psychotherapy: A Meta-analysis
Leichsenring, F. (2008) · JAMA · doi:10.1001/jama.300.13.1551 · Cited by 65 · cited 3×
“That this question is considered relevant is reflected by several reactions on the first meta-analysis on LTPP by Leichsenring and Rabung (2008), which used similarly broad criteria (Beck, 2009; Kriston et al., 2009; Bhar et al., 2010).”
BackgroundLittell, J. H., & Shlonsky, A. (2011) · Clinical Social Work Journal · doi:10.1007/s10615-010-0308-z · Cited by 13
Cited by
Doi, S. A. R., Barendregt, J. J., Khan, S., Thalib, L., & Williams, G. M. (2015) · Contemporary Clinical Trials · doi:10.1016/j.cct.2015.05.009 · Cited by 581
Antimicrobial resistance in wildlife
Vittecoq, M., Godreuil, S., Prugnolle, F., Durand, P., Brazier, L., Renaud, N., Arnal, A., Aberkane, S., Jean‐Pierre, H., Gauthier‐Clerc, M., et al. (2016) · Journal of Applied Ecology · doi:10.1111/1365-2664.12596 · Cited by 284
Long-term effects of parental divorce on mental health – A meta-analysis
Auersperg, F., Vlasak, T., Ponocny, I., & Barth, A. (2019) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2019.09.011 · Cited by 223
Linde, K., Sigterman, K., Kriston, L., Rucker, G., Jamil, S., Meissner, K., & Schneider, A. (2015) · The Annals of Family Medicine · doi:10.1370/afm.1719 · Cited by 196
Self-citationLim, S. M., Shiau, C. W. C., Cheng, L. J., & Lau, Y. (2022) · Behavior Therapy · doi:10.1016/j.beth.2021.09.007 · Cited by 163
Metadata
- Authors
- Kriston, L.
- Venue
- International Journal of Methods in Psychiatric Research
- Publisher
- Wiley
- Published
- 2013-03
- Volume
- 22
- Issue
- 1
- Pages
- 1-15
- ISSN
- 1049-8931 (ISSN-L)
- 1557-0657
- OpenAlex
- W1536115762
- Cited by
- 94
- References
- 21
- FWCI
- 9.32
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.