Individual participant data meta-analyses should not ignore clustering
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
OBJECTIVES: Individual participant data (IPD) meta-analyses often analyze their IPD as if coming from a single study. We compare this approach with analyses that rather account for clustering of patients within studies. STUDY DESIGN AND SETTING: Comparison of effect estimates from logistic regression models in real and simulated examples. RESULTS: The estimated prognostic effect of age in patients with traumatic brain injury is similar, regardless of whether clustering is accounted for. However, a family history of thrombophilia is found to be a diagnostic marker of deep vein thrombosis [odds ratio, 1.30; 95% confidence interval (CI): 1.00, 1.70; P = 0.05] when clustering is accounted for but not when it is ignored (odds ratio, 1.06; 95% CI: 0.83, 1.37; P = 0.64). Similarly, the treatment effect of nicotine gum on smoking cessation is severely attenuated when clustering is ignored (odds ratio, 1.40; 95% CI: 1.02, 1.92) rather than accounted for (odds ratio, 1.80; 95% CI: 1.29, 2.52). Simulations show models accounting for clustering perform consistently well, but downwardly biased effect estimates and low coverage can occur when ignoring clustering. CONCLUSION: Researchers must routinely account for clustering in IPD meta-analyses; otherwise, misleading effect estimates and conclusions may arise.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Beta blockade during and after myocardial infarction: An overview of the randomized trials
Yusuf, S., Peto, R., Lewis, J., Collins, R., & Sleight, P. (1985) · Progress in Cardiovascular Diseases · doi:10.1016/s0033-0620(85)80003-7 · Cited by 3,205 · cited 1×
“The ManteleHaenszel and Peto methods have also been suggested to overcome this issue [42,43], but model (1) can more easily be extended to include multiple factors and continuous variables so is our preferred method.”
MethodMeta-analysis of individual participant data: rationale, conduct, and reporting
Riley, R. D., Lambert, P. C., & Abo-Zaid, G. (2010) · BMJ · doi:10.1136/bmj.c221 · Cited by 1,701
Self-citationMuch ado about nothing: a comparison of the performance of meta-analytical methods with rare events
Bradburn, M. J., Deeks, J. J., Berlin, J. A., & Russell Localio, A. (2007) · Statistics in Medicine · doi:10.1002/sim.2528 · Cited by 938
Confounding and Collapsibility in Causal Inference
Greenland, S., Pearl, J., & Robins, J. M. (1999) · Statistical Science · doi:10.1214/ss/1009211805 · Cited by 933 · cited 1×
“For logistic regression, this phenomenon is also known as noncollapsibility of the odds ratio [40] as conditional odds ratios are typically larger than marginal odds ratios after conditioning on important covariates, with the increase becoming higher as the true odds ratio increases and the number of included importan…”
BackgroundMeta‐Analysis Of Controlled Clinical Trials
Whitehead, A. (2002) · doi:10.1002/0470854200 · Cited by 827
Stijnen, T., Hamza, T. H., & Özdemir, P. (2010) · Statistics in Medicine · doi:10.1002/sim.4040 · Cited by 757 · cited 4×
“For binary outcome data, there may be some advantage of a one-step approach when the event risk or rate is low or the sample size is small; in contrast to the two-step approach, the one-step approach allows the exact binomial distribution to be used and does not require continuity corrections when zero events occur [1…”
BackgroundMethodResultInfluentialPatient age and outcome following severe traumatic brain injury: an analysis of 5600 patients
Hukkelhoven, C. W. P. M., Steyerberg, E. W., Rampen, A. J. J., Farace, E., Habbema, J. D. F., Marshall, L. F., Murray, G. D., & Maas, A. I. R. (2003) · Journal of Neurosurgery · doi:10.3171/jns.2003.99.4.0666 · Cited by 641 · cited 1×
“[23] performed a meta-analysis of 14 prospective studies to assess the 6-month mortality risk in patients with traumatic brain injury (TBI).”
MethodSelf-citationAhmed, I., Sutton, A. J., & Riley, R. D. (2012) · BMJ · doi:10.1136/bmj.d7762 · Cited by 404 · cited 1×
“Some covariates may not be available for all IPD studies, and IPD may not be available from all studies requested [45].”
BackgroundSelf-citation
Research works printed but not linked: 37
- Stewart, 1993, Meta-analysis of the literature or of individual patient data: is there a difference?, Lancet, 341, 418, 10.1016/0140-6736(93)93004-K · DOI · Paper page
- Stewart, 2002, To IPD or not to IPD? Advantages and disadvantages of systematic reviews using individual patient data, Eval Health Prof, 25, 76, 10.1177/0163278702025001006 · DOI · Paper page
- Simmonds, 2005, Meta-analysis of individual patient data from randomized trials: a review of methods used in practice, Clin Trials, 2, 209, 10.1191/1740774505cn087oa · DOI · Paper page
- Higgins, 2001, Meta-analysis of continuous outcome data from individual patients, Stat Med, 20, 2219, 10.1002/sim.918 · DOI · Paper page
- Whitehead, 2001, Meta-analysis of ordinal outcomes using individual patient data, Stat Med, 20, 2243, 10.1002/sim.919 · DOI · Paper page
- Tudur-Smith, 2005, Investigating heterogeneity in an individual patient data meta-analysis of time to event outcomes, Stat Med, 24, 1307, 10.1002/sim.2050 · DOI · Paper page
- Jones, 2009, Meta-analysis of individual patient data versus aggregate data from longitudinal clinical trials, Clin Trials, 6, 16, 10.1177/1740774508100984 · DOI · Paper page
- Riley, 2008, Meta-analysis of continuous outcomes combining individual patient data and aggregate data, Stat Med, 27, 1870, 10.1002/sim.3165 · DOI · Paper page
- Riley, 2008, Meta-analysis of diagnostic test studies using individual patient data and aggregate data, Stat Med, 27, 6111, 10.1002/sim.3441 · DOI · Paper page
- Olkin, 1998, Comparison of meta-analysis versus analysis of variance of individual patient data, Biometrics, 54, 317, 10.2307/2534018 · DOI · Paper page
- Mathew, 1999, On the equivalence of meta-analysis using literature and using individual patient data, Biometrics, 55, 1221, 10.1111/j.0006-341X.1999.01221.x · DOI · Paper page
- Tudur Smith, 2007, A comparison of methods for fixed effects meta-analysis of individual patient data with time to event outcomes, Clin Trials, 4, 621, 10.1177/1740774507085276 · DOI · Paper page
- Matthew, 2010, Comparison of one-step and two-step meta-analysis models using individual patient data, Biometrical J, 52, 271, 10.1002/bimj.200900143 · DOI · Paper page
- Hamza, 2008, The binomial distribution of meta-analysis was preferred to model within-study variability, J Clin Epidemiol, 61, 41, 10.1016/j.jclinepi.2007.03.016 · DOI · Paper page
- Abo-Zaid, 2012, Individual participant data meta-analysis of prognostic factor studies: state of the art?, BMC Med Res Methodol, 12, 56, 10.1186/1471-2288-12-56 · DOI · Paper page
- DerSimonian, 1986, Meta-analysis in clinical trials, Control Clin Trials, 7, 177, 10.1016/0197-2456(86)90046-2 · DOI · Paper page
- Carlsson, 1968, Factors affecting the clinical course of patients with severe head injuries. 1. Influence of biological factors. 2. Significance of posttraumatic coma, J Neurosurg, 29, 242, 10.3171/jns.1968.29.3.0242 · DOI · Paper page
- Kraaijenhagen, 2002, Simplification of the diagnostic management of suspected deep vein thrombosis, Arch Intern Med, 162, 907, 10.1001/archinte.162.8.907 · DOI · Paper page
- Toll, 2006, Excluding deep vein thrombosis safely in primary care, J Fam Pract, 55, 613
- Anderson, 2003, Combined use of clinical assessment and d-dimer to improve the management of patients presenting to the emergency department with suspected deep vein thrombosis (the EDITED Study), J Thromb Haemost, 1, 645, 10.1046/j.1538-7836.2003.00131.x · doi:10.1046/j.1538-7836.2003.00131.x
- Stevens, 2004, Withholding anticoagulation after a negative result on duplex ultrasonography for suspected symptomatic deep venous thrombosis, Ann Intern Med, 140, 985, 10.7326/0003-4819-140-12-200406150-00007 · DOI
- Wells, 2003, Evaluation of D-dimer in the diagnosis of suspected deep-vein thrombosis, N Engl J Med, 349, 1227, 10.1056/NEJMoa023153 · DOI · Paper page
- Toll, 2008, A new diagnostic rule for deep vein thrombosis: safety and efficiency in clinically relevant subgroups, Fam Pract, 25, 3, 10.1093/fampra/cmm075 · DOI
- Rice, 2001, Nursing interventions for smoking cessation, Cochrane Database Syst Rev (Complete Reviews), CD001188
- Altman, 2002, Meta-analysis, Simpson's paradox, and the number needed to treat, BMC Med Res Methodol, 2, 3, 10.1186/1471-2288-2-3 · DOI · Paper page
- Peters, 2003, Comparison of methods for analysing cluster randomized trials: an example involving a factorial design, Int J Epidemiol, 32, 840, 10.1093/ije/dyg228 · DOI · Paper page
- Bland, 2004, Cluster randomised trials in the medical literature: two bibliometric surveys, BMC Med Res Methodol, 4, 21, 10.1186/1471-2288-4-21 · DOI · Paper page
- Lee, 2005, The use of random effects models to allow for clustering in individually randomized trials, Clin Trials, 2, 163, 10.1191/1740774505cn082oa · DOI · Paper page
- Steyerberg, 2000, Clinical trials in acute myocardial infarction: should we adjust for baseline characteristics?, Am Heart J, 139, 745, 10.1016/S0002-8703(00)90001-2 · DOI
- Hernández, 2004, Covariate adjustment in randomized controlled trials with dichotomous outcomes increases statistical power and reduces sample size requirements, J Clin Epidemiol, 57, 454, 10.1016/j.jclinepi.2003.09.014 · DOI · Paper page
- Turner, 2012, Covariate adjustment increased power in randomized controlled trials: an example in traumatic brain injury, J Clin Epidemiol, 65, 474, 10.1016/j.jclinepi.2011.08.012 · DOI · Paper page
- Robinson, 1991, Some surprising results about covariate adjustment in logistic regression models, Int Stat Rev, 58, 227, 10.2307/1403444 · DOI · Paper page
- Gail, 1984, Biased estimates of treatment effect in randomized experiments with nonlinear regressions and omitted covariates, Biometrika, 71, 431, 10.1093/biomet/71.3.431 · DOI · Paper page
- Mantel, 1959, Statistical aspects of the analysis of data from retrospective studies of disease, J Natl Cancer Inst, 22, 719
- Chu, 2006, Bivariate meta-analysis of sensitivity and specificity with sparse data: a generalized linear mixed model approach, J Clin Epidemiol, 59, 1331, 10.1016/j.jclinepi.2006.06.011 · DOI · Paper page
- Riley, 2007, Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods, J Clin Epidemiol, 60, 431, 10.1016/j.jclinepi.2006.09.009 · DOI · Paper page
- Jackson, 2009, Systematically missing confounders in individual participant data meta-analysis of observational cohort studies, Stat Med, 28, 1218, 10.1002/sim.3540 · DOI · Paper page
Cited by
Stewart, L. A., Clarke, M., Rovers, M., Riley, R. D., Simmonds, M., Stewart, G., & Tierney, J. F. (2015) · JAMA · doi:10.1001/jama.2015.3656 · Cited by 2,134
Self-citationBurke, D. L., Ensor, J., & Riley, R. D. (2017) · Statistics in Medicine · doi:10.1002/sim.7141 · Cited by 538
Self-citationTierney, J. F., Vale, C., Riley, R., Smith, C. T., Stewart, L., Clarke, M., & Rovers, M. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001855 · Cited by 361
Self-citationGet real in individual participant data (IPD) meta‐analysis: a review of the methodology
Debray, T. P. A., Moons, K. G. M., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H. H., Reitsma, J. B., & on behalf of the GetReal methods review group (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1160 · Cited by 348
“Researchers typically allow for heterogeneous study effects by estimating a separate intercept αi in each study (Abo-Zaid et al., 2013; Higgins et al., 2001; Thompson et al., 2001; Turner et al., 2000) or by specifying a random-effects distribution on α (Broström and Holmberg, 2011; Simmonds and Higgins, 2007; Goldste…”
BackgroundSelf-citationGaksch, M., Jorde, R., Grimnes, G., Joakimsen, R., Schirmer, H., Wilsgaard, T., Mathiesen, E. B., Njølstad, I., Løchen, M.-L., März, W., et al. (2017) · PLOS ONE · doi:10.1371/journal.pone.0170791 · Cited by 348
Metadata
- Authors
- Abo-Zaid, G., Guo, B., Deeks, J. J., Debray, T. P. A., Steyerberg, E. W., Moons, K. G. M., & Riley, R. D.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2013-08
- Volume
- 66
- Issue
- 8
- Pages
- 865-873.e4
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2110032017
- Cited by
- 130
- References
- 8
- FWCI
- 10.04
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- creativecommons.org/licenses/by/3.0/ (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.