Comparison of aggregate and individual participant data approaches to meta-analysis of randomised trials: An observational study
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
BACKGROUND: It remains unclear when standard systematic reviews and meta-analyses that rely on published aggregate data (AD) can provide robust clinical conclusions. We aimed to compare the results from a large cohort of systematic reviews and meta-analyses based on individual participant data (IPD) with meta-analyses of published AD, to establish when the latter are most likely to be reliable and when the IPD approach might be required. METHODS AND FINDINGS: We used 18 cancer systematic reviews that included IPD meta-analyses: all of those completed and published by the Meta-analysis Group of the MRC Clinical Trials Unit from 1991 to 2010. We extracted or estimated hazard ratios (HRs) and standard errors (SEs) for survival from trial reports and compared these with IPD equivalents at both the trial and meta-analysis level. We also extracted or estimated the number of events. We used paired t tests to assess whether HRs and SEs from published AD differed on average from those from IPD. We assessed agreement, and whether this was associated with trial or meta-analysis characteristics, using the approach of Bland and Altman. The 18 systematic reviews comprised 238 unique trials or trial comparisons, including 37,082 participants. A HR and SE could be generated for 127 trials, representing 53% of the trials and approximately 79% of eligible participants. On average, trial HRs derived from published AD were slightly more in favour of the research interventions than those from IPD (HRAD to HRIPD ratio = 0.95, p = 0.007), but the limits of agreement show that for individual trials, the HRs could deviate substantially. These limits narrowed with an increasing number of participants (p < 0.001) or a greater number (p < 0.001) or proportion (p < 0.001) of events in the AD. On average, meta-analysis HRs from published AD slightly tended to favour the research interventions whether based on fixed-effect (HRAD to HRIPD ratio = 0.97, p = 0.088) or random-effects (HRAD to HRIPD ratio = 0.96, p = 0.044) models, but the limits of agreement show that for individual meta-analyses, agreement was much more variable. These limits tended to narrow with an increasing number (p = 0.077) or proportion of events (p = 0.11) in the AD. However, even when the information size of the AD was large, individual meta-analysis HRs could still differ from their IPD equivalents by a relative 10% in favour of the research intervention to 5% in favour of control. We utilised the results to construct a decision tree for assessing whether an AD meta-analysis includes sufficient information, and when estimates of effects are most likely to be reliable. A lack of power at the meta-analysis level may have prevented us identifying additional factors associated with the reliability of AD meta-analyses, and we cannot be sure that our results are generalisable to all outcomes and effect measures. CONCLUSIONS: In this study we found that HRs from published AD were most likely to agree with those from IPD when the information size was large. Based on these findings, we provide guidance for determining systematically when standard AD meta-analysis will likely generate robust clinical conclusions, and when the IPD approach will add considerable value.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965
Measuring agreement in method comparison studies
Bland, J. M., & Altman, D. G. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800204 · Cited by 9,619 · cited 2×
“This involved plotting the differences between the AD and IPD log HRs against each characteristic and testing for a non-zero regression slope for the average agreement and for non-constant limits of agreement [40].”
BackgroundMethodMeasuring agreement in method comparison studies
Bland, J. M., & Altman, D. G. (1999) · Statistical Methods in Medical Research · doi:10.1191/096228099673819272 · Cited by 9,619
Practical methods for incorporating summary time-to-event data into meta-analysis.
Tierney, J. F., Stewart, L. A., Ghersi, D., Burdett, S., & Sydes, M. R. (2007) · Trials · doi:10.1186/1745-6215-8-16 · Cited by 6,208
Self-citationAssessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755
Moher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545
Comparing methods of measurement: why plotting difference against standard method is misleading
Bland, J. M., & Altman, D. G. (1995) · The Lancet · doi:10.1016/s0140-6736(95)91748-9 · Cited by 2,343
Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis
Wetterslev, J., Thorlund, K., Brok, J., & Gluud, C. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.013 · Cited by 1,881
Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133 · cited 1×
“Only about half of the eligible trials were included in the AD meta-analyses, but these trials represented around 80% of participants, minimising the impact of selective outcome reporting bias [58] on our findings.”
BackgroundPage, M. J., Shamseer, L., Altman, D. G., Tetzlaff, J., Sampson, M., Tricco, A. C., Catalá-López, F., Li, L., Reid, E. K., Sarkis-Onofre, R., et al. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002028 · Cited by 768 · cited 2×
“From recent data [1], we estimate that approximately 1,200 oncology intervention reviews are published each year, which may be of variable quality, so we expect our findings to be of widespread use.”
BackgroundSterne, J. A., Egger, M., & Moher, D. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch10 · Cited by 651
NSCLC Meta-Analyses Collaborative Group (2008) · Journal of Clinical Oncology · doi:10.1200/jco.2008.17.7162 · Cited by 604
Röver, C., Knapp, G., & Friede, T. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0091-1 · Cited by 527
Pogue, J. M., & Yusuf, S. (1997) · Controlled Clinical Trials · doi:10.1016/s0197-2456(97)00051-2 · Cited by 415 · cited 1×
“Our results support the assertion that in order for a meta-analysis to be reliable, the information size should be at least as large as an adequately powered trial [46].”
ResultMichiels, S., Piedbois, P., Burdett, S., Syz, N., Stewart, L., & Pignon, J.-P. (2005) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462305050154 · Cited by 154 · cited 1×
“, median survival [45] or survival rates); 8 (10%) included a KM curve, but with insufficient information to estimate censoring; 15 (19%) presented survival results, but not for the specific treatment comparison and/or data sample of interest; and 6 (8%) reports could not be accessed.”
BackgroundSelf-citationBrouwer, I. A., Raitt, M. H., Dullemeijer, C., Kraemer, D. F., Zock, P. L., Morris, C., Katan, M. B., Connor, W. E., Camm, J. A., Schouten, E. G., et al. (2008) · European Heart Journal · doi:10.1093/eurheartj/ehp003 · Cited by 120
Baudard, M., Yavchitz, A., Ravaud, P., Perrodeau, E., & Boutron, I. (2017) · BMJ · doi:10.1136/bmj.j448 · Cited by 107
Cited by
ECMO for severe ARDS: systematic review and individual patient data meta-analysis
Combes, A., Peek, G. J., Hajage, D., Hardy, P., Abrams, D., Schmidt, M., Dechartres, A., & Elbourne, D. (2020) · Intensive Care Medicine · doi:10.1007/s00134-020-06248-3 · Cited by 428
Evaluation of various estimators for standardized mean difference in meta‐analysis
Lin, L., & Aloe, A. M. (2021) · Statistics in Medicine · doi:10.1002/sim.8781 · Cited by 297
Galante, J., Friedrich, C., Collaboration of Mindfulness Trials (CoMinT), Aeamla-Or, N., Arts-de Jong, M., Barrett, B., Bögels, S. M., Buitelaar, J. K., Checovich, M. M., Christopher, M. S., et al. (2023) · Nature Mental Health · doi:10.1038/s44220-023-00081-5 · Cited by 113
“As it allows for effect modification testing at the participant level, IPD meta-analysis is regarded as the ideal approach for estimating the modification effects of individual differences 22,23,54–57 .”
A framework for prospective, adaptive meta-analysis (FAME) of aggregate data from randomised trials.
Tierney, J. F., Fisher, D. J., Vale, C. L., Burdett, S., Rydzewska, L. H., Rogozińska, E., Godolphin, P. J., White, I. R., & Parmar, M. K. B. (2021) · PLOS Medicine · doi:10.1371/journal.pmed.1003629 · Cited by 31
“IPD may be required for a more thorough investigation of potential effect modifiers, other detailed or complex analyses [8], or to ensure reliable estimates of effect [16].”
BackgroundSelf-citation
Metadata
- Authors
- Tierney, J. F., Fisher, D. J., Burdett, S., Stewart, L. A., & Parmar, M. K. B.
- Venue
- PLOS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-01-31
- Volume
- 17
- Issue
- 1
- Pages
- e1003019
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- OpenAlex
- W3003392869
- Cited by
- 76
- References
- 18
- FWCI
- 2.91
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.