Impact of analysing continuous outcomes using final values, change scores and analysis of covariance on the performance of meta‐analytic methods: a simulation study
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
When meta‐analysing intervention effects calculated from continuous outcomes, meta‐analysts often encounter few trials, with potentially a small number of participants, and a variety of trial analytical methods. It is important to know how these factors affect the performance of inverse‐variance fixed and DerSimonian and Laird random effects meta‐analytical methods. We examined this performance using a simulation study. Meta‐analysing estimates of intervention effect from final values, change scores, ANCOVA or a random mix of the three yielded unbiased estimates of pooled intervention effect. The impact of trial analytical method on the meta‐analytic performance measures was important when there was no or little heterogeneity, but was of little relevance as heterogeneity increased. On the basis of larger than nominal type I error rates and poor coverage, the inverse‐variance fixed effect method should not be used when there are few small trials. When there are few small trials, random effects meta‐analysis is preferable to fixed effect meta‐analysis. Meta‐analytic estimates need to be cautiously interpreted; type I error rates will be larger than nominal, and confidence intervals will be too narrow. Use of trial analytical methods that are more efficient in these circumstances may have the unintended consequence of further exacerbating these issues. © 2015 The Authors. Research Synthesis Methods published by John Wiley & Sons, Ltd. © 2015 The Authors. Research Synthesis Methods published by John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deeks, J. J., Macaskill, P., & Irwig, L. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.01.016 · Cited by 2,910 · cited 1×
“When the size of the trials are small, an alternative weighting system may be preferable, such as weighting by the number of participants (Senn, 2000; Senn, 2007b) or the effective sample size (Deeks et al., 2005).”
BackgroundSelf-citationA Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569 · cited 1×
“…additional justification for the recommendations of others that the decision of whether to synthesise the results, or choice of meta-analytic model to use if the results are synthesised, should not be based on the I2 statistic (Ioannidis et al., 2008; Rucker et al., 2008; Riley et al., 2011).”
BackgroundUndue reliance on I 2 in assessing heterogeneity may mislead
Rücker, G., Schwarzer, G., Carpenter, J. R., & Schumacher, M. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-79 · Cited by 1,287
Turner, R. M., Davey, J., Clarke, M. J., Thompson, S. G., & Higgins, J. P. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dys041 · Cited by 837
Van Breukelen, G. J. P. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.02.007 · Cited by 765
A comparison of statistical methods for meta‐analysis
Brockwell, S. E., & Gordon, I. R. (2001) · Statistics in Medicine · doi:10.1002/sim.650 · Cited by 573 · cited 3×
“These methods are often more complex to implement and may still be problematic when there are a small number of trials (Brockwell and Gordon, 2001; Borenstein et al., 2010).”
BackgroundMethodOn tests of the overall treatment effect in meta-analysis with normally distributed responses
Hartung, J., & Knapp, G. (2001) · Statistics in Medicine · doi:10.1002/sim.791 · Cited by 543
Reasons or excuses for avoiding meta-analysis in forest plots
Ioannidis, J. P. A., Patsopoulos, N. A., & Rothstein, H. R. (2008) · BMJ · doi:10.1136/bmj.a117 · Cited by 457 · cited 1×
“…additional justification for the recommendations of others that the decision of whether to synthesise the results, or choice of meta-analytic model to use if the results are synthesised, should not be based on the I2 statistic (Ioannidis et al., 2008; Rucker et al., 2008; Riley et al., 2011).”
BackgroundDavey, J., Turner, R. M., Clarke, M. J., & Higgins, J. P. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-160 · Cited by 404 · cited 4×
“, 2011)) and the number of trials per meta-analysis (which are generally few) (Davey et al., 2011).”
BackgroundMethodInfluentialA comparison of heterogeneity variance estimators in combining results of studies
Sidik, K., & Jonkman, J. N. (2007) · Statistics in Medicine · doi:10.1002/sim.2688 · Cited by 380
Assessing the impact of attrition in randomized controlled trials
Hewitt, C. E., Kumaravel, B., Dumville, J. C., & Torgerson, D. J. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.01.010 · Cited by 139
Research works printed but not linked: 36
- Armitage P, 2002, Statistical Methods in Medical Research, 10.1002/9780470773666 · DOI · Paper page
- Banerjee S, 2008, Caveats in the Meta‐analyses of Continuous Data: A Simulation Study
- 10.1002/pon.3205 · DOI · Paper page
- 10.1201/9781420035926 · DOI · Paper page
- 10.1002/jrsm.12 · DOI · Paper page
- 10.1016/j.jclinepi.2007.02.006 · DOI · Paper page
- 10.1002/sim.2673 · DOI · Paper page
- Cohen J, 1988, Statistical Power Analysis for the Behavioural Sciences
- 10.1016/0197-2456(86)90046-2 · DOI · Paper page
- 10.1111/j.0006-341X.1999.00732.x · DOI · Paper page
- 10.1002/sim.4780111304 · DOI · Paper page
- 10.1111/j.0006-341X.2004.00235.x · doi:10.1111/j.0006-341x.2004.00235.x
- 10.1002/(SICI)1097-0258(19960330)15:6<619::AID-SIM188>3.0.CO;2-A · DOI · Paper page
- 10.1002/(SICI)1521-4036(199912)41:8<901::AID-BIMJ901>3.0.CO;2-W · DOI
- 10.1002/sim.1186 · DOI · Paper page
- Larsen RJ, 1986, An Introduction to Mathematical Statistics and Its Applications
- MckenzieJE.2011.Methodological issues in meta‐analysis of randomised controlled trials with continuous outcomes (PhD thesis). Doctor of Philosophy Monash University.
- 10.1002/(SICI)1097-0258(19990215)18:3<321::AID-SIM28>3.0.CO;2-P · DOI · Paper page
- 10.6028/jres.087.022 · DOI · Paper page
- Raudenbush SW, 2009, The Handbook of Research Synthesis and Meta‐analysis
- 10.1016/j.jclinepi.2010.08.002 · DOI
- 10.1002/sim.5726 · DOI · Paper page
- 10.1037/1082-989X.13.1.31 · DOI · Paper page
- 10.1016/S0197-2456(00)00105-7 · doi:10.1016/s0197-2456(00)00105-7
- 10.1002/sim.2682 · DOI · Paper page
- 10.1002/9780470723586 · DOI · Paper page
- 10.1002/9780470723586.ch16 · doi:10.1002/9780470723586.ch16
- 10.1002/sim.4780080410 · DOI · Paper page
- 10.1080/10543409308835065 · DOI
- Sharp S, 1997, sbe16: meta‐analysis, Stata Technical Bulletin, 38, 9
- Sharp S, 1998, sbe16.1: new syntax and output for the meta‐analysis command, Stata Technical Bulletin, 42, 6
- Sharp S, 1998, sbe16.2: corrections to the meta‐analysis command, Stata Technical Bulletin, 43, 15
- 10.1111/j.1467-9876.2005.00489.x · DOI · Paper page
- STATACORP, 2007, Stata Statistical Software: Release 10
- 10.1177/009286150103500417 · DOI
- Ziegler S, 2001, Deficits and remedy of the standard random effects methods in meta‐analysis, Methods of Information in Medicine, 40, 148, 10.1055/s-0038-1634478 · DOI
Cited by
Galante, J., Friedrich, C., Dawson, A. F., Modrego-Alarcón, M., Gebbing, P., Delgado-Suárez, I., Gupta, R., Dean, L., Dalgleish, T., White, I. R., et al. (2021) · PLOS Medicine · doi:10.1371/journal.pmed.1003481 · Cited by 319
“calculated SMD using the ANCOVA estimate [51].”
Brown, T. J., Brainard, J., Song, F., Wang, X., Abdelhamid, A., & Hooper, L. (2019) · BMJ · doi:10.1136/bmj.l4697 · Cited by 313
Cregg, D. R., & Cheavens, J. S. (2021) · Journal of Happiness Studies · doi:10.1007/s10902-020-00236-6 · Cited by 203
Galante, J., Friedrich, C., Collaboration of Mindfulness Trials (CoMinT), Aeamla-Or, N., Arts-de Jong, M., Barrett, B., Bögels, S. M., Buitelaar, J. K., Checovich, M. M., Christopher, M. S., et al. (2023) · Nature Mental Health · doi:10.1038/s44220-023-00081-5 · Cited by 113
“We used the ANCOVA estimate as effect measure (final score adjusted for baseline score and the available pre-specified effect modifiers) 54,98 .”
Metadata
- Authors
- McKenzie, J. E., Herbison, G. P., & Deeks, J. J.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2015-12-29
- Volume
- 7
- Issue
- 4
- Pages
- 371-386
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2219274198
- Cited by
- 40
- References
- 11
- FWCI
- 1.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Funders
- University of Otago
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.