CoChat
Sign Up
Cited by 26Statistical Methods in Clinical TrialsStatistical Methods and Bayesian InferenceMeta-analysis and systematic reviews

A comparison of methods for analysing multiple outcome measures in randomised controlled trials using a simulation study

Open access
Vickerstaff, Victoria, Ambler, Gareth, Omar, Rumana Z.·2021·Biometrical Journal
DOI 10.1002/bimj.201900040PMC PMC7984364
Open this paper in CoChat — free accountOpen PDF

Abstract

Multiple primary outcomes are sometimes collected and analysed in randomised controlled trials (RCTs), and are used in favour of a single outcome. By collecting multiple primary outcomes, it is possible to fully evaluate the effect that an intervention has for a given disease process. A simple approach to analysing multiple outcomes is to consider each outcome separately, however, this approach does not account for any pairwise correlations between the outcomes. Any cases with missing values must be ignored, unless an additional imputation step is performed. Alternatively, multivariate methods that explicitly model the pairwise correlations between the outcomes may be more efficient when some of the outcomes have missing values. In this paper, we present an overview of relevant methods that can be used to analyse multiple outcome measures in RCTs, including methods based on multivariate multilevel (MM) models. We perform simulation studies to evaluate the bias in the estimates of the intervention effects and the power of detecting true intervention effects observed when using selected methods. Different simulation scenarios were constructed by varying the number of outcomes, the type of outcomes, the degree of correlations between the outcomes and the proportions and mechanisms of missing data. We compare multivariate methods to univariate methods with and without multiple imputation. When there are strong correlations between the outcome measures (ρ > .4), our simulation studies suggest that there are small power gains when using the MM model when compared to analysing the outcome measures separately. In contrast, when there are weak correlations (ρ < .4), the power is reduced when using univariate methods with multiple imputation when compared to analysing the outcome measures separately.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/bimj.201900040

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A single SEM can simultaneously estimate multiple dependent variables and their interdependences.

    “A single SEM can simultaneously estimate multiple dependent variables and their interdependences (Hair Jr. et al., 2010; Vickerstaff et al., 2021).”

    From The role of critical thinking on undergraduates' reliance behaviours on generative <scp>AI</scp> in problem‐solving (2025) · cites this paper as Vickerstaff et al., 2021

    The Statistical Methods > Multivariate Analysis section says that multivariate methods analyse multiple outcomes simultaneously while accounting for correlations between them. It discusses latent-variable and multilevel models, but does not identify these methods as SEM or specifically establish the claim about a single SEM. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. Agusti, A., & Vestbo, J. (2011). Current controversies and future perspectives in chronic obstructive pulmonary disease. American Journal of Respiratory and Critical Care Medicine, 184, 507– 513. 21680951
  2. Bretz, F., Hothorn, T., & Westfall, P. (2010). Multiple comparisons using R,,. Boca Raton, FL: CRC Press.
  3. Catalano, P. J., & Ryan, L. M. (1992). Bivariate latent variable models for clustered discrete and continuous outcomes. Journal of the American Statistical Association, 87, 651– 658.
  4. Chen, Y., & Hanson, T. (2017). Copula regression models for discrete and mixed bivariate responses. Journal of Statistical Theory and Practice, 11, 515– 530.
  5. De Leon, A. R., & Wu, B. (2011). Copula‐based regression models for a bivariate mixed discrete and continuous outcome. Statistics in Medicine, 30, 175– 185. 20963753
  6. De Los Reyes, A., Kundey, S. M. A., & Wang, M. (2011). The end of the primary outcome measure: A research agenda for constructing its replacement. Clinical Psychology Review, 31, 829– 838. 21545781
  7. Dmitrienko, A., & D'Agostino, R. (2013). Traditional multiplicity adjustment methods in clinical trials. Statistics in Medicine, 32, 5172– 5218. 24114861
  8. Dmitrienko, A., Tamhane, A. C., & Bretz, F. (2009). Multiple testing problems in pharmaceutical statistics. Boca Raton, FL: CRC Press.
  9. Dunson, D. B. (2000). Bayesian latent variable models for clustered mixed outcomes. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 62, 355– 366.
  10. Fitzmaurice, G. M., & Laird, N. M. (1995). Regression models for a bivariate discrete and continuous outcome with clustering. Journal of the American Statistical Association, 90, 845– 852.
  11. Gazzard, G., Konstantakopoulou, E., Garway‐Heath, D., Garg, A., Vickerstaff, V., Hunter, R., … Nathwani, N. (2019). Selective laser trabeculoplasty versus eye drops for first‐line treatment of ocular hypertension and glaucoma (LiGHT): A multicentre randomised controlled trial. Lancet, 393, 1505– 1516. 30862377 PMC6495367
  12. Goldstein, H. (2011). Multilevel statistical models. Hoboken, NJ: Wiley.
  13. Goldstein, H., Rasbash, J., Yang, M., Woodhouse, G., Pan, H., Nuttall, D., & Thomas, S. (1993). A multilevel analysis of school examination results. Oxford Review of Education, 19, 425– 433.
  14. Grilli, L., & Rampichini, C. (2006). A review of random effects modelling using gllamm in Stata. Department of Statistics, University of Florence.
  15. Hassiotis, A., Robotham, D., Canagasabey, A., Romeo, R., Langridge, D., Blizard, R., … King, M. (2009). Randomized, single‐blind, controlled trial of a specialist behavior therapy team for challenging behavior in adults with intellectual disabilities. American Journal of Psychiatry, 166, 1278– 1285. 19687128
  16. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65– 70.
  17. King, M., Davidson, O., Taylor, F., Haines, A., Sharp, D., & Turner, R. (2002). Effectiveness of teaching general practitioners skills in brief cognitive behaviour therapy to treat patients with depression: Randomised controlled trial. BMJ, 324, 947. 11964340 PMC102328
  18. Maas, J., Verheij, R. A., Spreeuwenberg, P., & Groenewegen, P. P. (2008). Physical activity as a possible mechanism behind the relationship between green space and health: A multilevel analysis. BMC Public Health, 8, 206. 18544169 PMC2438348
  19. Mayo, N. E., & Scott, S. (2011). Evaluating a complex intervention with a single outcome may not be a good idea: An example from a randomised trial of stroke case management. Age and Ageing, 40, 718– 724. 22016344
  20. Mcculloch, C. (2008). Joint modelling of mixed outcome types using latent variables. Statistical Methods in Medical Research, 17, 53– 73. 17855745
  21. Mohan, J., Twigg, L., & Taylor, J. (2011). Mind the double gap using multivariate multilevel modelling to investigate public perceptions of crime trends. British Journal of Criminology, 51, 1035– 1053.
  22. Montori, V. M., Permanyer‐Miralda, G., Ferreira‐González, I., Busse, J. W., Pacheco‐Huergo, V., Bryant, D., … Mills, E. (2005). Validity of composite end points in clinical trials. BMJ: British Medical Journal, 330, 594. 15761002 PMC554039
  23. O'Brien, P. (1984). Procedures for comparing samples with multiple endpoints. Biometrics, 40, 1079– 1087. 6534410
  24. Pituch, K. A., Whittaker, T. A., & Chang, W. (2016). Multivariate models for normal and binary responses in intervention studies. American Journal of Evaluation, 37, 270– 286.
  25. Pogue, J., Devereaux, P., Thabane, L., & Yusuf, S. (2012). Designing and analyzing clinical trials with composite outcomes: Consideration of possible treatment differences between the individual outcomes. PLoS One, 7, e34785. 22529934 PMC3328496
  26. Prentice, R. L., & Zhao, L. P. (1991). Estimating equations for parameters in means and covariances of multivariate discrete and continuous responses. Biometrics, 47(3), 825– 839. 1742441
  27. Rabe‐Hesketh, S., Skrondal, A., & Pickles, A. (2005). Maximum likelihood estimation of limited and discrete dependent variable models with nested random effects. Journal of Econometrics, 128, 301– 323.
  28. Roy, J., Lin, X., & Ryan, L. M. (2003). Scaled marginal models for multiple continuous outcomes. Biostatistics, 4, 371– 383. 12925505
  29. Rubin, D. B. (2004). Multiple imputation for nonresponse in surveys. Hoboken, NJ: Wiley.
  30. Sammel, M. D., Ryan, L. M., & Legler, J. M. (1997). Latent variable models for mixed discrete and continuous outcomes. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 59, 667– 678.
  31. Skrondal, A., & Rabe‐Hesketh, S. (2004). Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models. Boca Raton: CRC Press.
  32. Snijders, T., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). London: Sage Publishers.
  33. Sterne, J. A., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., … Carpenter, J. R. (2009). Multiple imputation for missing data in epidemiological and clinical research: Potential and pitfalls. BMJ, 338, b2393. 19564179 PMC2714692
  34. Sullivan, T. R., White, I. R., Salter, A. B., Ryan, P., & Lee, K. J. (2018). Should multiple imputation be the method of choice for handling missing data in randomized trials? Statistical Methods in Medical Research, 27, 2610– 2626. 28034175 PMC5393436
  35. Teixeira‐Pinto, A., & Harezlak, J. (2013). Factorization and latent variable models for joint analysis of binary and continuous outcomes. In A. R. de Leon & K. C. Chough (Eds.), Analysis of mixed data: Methods & applications, 1st, (pp. 81– 91). Boca Raton, FL: CRC Press, Taylor & Francis Group.
  36. Teixeira‐Pinto, A., & Normand, S.‐L. (2011). Missing data in regression models for non‐commensurate multiple outcomes. Revstat Statistical Journal, 9, 37– 55. 23504447 PMC3595565
  37. Teixeira‐Pinto, A., & Normand, S. L. T. (2008). Statistical methodology for classifying units on the basis of multiple‐related measures. Statistics in Medicine, 27, 1329– 1350. 18181221 PMC2278020
  38. Teixeira‐Pinto, A., & Normand, S. L. T. (2009). Correlated bivariate continuous and binary outcomes: Issues and applications. Statistics in Medicine, 28, 1753– 1773. 19358234 PMC2818753
  39. Teixeira‐Pinto, A., Siddique, J., Gibbons, R., & Normand, S.‐L. (2009). Statistical approaches to modeling multiple outcomes in psychiatric studies. Psychiatric Annals, 39, 729– 735. 20161512 PMC2798811
  40. Tseloni, A., & Zarafonitou, C. (2008). Fear of crime and victimization: A multivariate multilevel analysis of competing measurements. European Journal of Criminology, 5, 387– 409.
  41. Vickerstaff, V., Ambler, G., King, M., Nazareth, I., & Omar, R. Z. (2015). Are multiple primary outcomes analysed appropriately in randomised controlled trials? A review. Contemporary Clinical Trials, 45, 8– 12. 26215934
  42. White, I. R., Royston, P., & Wood, A. M. (2011). Multiple imputation using chained equations: Issues and guidance for practice. Statistics in Medicine, 30, 377– 399. 21225900
  43. Yang, M., Goldstein, H., Browne, W., & Woodhouse, G. (2002). Multivariate multilevel analyses of examination results. Journal of the Royal Statistical Society: Series A (Statistics in Society), 165, 137– 153.

Cited by

Sorted by citations

Authors
Vickerstaff, V., Ambler, G., & Omar, R. Z.
Affiliations
University College London (GB)
Corresponding author
Vickerstaff, Victoria
ORCID
Venue
Biometrical Journal
Publisher
Wiley
Published
2020-12-14
Volume
63
Issue
3
Pages
599-615
ISSN
  • 0323-3847 (ISSN-L)
  • 1521-4036
DOI
10.1002/bimj.201900040
PMC
PMC7984364
OpenAlex
W3112226428
Cited by
26
References
6
Citations by year
2026: 5 · 2025: 4 · 2024: 7 · 2023: 7 · 2022: 1 · 2021: 1 · 2020: 1
FWCI
0.41
Type
journal article
Language
en
Open access
Licence
Locations
  • Biometrical Journal (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • UCL Discovery (University College London) (submittedVersion)
Funders
  • National Institute for Health and Care Research
  • Marie Curie (MCCC-FCO-11-U)
  • University College London
  • Moorfields Eye Hospital NHS Foundation Trust
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
multiple imputation, missing data, statistical power, randomized controlled trials
MeSH terms
Computer Simulation, Data Interpretation, Statistical, Research Design, Bias, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review