The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data
Open accessAbstract
Multiple imputation can be used as a tool in the process of constructing prediction models in medical and epidemiological studies with missing covariate values. Such models can be used to make predictions for model performance assessment, but the task is made more complicated by the multiple imputation structure. We summarize various predictions constructed from covariates, including multiply imputed covariates, and either the set of imputation‐specific prediction model coefficients or the pooled prediction model coefficients. We further describe approaches for using the predictions to assess model performance. We distinguish between ideal model performance and pragmatic model performance , where the former refers to the model's performance in an ideal clinical setting where all individuals have fully observed predictors and the latter refers to the model's performance in a real‐world clinical setting where some individuals have missing predictors. The approaches are compared through an extensive simulation study based on the UK700 trial. We determine that measures of ideal model performance can be estimated within imputed datasets and subsequently pooled to give an overall measure of model performance. Alternative methods to evaluate pragmatic model performance are required and we propose constructing predictions either from a second set of covariate imputations which make no use of observed outcomes, or from a set of partial prediction models constructed for each potential observed pattern of covariate. Pragmatic model performance is generally lower than ideal model performance. We focus on model performance within the derivation data, but describe how to extend all the methods to a validation dataset.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A method for combining the m models is to use them to make predictions for the new person and then average these m predictions, conceptually similar to stacking in machine learning.
“Another method for combining the m models is to use them to make predictions for the new person and then average these m predictions,79 a procedure conceptually similar to stacking in machine learning.”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [79]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(1990) · Choice Reviews Online · doi:10.5860/choice.27-3338 · Cited by 35,305 · see citing works
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701 · see citing works
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,384 · see citing works
Multiple imputation using chained equations: Issues and guidance for practice
White, I. R., Royston, P., & Wood, A. M. (2011) · Statistics in Medicine · doi:10.1002/sim.4067 · Cited by 9,792 · see citing works
Self-citationKupper, L. L., Hosmer, D. W., & Lemeshow, S. A. (1990) · Journal of the American Statistical Association · doi:10.2307/2290035 · Cited by 7,690 · see citing works
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299 · see citing works
Self-citationAnalysis of Incomplete Multivariate Data
Schafer, J. L. (1997) · doi:10.1201/9781439821862 · Cited by 6,012 · see citing works
Analysis of Incomplete Multivariate Data
Booth, D. E., & Schafer, J. L. (2000) · Technometrics · doi:10.2307/1271466 · Cited by 5,627 · see citing works
Statistical Analysis With Missing Data
Ghosh, S., Little, R. J. A., & Rubin, D. B. (1988) · Technometrics · doi:10.2307/1269814 · Cited by 4,730 · see citing works
Steyerberg, E. W., Mushkudiani, N. A., Perel, P., Butcher, I., Lü, J., McHugh, G. S., Murray, G. D., Marmarou, A., Roberts, I., Habbema, J. D. F., et al. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050165 · Cited by 1,437 · see citing works
O'Mahony, C., Jichi, F., Pavlou, M., Monserrat, L., Anastasakis, A., Rapezzi, C., Biagini, E., Gimeno, J. R., Limongelli, G., McKenna, W. J., et al. (2014) · European Heart Journal · doi:10.1093/eurheartj/eht439 · Cited by 1,251 · see citing works
Risk prediction models: II. External validation, model updating, and impact assessment
Moons, K. G. M., Kengne, A. P., Grobbee, D. E., Royston, P., Vergouwe, Y., Altman, D. G., & Woodward, M. (2012) · Heart · doi:10.1136/heartjnl-2011-301247 · Cited by 1,195 · see citing works
Self-citationMissing-Data Adjustments in Large Surveys
Little, R. J. A. (1988) · Journal of Business and Economic Statistics · doi:10.1080/07350015.1988.10509663 · Cited by 1,109 · see citing works
Hippisley-Cox, J., Coupland, C., Vinogradova, Y., Robson, J., May, M., & Brindle, P. (2007) · British medical journal · doi:10.1136/bmj.39261.471806.55 · Cited by 1,070 · see citing works
Using the outcome for imputation of missing predictor values was preferred
Moons, K. G. M., Donders, R. A. R. T., Stijnen, T., & Harrell, F. E. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.009 · Cited by 994 · see citing works
Multiple Imputation and its Application
Carpenter, J. R., & Kenward, M. G. (2013) · doi:10.1002/9781119942283 · Cited by 913 · cited 1× · see citing works
“We (Wood et al., 2008; White et al., 2011) and others (Ambler et al., 2007; Vergouw et al., 2010; Carpenter and Kenward, 2013) have previously described multiple imputation methods to deal with missing values for prediction model derivation.”
MethodMoons, K. G. M., Kengne, A. P., Woodward, M., Royston, P., Vergouwe, Y., Altman, D. G., & Grobbee, D. E. (2012) · Heart · doi:10.1136/heartjnl-2011-301246 · Cited by 889 · see citing works
Self-citationMarshall, A., Altman, D. G., HOLDER, R. L., & Royston, P. J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-57 · Cited by 883 · see citing works
Self-citationTwo further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556 · see citing works
How should variable selection be performed with multiply imputed data?
Wood, A., White, I., & Royston, P. (2008) · Statistics in Medicine · doi:10.1002/sim.3177 · Cited by 464 · see citing works
Self-citationCorrecting for Optimistic Prediction in Small Data Sets
Smith, G. C. S., Seaman, S. R., Wood, A., Royston, P. J., & White, I. R. (2014) · American Journal of Epidemiology · doi:10.1093/aje/kwu140 · Cited by 354 · see citing works
Self-citationIntensive versus standard case management for severe psychotic illness: a randomised trial
Burns, T., Creed, F. H., Fahy, T., Thompson, S. G., & Tyrer, P. J. (1999) · The Lancet · doi:10.1016/s0140-6736(98)12191-8 · Cited by 321 · see citing works
Development and validation of a prediction model with missing predictor data: a practical approach
Vergouwe, Y., Royston, P. J., Moons, K. G. M., & Altman, D. G. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.03.017 · Cited by 288 · see citing works
Self-citationMissing-Data Adjustments in Large Surveys
Little, R. J. A. (1988) · Journal of Business and Economic Statistics · doi:10.2307/1391878 · Cited by 286 · see citing works
Collins, P. G. S., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c2442 · Cited by 250 · see citing works
Ambler, G., Omar, R. Z., & Royston, P. (2007) · Statistical Methods in Medical Research · doi:10.1177/0962280206074466 · Cited by 233 · see citing works
Developing a prognostic model in the presence of missing data
Clark, T. G., & Altman, D. G. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00539-5 · Cited by 214 · see citing works
Collins, P. G. S., & Altman, D. G. (2009) · BMJ · doi:10.1136/bmj.b2584 · Cited by 192 · see citing works
Collins, G. S., & Altman, D. (2012) · British Journal of Cancer · doi:10.1038/bjc.2012.266 · Cited by 60 · see citing works
The search for stable prognostic models in multiple imputed data sets
Vergouw, D., Heymans, M. W., Peat, G., Kuijpers, T., Croft, P. R., de Vet, H. C. W., van der Horst, H. E., & van der Windt, D. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-81 · Cited by 34 · see citing works
van de Laar, R., IntHout, J., Gorp, T. V., et al. (2013) · British Journal of Cancer · doi:10.1038/bjc.2013.717 · Cited by 27 · see citing works
Multiple imputation of missing blood pressure covariates in survival analysis
van Buuren, S., Boshuizen, H. C., & Knook, D. L. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990330)18:6
Flagged first, then by citations
Cited by
Sorted by citations
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
Sisk, R., Sperrin, M., Peek, N., van Smeden, M., & Martin, G. P. (2023) · Statistical Methods in Medical Research · doi:10.1177/09622802231165001 · Cited by 99 · see citing works
- Authors
- Wood, A. M., Royston, P., & White, I. R.
- Affiliations
- University of Cambridge (GB)
- MRC Clinical Trials Unit at UCL (GB)
- Corresponding author
- Wood, Angela M.
- Venue
- Biometrical Journal
- Publisher
- Wiley
- Published
- 2015-01-29
- Volume
- 57
- Issue
- 4
- Pages
- 614-632
- ISSN
- 0323-3847 (ISSN-L)
- 1521-4036
- PMC
- PMC4515100
- OpenAlex
- W1528995643
- Cited by
- 78
- References
- 32
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 10 · 2023: 11 · 2022: 8 · 2021: 14 · 2020: 12 · 2019: 3 · 2018: 7 · 2017: 2 · 2016: 2 · 2012: 1
- FWCI
- 3.26
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Biometrical Journal (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Medical Research Council (G0701619, Unit Programme number U105260558)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- multiple imputation, prediction model performance, missing covariates, model validation
- MeSH terms
- Analysis of Variance, Biometry, Clinical Trials as Topic, Humans, Models, Statistical, Logistic Models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
