Bias arising from missing data in predictive models.
Abstract
Objective The purpose of this study is to determine the effect of three common approaches to handling missing data on the results of a predictive model.Study design and setting Monte Carlo simulation study using simulated data was used. A baseline logistic regression using complete data was performed to predict hospital admission, based on the white blood cell count (WBC) (dichotomized as normal or high), presence of fever, or procedures performed (PROC). A series of simulations was then performed in which WBC data were deleted for varying proportions (15-85%) of patients under various patterns of missingness. Three analytic approaches were used: analysis restricted to cases with complete data, missing data assumed to be normal (MAN), and use of imputed values.Results In the baseline analysis, all three predictors were all significantly associated with admission. Using either the MAN approach or imputation, the odds ratio (OR) for WBC was substantially over- or underestimated depending on the missingness pattern, and there was considerable bias toward the null in the OR estimates for fever. In the CC analyses, OR for WBC was consistently biased toward the null, OR for PROC was biased away from the null, and the OR for fever was biased toward or away from the null. Estimates for overall model discrimination were substantially biased using all analytic approaches.Conclusions All three methods of handling large amounts of missing data can lead to biased estimates of the OR and of model performance in predictive models. Predictor variables that are measured inconsistently can affect the validity of such models.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Commonly missing values are related to observed participant or disease characteristics.
“Commonly missing values are related to observed participant or disease characteristics.”
From Reporting and Methods in Clinical Prediction Research: A Systematic Review (2012) · cites this paper as [58]
Missing values in clinical study data rarely occur completely at random.
“Missing values in clinical study data rarely occur completely at random.”
From Reporting and Methods in Clinical Prediction Research: A Systematic Review (2012) · cites this paper as [58]
Exclusion of participants with missing values often leads to loss of statistical power and biased results.
“Exclusion of participants with missing values will therefore not only lead to loss of statistical power, but often also to biased results 10,12,13,23,24,57,58.”
From Reporting and Methods in Clinical Prediction Research: A Systematic Review (2012) · cites this paper as [58]
Cites
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,384 · see citing works
The APACHE III Prognostic System
Knaus, W. A., Wagner, D. P., Draper, E. A., Zimmerman, J. E., Bergner, M., BASTOS, P. G., Sirio, C., Murphy, D. J., Lotring, T., Damiano, A., et al. (1991) · CHEST Journal · doi:10.1378/chest.100.6.1619 · Cited by 3,524 · see citing works
The APACHE III Prognostic System
Teres, D., & Lemeshow, S. A. (1992) · CHEST Journal · doi:10.1378/chest.102.6.1919b · Cited by 2,880 · see citing works
Statistical Analysis with Missing Data.
Chatfield, C., Little, R. J. A., & Rubin, D. B. (1988) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2982783 · Cited by 2,203 · see citing works
Pollack, M. M., Patel, K. M., & Ruttimann, U. E. (1996) · Critical Care Medicine · doi:10.1097/00003246-199605000-00004 · Cited by 1,761 · see citing works
Regression with Missing X 's: A Review
Little, R. J. A. (1992) · Journal of the American Statistical Association · doi:10.1080/01621459.1992.10476282 · Cited by 968 · see citing works
Risk Adjustment for Measuring Healthcare Outcomes
Iezzoni, L. I. (1994) · Cited by 950 · see citing works
Regression With Missing X's: A Review
Little, R. J. A. (1992) · Journal of the American Statistical Association · doi:10.2307/2290664 · Cited by 647 · see citing works
Faris, P., Ghali, W. A., Brant, R., Norris, C. M., Galbraith, P. D., & Knudtson, M. L. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00433-4 · Cited by 130 · see citing works
Temperature and Total White Blood Cell Count as Indicators of Bacteremia
Jaffe, D. M., & FLEISHER, G. R. (1991) · PEDIATRICS · doi:10.1542/peds.87.5.670 · Cited by 106 · see citing works
Dealing with missing data in observational health care outcome analyses
Norris, C. M., Ghali, W. A., Knudtson, M. L., Naylor, C. D., & Saunders, L. D. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00181-x · Cited by 92 · see citing works
Chamberlain, J. M., Patel, K. M., & Pollack, M. M. (2005) · PEDIATRICS · doi:10.1542/peds.2004-0586 · Cited by 74 · see citing works
Chamberlain, J. M., Patel, K. M., Ruttimann, U. E., et al. (1998) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(98)70132-5 · Cited by 54 · see citing works
Frequency of variable measurement in 16 pediatric intensive care units
Pollack, M. M., Patel, K. M., Ruttimann, U. E., et al. (1996) · Critical Care Medicine · doi:10.1097/00003246-199601000-00013 · Cited by 40 · see citing works
Gorelick, M. H., Lee, C., Cronan, K. M., et al. (2001) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2001.tb01281.x · Cited by 35 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Moons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018 · see citing works
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · see citing works
Diagnostic and prognostic prediction models
Hendriksen, J. M. T., Geersing, G. J., Moons, K. G. M., & de Groot, J. A. H. (2013) · Journal of Thrombosis and Haemostasis · doi:10.1111/jth.12262 · Cited by 229 · see citing works
Lu, Y., Lavoie‐Gagne, O., Forlenza, E. M., Pareek, A., Kunze, K. N., Forsythe, B., Levy, B. A., & Krych, A. J. (2022) · Arthroscopy · doi:10.1016/j.arthro.2021.12.012 · Cited by 28 · see citing works
Prediction models in gynaecology: Transparent reporting needed for clinical application
Gelderblom, M. E., Stevens, K. Y. R., Houterman, S., Weyers, S., & Schoot, B. C. (2021) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/j.ejogrb.2021.08.013 · Cited by 11 · see citing works
- Authors
- Gorelick, M. H.
- Affiliations
- Medical College of Wisconsin (US)
- Corresponding author
- Gorelick, Marc H.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2006-10
- Volume
- 59
- Issue
- 10
- Pages
- 1115-1123
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2011744517
- Cited by
- 107
- References
- 15
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 23 · 2023: 5 · 2022: 4 · 2021: 11 · 2020: 4 · 2019: 4 · 2018: 3 · 2017: 4 · 2016: 3 · 2015: 4 · 2014: 6 · 2013: 6 · 2012: 3
- FWCI
- 3.14
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Agency for Healthcare Research and Quality (R03-HS11395)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- missing data, bias, prediction model, logistic regression, multiple imputation, complete case analysis, missing at random, model discrimination, hospital admission prediction, white blood cell count, fever, measurement discrepancy
- MeSH terms
- Adolescent, Child, Child, Preschool, Emergencies, Emergency Service, Hospital, Epidemiologic Methods, Fever, Hospitalization, Humans, Infant, Infant, Newborn, Leukocyte Count, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review