A New Insight Into Missing Data in Intensive Care Unit Patient Profiles: Observational Study
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The data missing from patient profiles in intensive care units (ICUs) are substantial and unavoidable. However, this incompleteness is not always random or because of imperfections in the data collection process. OBJECTIVE: This study aimed to investigate the potential hidden information in data missing from electronic health records (EHRs) in an ICU and examine whether the presence or missingness of a variable itself can convey information about the patient health status. METHODS: Daily retrieval of laboratory test (LT) measurements from the Medical Information Mart for Intensive Care III database was set as our reference for defining complete patient profiles. Missingness indicators were introduced as a way of representing presence or absence of the LTs in a patient profile. Thereafter, various feature selection methods (filter and embedded feature selection methods) were used to examine the predictive power of missingness indicators. Finally, a set of well-known prediction models (logistic regression [LR], decision tree, and random forest) were used to evaluate whether the absence status itself of a variable recording can provide predictive power. We also examined the utility of missingness indicators in improving predictive performance when used with observed laboratory measurements as model input. The outcome of interest was in-hospital mortality and mortality at 30 days after ICU discharge. RESULTS: Regardless of mortality type or ICU day, more than 40% of the predictors selected by feature selection methods were missingness indicators. Notably, employing missingness indicators as the only predictors achieved reasonable mortality prediction on all days and for all mortality types (for instance, in 30-day mortality prediction with LR, we achieved area under the curve of the receiver operating characteristic [AUROC] of 0.6836±0.012). Including indicators with observed measurements in the prediction models also improved the AUROC; the maximum improvement was 0.0426. Indicators also improved the AUROC for Simplified Acute Physiology Score II model-a well-known ICU severity of illness score-confirming the additive information of the indicators (AUROC of 0.8045±0.0109 for 30-day mortality prediction for LR). CONCLUSIONS: Our study demonstrated that the presence or absence of LT measurements is informative and can be considered a potential predictor of in-hospital and 30-day mortality. The comparative analysis of prediction models also showed statistically significant prediction improvement when indicators were included. Moreover, missing data might reflect the opinions of examining clinicians. Therefore, the absence of measurements can be informative in ICUs and has predictive power beyond the measured data themselves. This initial case study shows promise for more in-depth analysis of missing data and its informativeness in ICUs. Future studies are needed to generalize these results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299
Kuhn, M., & Johnson, K. (2013) · doi:10.1007/978-1-4614-6849-3 · Cited by 6,788
Fitzmaurice, G. M., Laird, N. M., & Ware, J. H. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9781119513469 · Cited by 4,639
Theoretical and Empirical Analysis of ReliefF and RReliefF
Robnik-Šikonja, M., & Kononenko, I. (2003) · Machine Learning · doi:10.1023/a:1025667309714 · Cited by 3,025 · cited 1×
“We selected 3 different metrices: LR beta value, relief algorithm [58], and information gain (InfGain) [59].”
MethodExternal validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544 · cited 1×
“Although researchers argue that sample selection based on the predefined eligibility criteria in randomized clinical trials can limit the external generalizability of these studies [26], CCA in studies using EHR data can also potentially threaten the external validity of a study [19] and cause bias as the literature s…”
BackgroundFlexible Imputation of Missing Data, Second Edition
van Buuren, S. (2018) · doi:10.1201/9780429492259 · Cited by 2,273
The prevention and treatment of missing data in clinical trials
Little, R. J., D'Agostino, R., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., Frangakis, C., Hogan, J. W., Molenberghs, G., Murphy, S. A., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmsr1203730 · Cited by 1,884 · cited 2×
“The data missing from clinical contexts are substantial [3,4] and unavoidable [5]; many studies have focused on resolving this issue [6-8].”
BackgroundMethodWeiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · cited 2×
“The increased adoption of electronic health record (EHR) systems has boosted interest in the secondary use of EHR data [1].”
BackgroundGroenwold, R. H. H., White, I. R., Donders, A. R. T., Carpenter, J. R., Altman, D. G., & Moons, K. G. M. (2012) · Canadian Medical Association Journal · doi:10.1503/cmaj.110977 · Cited by 538 · cited 1×
“However, in the current use of indicators, missing values are set to a fixed value (0 or the normal value for the variable) and the indicators are used as dummy variables in analytical models to indicate that a value was missing [46,47].”
MethodInterpreting Incremental Value of Markers Added to Risk Prediction Models
Pencina, M. J., D'Agostino, R. B., Pencina, K. M., Janssens, A. C. J. W., & Greenland, P. (2012) · American Journal of Epidemiology · doi:10.1093/aje/kws207 · Cited by 506 · cited 2×
“01) in AUROC meaningful and of clinical interest (because of insensitivity of AUROC) [62,64], including the indicators in our study could improve the average AUROC by 0.”
BackgroundMethod
Cited by
The myth of generalisability in clinical research and machine learning in health care
Futoma, J., Simons, M., Panch, T., Doshi-Velez, F., & Celi, L. A. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30186-2 · Cited by 467
Leisman, D. E., Harhay, M. O., Lederer, D. J., Abramson, M., Adjei, A. A., Bakker, J., Ballas, Z. K., Barreiro, E., Bell, S. C., Bellomo, R., et al. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004246 · Cited by 271
Self-citationOn Missingness Features in Machine Learning Models for Critical Care: Observational Study
Singh, J., Sato, M., & Ohkuma, T. (2021) · JMIR Medical Informatics · doi:10.2196/25022 · Cited by 17
“Perhaps the study most similar to this work was by Sharafoddini et al [50], which examined whether missing indicator features are informative.”
Metadata
- Authors
- Sharafoddini, A., Dubin, J. A., Maslove, D. M., & Lee, J.
- Venue
- JMIR Medical Informatics
- Publisher
- JMIR Publications Inc.
- Published
- 2019-01-08
- Volume
- 7
- Issue
- 1
- Pages
- e11605
- ISSN
- 2291-9694 (ISSN-L)
- OpenAlex
- W2910420059
- Cited by
- 70
- References
- 17
- FWCI
- 4.14
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.