On Missingness Features in Machine Learning Models for Critical Care: Observational Study
Open accessPublished version of a preprint
Abstract
BACKGROUND: Missing data in electronic health records is inevitable and considered to be nonrandom. Several studies have found that features indicating missing patterns (missingness) encode useful information about a patient's health and advocate for their inclusion in clinical prediction models. But their effectiveness has not been comprehensively evaluated. OBJECTIVE: The goal of the research is to study the effect of including informative missingness features in machine learning models for various clinically relevant outcomes and explore robustness of these features across patient subgroups and task settings. METHODS: A total of 48,336 electronic health records from the 2012 and 2019 PhysioNet Challenges were used, and mortality, length of stay, and sepsis outcomes were chosen. The latter dataset was multicenter, allowing external validation. Gated recurrent units were used to learn sequential patterns in the data and classify or predict labels of interest. Models were evaluated on various criteria and across population subgroups evaluating discriminative ability and calibration. RESULTS: Generally improved model performance in retrospective tasks was observed on including missingness features. Extent of improvement depended on the outcome of interest (area under the curve of the receiver operating characteristic [AUROC] improved from 1.2% to 7.7%) and even patient subgroup. However, missingness features did not display utility in a simulated prospective setting, being outperformed (0.9% difference in AUROC) by the model relying only on pathological features. This was despite leading to earlier detection of disease (true positives), since including these features led to a concomitant rise in false positive detections. CONCLUSIONS: This study comprehensively evaluated effectiveness of missingness features on machine learning models. A detailed understanding of how these features affect model performance may lead to their informed use in clinical settings especially for administrative tasks like length of stay prediction where they present the greatest benefit. While missingness features, representative of health care processes, vary greatly due to intra- and interhospital factors, they may still be used in prediction models for clinically relevant outcomes. However, their use in prospective models producing frequent predictions needs to be explored further.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Care-process shifts, especially changes in test-ordering and documentation practices, can alter both feature availability and the meaning of missingness (informative missingness), which can affect deterioration-prediction performance even when underlying patient mix is relatively stable.
“Care-process shifts – especially changes in test-ordering and documentation practices – can alter both feature availability and the meaning of missingness (“informative missingness”), which can affect deterioration-prediction performance even when underlying patient mix is relatively stable [11,12].”
From Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems (2026) · cites this paper as [12]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The third international consensus definitions for sepsis and septic shock (Sepsis-3).
Singer, M., Deutschman, C. S., Seymour, C. W., Shankar-Hari, M., Annane, D., Bauer, M., Bellomo, R., Bernard, G. R., Chiche, J.-D., Coopersmith, C. M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0287 · Cited by 28,250 · see citing works
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697 · see citing works
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925 · see citing works
Angus, D. C., Linde-Zwirble, W. T., Lidicker, J., Clermont, G., Carcillo, J., & Pinsky, M. R. (2001) · Critical Care Medicine · doi:10.1097/00003246-200107000-00002 · Cited by 8,503 · see citing works
Neural Information Processing Systems
(2018) · The MIT Press eBooks · doi:10.7551/mitpress/11474.003.0014 · Cited by 3,775 · see citing works
Multiple imputation by chained equations: what is it and how does it work?
Azur, M. J., Stuart, E. A., Frangakis, C., & Leaf, P. J. (2011) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.329 · Cited by 3,542 · see citing works
Recurrent Neural Networks for Multivariate Time Series with Missing Values
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2018) · Scientific Reports · doi:10.1038/s41598-018-24271-9 · Cited by 2,370 · see citing works
Shickel, B., Tighe, P. J., Bihorac, A., & Rashidi, P. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2767063 · Cited by 1,688 · see citing works
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · see citing works
Saeed, M., Villarroel, M., Reisner, A. T., Clifford, G., Lehman, L.-W., Moody, G., Heldt, T., Kyaw, T. H., Moody, B., & Mark, R. G. (2011) · Critical Care Medicine · doi:10.1097/ccm.0b013e31820a92c6 · Cited by 1,162 · see citing works
Xiao, C., Choi, E., & Sun, J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy068 · Cited by 792 · see citing works
A targeted real-time early warning score (TREWScore) for septic shock.
Henry, K. E., Hager, D. N., Pronovost, P. J., & Saria, S. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.aab3719 · Cited by 598 · see citing works
Machine Learning and Decision Support in Critical Care
Johnson, A. E. W., Ghassemi, M. M., Nemati, S., Niehaus, K. E., Clifton, D. A., & Clifford, G. D. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2501978 · Cited by 438 · see citing works
Agniel, D., Kohane, I. S., & Weber, G. M. (2018) · BMJ · doi:10.1136/bmj.k1479 · Cited by 391 · see citing works
Early Prediction of Sepsis From Clinical Data: The PhysioNet/Computing in Cardiology Challenge 2019
Reyna, M. A., Josef, C. S., Jeter, R., Shashikumar, S. P., Westover, M. B., Nemati, S., Clifford, G. D., & Sharma, A. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004145 · Cited by 383 · see citing works
Ozenne, B., Subtil, F., & Maucort-Boulch, D. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.02.010 · Cited by 379 · see citing works
Techniques to Improve Physicians' Use of Diagnostic Tests
Solomon, D. H., Hashimoto, H., Daltroy, L. H., & Liang, M. H. (1998) · JAMA · doi:10.1001/jama.280.23.2020 · Cited by 321 · see citing works
Lasko, T. A., Denny, J. C., & Levy, M. A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0066341 · Cited by 313 · see citing works
Deep Computational Phenotyping
Che, Z., Kale, D. C., Li, W., Bahadori, M. T., & Liu, Y. (2015) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2783365 · Cited by 247 · see citing works
Ghassemi, M., Pimentel, M. A. F., Naumann, T., Brennan, T. J., Clifton, D. A., Peter Szolovits, B., & Feng, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9209 · Cited by 201 · see citing works
Silva, I., Moody, G. B., Scott, D., Celi, L. A., & Mark, R. G. (2012) · PubMed · Cited by 167 · see citing works
Shickel, B., Loftus, T. J., Adhikari, L., Ozrazgat-Baslanti, T., Bihorac, A., & Rashidi, P. (2019) · Scientific Reports · doi:10.1038/s41598-019-38491-0 · Cited by 165 · see citing works
lnterhospital comparisons of patient outcome from intensive care
Dragsted, L., Jørgensen, J., JENSEN, N.-H., Bønsing, E., Jacobsen, E., Knaus, W. A., & Qvist, J. (1989) · Critical Care Medicine · doi:10.1097/00003246-198905000-00008 · Cited by 165 · see citing works
Early Prediction of Sepsis from Clinical Data: The PhysioNet/Computing in Cardiology Challenge 2019
Reyna, M. A., Shashikumar, S. P., Moody, B., Gu, P., Sharma, A., Nemati, S., & Clifford, G. D. (2019) · Computing in cardiology · doi:10.22489/cinc.2019.412 · Cited by 156 · see citing works
Exploiting missing clinical data in Bayesian network modeling for predicting medical problems
Lin, J.-H., & Haug, P. J. (2007) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2007.06.001 · Cited by 138 · see citing works
Rusanov, A., Weiskopf, N. G., Wang, S., & Weng, C. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-51 · Cited by 137 · see citing works
Sick patients have more data: the non-random completeness of electronic health records.
Weiskopf, N. G., Rusanov, A., & Weng, C. (2013) · Europe PMC (PubMed Central) · Cited by 99 · see citing works
Cardiology's 10 greatest discoveries of the 20th century.
Mehta, N. J., & Khan, I. A. (2002) · Europe PMC (PubMed Central) · Cited by 94 · see citing works
Optimal intensive care outcome prediction over time using machine learning
Meiring, C., Dixit, A., Harris, S. K., MacCallum, N. S., Brealey, D. A., Watkinson, P., Jones, A., Ashworth, S., Beale, R. J., Brett, S. J., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0206862 · Cited by 94 · see citing works
Identifying and mitigating biases in EHR laboratory tests
Pivovarov, R., Albers, D. J., Sepulveda, J. L., & Elhadad, N. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.03.016 · Cited by 91 · see citing works
General Practitioners?? Uncertainty, Risk Preference, and Use of Laboratory Tests
Zaat, J. O. M., & van Eijk, J. T. M. (1992) · Medical Care · doi:10.1097/00005650-199209000-00008 · Cited by 88 · see citing works
Patterns of Daily Costs Differ for Medical and Surgical Intensive Care Unit Patients
Gershengorn, H. B., Garland, A., & Gong, M. N. (2015) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201506-366bc · Cited by 79 · see citing works
A New Insight Into Missing Data in Intensive Care Unit Patient Profiles: Observational Study
Sharafoddini, A., Dubin, J. A., Maslove, D. M., & Lee, J. (2019) · JMIR Medical Informatics · doi:10.2196/11605 · Cited by 72 · see citing works
Freedman, D. B. (2015) · PubMed Central · Cited by 70 · see citing works
Leurquin, P., Casteren, V. V., & Maeseneer, J. D. (1995) · Europe PMC (PubMed Central) · Cited by 52 · see citing works
The General Practitioner and Laboratory Utilization: Why Does It Vary?
Kristiansen, I. S., & Hjortdahl, P. (1992) · Family Practice · doi:10.1093/fampra/9.1.22 · Cited by 52 · see citing works
Morrill, J., Kormilitzin, A., Nevado‐Holgado, A. J., Swaminathan, S., Howison, S., & Lyons, T. (2019) · Computing in cardiology · doi:10.22489/cinc.2019.014 · Cited by 44 · see citing works
Sepsis Prediction in Intensive Care Unit Using Ensemble of XGboost Models
Zabihi, M., Serkan Kiranyaz, B., & Gabbouj, M. (2019) · Computing in cardiology · doi:10.22489/cinc.2019.238 · Cited by 40 · see citing works
ICU acuity: real-time models versus daily models.
Hug, C. W., & Szolovits, P. (2009) · PubMed · Cited by 32 · see citing works
Rare Events in the ICU: An Emerging Challenge in Classification and Prediction
Leisman, D. E. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000002943 · Cited by 30 · see citing works
Chang, Y., Rubin, J. M., Boverman, G., Vij, S. G., Rahman, A., Natarajan, A., & Parvaneh, S. (2019) · Computing in cardiology · doi:10.22489/cinc.2019.110 · Cited by 16 · see citing works
Time series cluster kernels to exploit informative missingness and incomplete label information
Mikalsen, K. Ø., Soguero-Ruíz, C., Bianchi, F. M., Revhaug, A., & Jenssen, R. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107896 · Cited by 15 · see citing works
Utilizing Informative Missingness for Early Prediction of Sepsis
Singh, J., Oshiro, K., Krishnan, R., Sato, M., Ohkuma, T., & Kato, N. (2019) · Computing in cardiology · doi:10.22489/cinc.2019.280 · Cited by 10 · see citing works
Self-citationWith Severity Scores Updated on the Hour, Data Science Inches Closer to the Bedside*
Maslove, D. M. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000002945 · Cited by 7 · see citing works
Relationships among patient age, diagnosis, hospital type, and clinical laboratory utilization.
Taylor, J. B. (2005) · PubMed · Cited by 7 · see citing works
Time-Specific Metalearners for the Early Prediction of Sepsis
Vollmer, M., Luz, C. F., Sodmann, P., Sinha, B. N. M., & Kuhn, S. (2019) · Computing in cardiology · doi:10.22489/cinc.2019.029 · Cited by 4 · see citing works
Diagnosis of Atypical HUS Using Genetic Testing
Taylor, S., Nguyen, L. D., Piedra, B., & Walters, M. (2019) · American Society for Clinical Laboratory Science · doi:10.29074/ascls.2018000646 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- [9]Wei C Dong W Jian L Hao Z Lei L Yitan L BRITS: Bidirectional recurrent imputation for time series Adv Neural Inf Proc Syst 2018 6775 6785
- [12]De Brouwer E Simm J Arany A Moreau Y Gru-ode-bayes: continuous modeling of sporadically-observed time series Adv Neural Inf Proc Syst 2019 7379 7390
- [13]Lipton ZC Kale DC Wetzel R Modeling missing data in clinical time series with RNNs ArXiv. Preprint posted online June 13, 2016.
- [28]Futoma J Hariharan S Sendak M Brajer N Clement M Bedoya A O'Brien C Heller K An improved multi-output gaussian process rnn with real-time validation for early sepsis detection ArXiv. Preprint posted online on August 19, 2017.
- [37]Lipton Z Kale D Elkan C Wetzel R Learning to diagnose with lstm recurrent neural networks ArXiv. Preprint posted online on November 11, 2015.
- [40]Paszke A Gross S Massa F Lerer A Wallach H Larochelle H Beygelzimer A d'Alché-Buc F Pytorch: an imperative style, high-performance deep learning library Adv Neural Inf Proc Syst 2019 Red Hook Curran Associates, Inc 8024 8034
- [41]Kingma D Ba J Adam: a method for stochastic optimization ArXiv. Preprint posted online on December 22, 2014.
- [46]Ovadia Y Fertig E Ren J Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift ArXiv. Preprint posted online on June 6, 2019.
Cited by
Sorted by citations
Minimizing bias when using artificial intelligence in critical care medicine
Ranard, B. L., Park, S., Jia, Y., Zhang, Y., Alwan, F. S., Celi, L. A., & Lusczek, E. R. (2024) · Journal of Critical Care · doi:10.1016/j.jcrc.2024.154796 · Cited by 27 · see citing works
A Comprehensive Machine Learning Based Pipeline for an Accurate Early Prediction of Sepsis in ICU
Srimedha, B. C., Raj, R. N., & Mayya, V. (2022) · IEEE Access · doi:10.1109/access.2022.3210575 · Cited by 17 · see citing works
Impact of midwife continuity of carer on stillbirth rate and first feed in England
Roebuck, C., Sandall, J., West, R. M., Atherden, C., Parkyn, K., & Johnson, O. A. (2025) · Communications Medicine · doi:10.1038/s43856-025-01025-z · Cited by 10 · see citing works
Velez, T., Ibrahim, Z., Duru, K., Velez, D., Triantafyllou, M., McKinley, K. W., Saif, P., Kratimenos, P., Clark, A., & Koutroulis, I. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105905 · Cited by 7 · see citing works
Position Paper: Integrating Explainability and Uncertainty Estimation in Medical AI
Fan, X. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228229 · Cited by 5 · see citing works
Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems
Kopanitsa, G. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001534 · Cited by 3 · see citing works
“Care-process shifts – especially changes in test-ordering and documentation practices – can alter both feature availability and the meaning of missingness (“informative missingness”), which can affect deterioration-prediction performance even when underlying patient mix is relatively stable [11,12].”
Getzen, E. J., Tan, A. L. M., Brat, G. A., Omenn, G. S., Strasser, Z. H., Long, Q., Holmes, J. H., & Mowery, D. L. (2023) · medRxiv · doi:10.1101/2023.12.18.23300181 · Cited by 3 · see citing works
Do, D.-K., Rockenschaub, P., Boie, S. D., Kumpf, O., Volk, H., Balzer, F., von Dincklage, F., & Lichtner, G. (2026) · Journal of Medical Internet Research · doi:10.2196/72083 · Cited by 6 · see citing works
Ratchatorn, A., Ketdao, N., Sonsilphong, S., Triamwichanon, D., & Panitchote, A. (2026) · Critical Care · doi:10.1186/s13054-026-05942-w
Chang, S., Samoli, E., Diapouli, E., Dimakopoulou, K., Alahmad, B., & Koutrakis, P. (2026) · PLoS ONE · doi:10.1371/journal.pone.0352975
Adversarial Validation Reveals Diagnostic Workflow Leakage in PCOS Machine Learning ModelsPreprint
Katarynczuk, K., Stachowiak, A., Piórkowska, N., Ostromęcki, A., Franik, G., & Bizoń, A. (2026) · medRxiv · doi:10.64898/2026.07.22.26358682
Liu, Y., Liu, Y., Zhao, Y., Luo, Y., & Hao, X. (2026) · Frontiers in Endocrinology · doi:10.3389/fendo.2026.1893963
Zhang, G., Yang, C., Jiang, S., Li, X., & Duan, H. (2026) · Research Square · doi:10.21203/rs.3.rs-10708009/v1
Towards Interpretable Risk: Multidimensional Context for ICU Mortality PredictionsPreprint
Gupta, S., Das, A., Anto, M. S., Alam, Z., Datta, A., Gupta, T., Pias, T. S., & Islam, H. (2026) · medRxiv · doi:10.64898/2026.09.03.26362203
Shen, Y., Liu, Y., Dong, C., Pal, S., Yang, Y., & Shen, J. (2026) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-026-00277-5
AI/ML-enabled sepsis prediction systems: laboratory considerations from development to deployment
Durant, T. J. S., Aldewereld, Z., Peaper, D. R., & Wheeler, S. (2026) · Journal of Clinical Microbiology · doi:10.1128/jcm.01461-25
Clinical Readiness of Vital Sign–Based Sepsis Prediction
Li, L., Lei, S., & Ngan, C. (2026) · JMIR Medical Informatics · doi:10.2196/111475
- Authors
- Singh, J., Sato, M., & Ohkuma, T.
- Affiliations
- Fuji Xerox (Japan) (JP)
- Corresponding author
- Singh, Janmajay
- Venue
- JMIR Medical Informatics
- Publisher
- JMIR Publications Inc.
- Published
- 2021-12-08
- Volume
- 9
- Issue
- 12
- Pages
- e25022
- ISSN
- 2291-9694 (ISSN-L)
- PMC
- PMC8701717
- OpenAlex
- W4200099769
- Cited by
- 17
- References
- 48
- Citations by year
- 2026: 11 · 2025: 3 · 2024: 1 · 2023: 1 · 2022: 1
- FWCI
- 0.27
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Informatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
- Keywords
- missing data, electronic health records, intensive care, clinical prediction model, sepsis prediction, mortality prediction, length of stay prediction, gated recurrent unit, model calibration, patient subgroups, prospective validation, PhysioNet Challenge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
