Preserving Informative Presence: How Missing Data and Imputation Strategies Affect the Performance of an AI-Based Early Warning Score
Open accessAbstract
Background/Objectives: Data availability can affect the performance of AI-based early warning scores (EWSs). This study evaluated how the extent of missing data and imputation strategies influence the predictive performance of the VitalCare–Major Adverse Event Score (VC-MAES), an AI-based EWS that uses last observation carried forward and normal-value imputation for missing values, to forecast clinical deterioration events, including unplanned ICU transfers, cardiac arrests, or death, up to 6 h in advance. Methods: We analyzed real-world data from 6039 patient encounters at Keimyung University Dongsan Hospital, Republic of Korea. Performance was evaluated under three scenarios: (1) using only vital signs and age, treating all other variables as missing; (2) reintroducing a full set of real-world clinical variables; and (3) imputing missing values drawn from a distribution within one standard deviation of the observed mean or using Multiple Imputation by Chained Equations (MICE). Results: VC-MAES achieved the area under the receiver operating characteristic curve (AUROC) of 0.896 using only vital signs and age, outperforming traditional EWSs, including the National Early Warning Score (0.797) and the Modified Early Warning Score (0.722). Reintroducing full clinical variables improved the AUROC to 0.918, whereas mean-based imputation or MICE decreased the performance to 0.885 and 0.827, respectively. Conclusions: VC-MAES demonstrates robust predictive performance with limited inputs, outperforming traditional EWSs. Incorporating actual clinical data significantly improved accuracy. In contrast, mean-based or MICE imputation yielded poorer results than the default normal-value imputation, potentially due to disregarding the “informative presence” embedded in missing data patterns. These findings underscore the importance of understanding missingness patterns and employing imputation strategies that consider the decision-making context behind data availability to enhance model reliability.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Care-process shifts, especially changes in test-ordering and documentation practices, can alter both feature availability and the meaning of missingness (informative missingness), which can affect deterioration-prediction performance even when underlying patient mix is relatively stable.
“Care-process shifts – especially changes in test-ordering and documentation practices – can alter both feature availability and the meaning of missingness (“informative missingness”), which can affect deterioration-prediction performance even when underlying patient mix is relatively stable [11,12].”
From Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems (2026) · cites this paper as [11]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The prevention and treatment of missing data in clinical trials
Little, R. J., D'Agostino, R., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., Frangakis, C., Hogan, J. W., Molenberghs, G., Murphy, S. A., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmsr1203730 · Cited by 1,884 · see citing works
Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936 · see citing works
Real-world data: A brief review of the methods, applications, challenges and opportunities
Liu, F., & Panagiotakos, D. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01768-6 · Cited by 496 · see citing works
Last observation carry‐forward and last observation analysis
Shao, J., & Zhong, B. (2003) · Statistics in Medicine · doi:10.1002/sim.1519 · Cited by 279 · see citing works
Machine Learning–Based Early Warning Systems for Clinical Deterioration: Systematic Scoping Review
Muralitharan, S., Nelson, W., Di, S., McGillion, M. H., Devereaux, P. J., Barr, N. G., & Petch, J. (2020) · Journal of Medical Internet Research · doi:10.2196/25187 · Cited by 179 · see citing works
Ayilara, O. F., Zhang, L., Sajobi, T. T., Sawatzky, R. G., Bohm, É., & Lix, L. M. (2019) · Health and Quality of Life Outcomes · doi:10.1186/s12955-019-1181-2 · Cited by 176 · see citing works
The impact of imputation quality on machine learning classifiers for datasets with missing values
Shadbahr, T., Roberts, M. T., Stanczuk, J., Gilbey, J., Teare, P., Dittmer, S., Thorpe, M., Viñas, R., Sala, E., Lió, P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00356-z · Cited by 154 · see citing works
Mining for equitable health: Assessing the impact of missing data in electronic health records
Getzen, E. J., Ungar, L., Mowery, D. L., Jiang, X., & Long, Q. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104269 · Cited by 136 · see citing works
Prevalence of Missing Data in the National Cancer Database and Association with Overall Survival
Yang, D. X., Khera, R., Miccio, J. A., Jairam, V., Chang, E., Yu, J. B., Park, H. S., Krumholz, H. M., & Aneja, S. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.1793 · Cited by 135 · see citing works
Evaluating the state of the art in missing data imputation for clinical data
Luo, Y. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab489 · Cited by 130 · see citing works
Evaluating the impact of multivariate imputation by MICE in feature selection
Mera-Gaona, M., Neumann, U., Vargas-Canas, R., & López, D. M. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254720 · Cited by 96 · see citing works
Lee, K. J., Carlin, J. B., Simpson, J. A., & Moreno‐Betancur, M. (2023) · International Journal of Epidemiology · doi:10.1093/ije/dyad008 · Cited by 83 · see citing works
Self-citationA Data-Centric Approach to improve performance of deep learning models
Bhatt, N., Bhatt, N., Prajapati, P., Sorathiya, V. P., AlShathri, S. I., & El‐Shafai, W. (2024) · Scientific Reports · doi:10.1038/s41598-024-73643-x · Cited by 73 · see citing works
Tsiampalis, T., & Panagiotakos, D. B. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01038-3 · Cited by 71 · see citing works
Zhu, X. (2014) · Open Journal of Statistics · doi:10.4236/ojs.2014.411088 · Cited by 59 · see citing works
Babar, M. A., Qureshi, B., & Koubâa, A. (2024) · PLoS ONE · doi:10.1371/journal.pone.0302539 · Cited by 53 · see citing works
Wang, H., Tang, J., Wu, M., Wang, X., & Zhang, T. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01752-6 · Cited by 50 · see citing works
Missing data in primary care research: importance, implications and approaches
Marino, M., Lucas, J., Latour, E. J., & Heintzman, J. D. (2020) · Family Practice · doi:10.1093/fampra/cmaa134 · Cited by 44 · see citing works
Logothetis, S. B., Green, D., Holland, M., & Moubayed, N. A. (2023) · Scientific Reports · doi:10.1038/s41598-023-40661-0 · Cited by 37 · see citing works
Lewis, M., Elad, G., Beladev, M., Maor, G., Radinsky, K., Hermann, D., Litani, Y., Geller, T. P., Pines, J. M., Shapiro, N. L., et al. (2021) · Scientific Reports · doi:10.1038/s41598-020-80856-3 · Cited by 32 · see citing works
Sung, M., Hahn, S., Han, C. H., Lee, J. M., Lee, J., Yoo, J., Heo, J., Kim, Y. S., & Chung, K. S. (2021) · JMIR Medical Informatics · doi:10.2196/26426 · Cited by 27 · see citing works
Self-citationCharacterization of Missing Data in Clinical Registry Studies
Mendelsohn, A. B., Dreyer, N. A., Mattox, P. W., Su, Z., Swenson, A., Li, R., Turner, J. R., & Velentgas, P. (2014) · Therapeutic Innovation & Regulatory Science · doi:10.1177/2168479014532259 · Cited by 24 · see citing works
Correction: Evaluating the impact of multivariate imputation by MICE in feature selection
Mera-Gaona, M., Neumann, U., Vargas-Cañas, R., & López, D. M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0261739 · Cited by 18 · see citing works
Karakaya, J., Karabulut, E., & Yucel, R. M. (2014) · Journal of Statistical Computation and Simulation · doi:10.1080/00949655.2014.983111 · Cited by 10 · see citing works
Reardon, P. M., Parimbelli, E., Szymon Wilk, A., Michalowski, W. J., Murphy, K., Shen, J. M., Herritt, B., Gershkovich, B., Tanuseputro, P., & Kyeremanteng, K. (2019) · Critical Care Explorations · doi:10.1097/cce.0000000000000023 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Kim, J.-H., Cho, E. Y., Choi, Y., Won, J.-Y., Cheon, S. H., Kim, Y. A., Lee, K.-B., Kim, K. J., Kim, H. G., & Sim, T. (2025) · JMIR Medical Informatics · doi:10.2196/72232 · Cited by 4 · see citing works
Self-citationValidation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems
Kopanitsa, G. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001534 · Cited by 3 · see citing works
Sim, T. Y., Cho, E. Y., Kim, J., Kim, H. G., & Kim, S. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14134444
Self-citationJang, J., Choi, Y. J., Sim, T. Y., Lee, K.-B., Kim, J., Cho, E. Y., Choi, Y., Hong, S., Jung, B. M., KIM, S., et al. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15145753
Self-citationArtificial intelligence-assisted photodynamic diagnosis and photodynamic therapy against cancer
Huang, J., Law, S., Leung, A. W. N., & Xu, C. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1932816
Mun, Y., Ahn, J. H., Lee, C. Y., Ahn, B., Huh, J. W., & Chung, K. S. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag124
Shen, Y., Liu, Y., Dong, C., Pal, S., Yang, Y., & Shen, J. (2026) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-026-00277-5
- Authors
- Sim, T., Hahn, S., Kim, K.-J., Cho, E.-Y., Jeong, Y., Kim, J.-H., Ha, E.-Y., Kim, I.-C., Park, S.-H., Cho, C.-H., Yu, G.-I., Cho, H., & Lee, K.-B.
- Affiliations
- AITRICS Corp., Seoul 06221, Republic of Korea
- Yonsei University (KR)
- Keimyung University Dongsan Medical Center (KR)
- Keimyung University Daegu Dongsan Hospital (KR)
- Keimyung University Dongsan Hospital (KR)
- Department of Internal Medicine, Keimyung University Dongsan Hospital, Keimyung University School of Medicine, Daegu 42601, Republic of Korea
- Hallym University Chuncheon Sacred Heart Hospital (KR)
- Corresponding author
- Lee, Ki-Byung
- ORCID
- Venue
- Journal of Clinical Medicine
- Publisher
- MDPI AG
- Published
- 2025-03-24
- Volume
- 14
- Issue
- 7
- Pages
- 2213
- ISSN
- 2077-0383 (ISSN-L)
- PMC
- PMC11989256
- OpenAlex
- W4408789659
- Cited by
- 7
- References
- 24
- Citations by year
- 2026: 5 · 2025: 2
- FWCI
- 2.81
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Journal of Clinical Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Ministry of Health and Welfare
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- early warning scores, missing data, missing data imputation, multiple imputation by chained equations, last observation carried forward, clinical deterioration prediction, National Early Warning Score, Modified Early Warning Score, vital signs, intensive care unit admission, cardiac arrest, mortality prediction, missingness patterns
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
