Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 45 of the 48 references it cites were found in a registry or an index.

Low responsiveness of machine learning models to critical or deteriorating health conditions

Open access

Published version of a preprint

Abstract

BACKGROUND: Machine learning (ML) based mortality prediction models can be immensely useful in intensive care units. Such a model should generate warnings to alert physicians when a patient's condition rapidly deteriorates, or their vitals are in highly abnormal ranges. Before clinical deployment, it is important to comprehensively assess a model's ability to recognize critical patient conditions. METHODS: We develop multiple medical ML testing approaches, including a gradient ascent method and neural activation map. We systematically assess these machine learning models' ability to respond to serious medical conditions using additional test cases, some of which are time series. Guided by medical doctors, our evaluation involves multiple machine learning models, resampling techniques, and four datasets for two clinical prediction tasks. RESULTS: We identify serious deficiencies in the models' responsiveness, with the models being unable to recognize severely impaired medical conditions or rapidly deteriorating health. For in-hospital mortality prediction, the models tested using our synthesized cases fail to recognize 66% of the injuries. In some instances, the models fail to generate adequate mortality risk scores for all test cases. Our study identifies similar kinds of deficiencies in the responsiveness of 5-year breast and lung cancer prediction models. CONCLUSIONS: Using generated test cases, we find that statistical machine-learning models trained solely from patient data are grossly insufficient and have many dangerous blind spots. Most of the ML models tested fail to respond adequately to critically ill patients. How to incorporate medical knowledge into clinical machine learning models is an important future research direction.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s43856-025-00775-0

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Number of generated test cases for evaluating models trained on in-hospital mortality risk prediction and 5-year cancer survivability prediction models. The left side illustrates the generated test case of each category for testing in-hospital mortality risk prediction models trained on MIMIC III or eICU dataset. The right side represents the generated test cases to test 5-year breast cancer survivability (BCS) prediction models. The SEER lung cancer survivability (LCS) models are tested similarly using the single-attribute test cases.
Fig. 1 Number of generated test cases for evaluating models trained on in-hospital mortality risk prediction and 5-year cancer survivability prediction models. The left side illustrates the generated test case of each category for testing in-hospital mortality risk prediction models trained on MIMIC III or eICU dataset. The right side represents the generated test cases to test 5-year breast cancer survivability (BCS) prediction models. The SEER lung cancer survivability (LCS) models are tested similarly using the single-attribute test cases.CC BY-NC-ND · PubMed Central
Fig. 2 Mortality risk (MR) prediction for Glasgow Coma Scale for different combinations using three machine learning models. MR predicted by (a) channel-wise LSTM model for three injury cases (E4M1V3, E4M1V5, and E4M3V5) represented by 3 bars and the X-axis, b LSTM model for three injury cases (E1M1V5, E1M3V5, and E2M2V5) represented by 3 bars and the X-axis, and c Logistic regression for injury cases for all combinations of GCS scores (GCS total 3–15) represented by the bars and the Y-axis. The error bar represents standard deviations calculated from five independent experimental trials (models). MR prediction of injury cases defined by different combinations of GCS eye and motor response scores by d LSTM and e logistic regression model. MR predicted by f LSTM and g logistic regression using injury cases defined by different combinations of GCS eye and verbal response scores. MR prediction of injury cases defined by different combinations of GCS motor and motor response scores by h LSTM and i logistic regression.
Fig. 2 Mortality risk (MR) prediction for Glasgow Coma Scale for different combinations using three machine learning models. MR predicted by (a) channel-wise LSTM model for three injury cases (E4M1V3, E4M1V5, and E4M3V5) represented by 3 bars and the X-axis, b LSTM model for three injury cases (E1M1V5, E1M3V5, and E2M2V5) represented by 3 bars and the X-axis, and c Logistic regression for injury cases for all combinations of GCS scores (GCS total 3–15) represented by the bars and the Y-axis. The error bar represents standard deviations calculated from five independent experimental trials (models). MR prediction of injury cases defined by different combinations of GCS eye and motor response scores by d LSTM and e logistic regression model. MR predicted by f LSTM and g logistic regression using injury cases defined by different combinations of GCS eye and verbal response scores. MR prediction of injury cases defined by different combinations of GCS motor and motor response scores by h LSTM and i logistic regression.CC BY-NC-ND · PubMed Central
Fig. 3 Mortality risk prediction for single vital-sign tests using three machine learning models (LSTM, Channel-wise LSTM, and Logistic Regression) and visualizing the neural activation map of the LSTM layer consisting of 16 neurons. LSTM, Channel-wise LSTM, and Logistic Regression (LR) predict the mortality risk (MR) of a respiratory rate, b body temperature, c glucose, d diastolic blood pressure, e systolic blood pressure, and f oxygen saturation test sets (synthesized). The mortality risk (MR) is represented by X-axis and MR above and below a red horizontal line (threshold = 0.22) indicates a high or low mortality risk zone, respectively. The shading represents the standard deviation calculated from five independent experimental trials (models). The entire range of each vital sign (except oxygen saturation) value is divided into three segments, low, normal, and high, by the blue vertical lines. The low and high values within these ranges indicate critical health conditions. Figures in the last row (g–j) represent the neural activation map. These are the neural activation values, calculated after applying the sigmoid function, when the model is fed with test cases varying a single vital, such as g glucose, h diastolic blood pressure, i temperature, and j respiratory rate.
Fig. 3 Mortality risk prediction for single vital-sign tests using three machine learning models (LSTM, Channel-wise LSTM, and Logistic Regression) and visualizing the neural activation map of the LSTM layer consisting of 16 neurons. LSTM, Channel-wise LSTM, and Logistic Regression (LR) predict the mortality risk (MR) of a respiratory rate, b body temperature, c glucose, d diastolic blood pressure, e systolic blood pressure, and f oxygen saturation test sets (synthesized). The mortality risk (MR) is represented by X-axis and MR above and below a red horizontal line (threshold = 0.22) indicates a high or low mortality risk zone, respectively. The shading represents the standard deviation calculated from five independent experimental trials (models). The entire range of each vital sign (except oxygen saturation) value is divided into three segments, low, normal, and high, by the blue vertical lines. The low and high values within these ranges indicate critical health conditions. Figures in the last row (g–j) represent the neural activation map. These are the neural activation values, calculated after applying the sigmoid function, when the model is fed with test cases varying a single vital, such as g glucose, h diastolic blood pressure, i temperature, and j respiratory rate.CC BY-NC-ND · PubMed Central
Fig. 4 Multi-attribute test results, including the mortality risk prediction under double-attribute variation tests and the mortality risk difference (ΔMR) between the seed and 6-attribute variation test cases. Risk prediction under varying respiratory rate and heart rate by a logistic regression, b LSTM model, and c CW-LSTM model. Risk prediction under varying systolic and diastolic blood pressure by d logistic regression, e LSTM model, and f CW-LSTM model. Risk prediction under varying glucose and diastolic blood pressure by g logistic regression, h LSTM model, and i CW-LSTM model. j–l represent ΔMR for high critical range cases and m–o represent ΔMR for low critical range cases. The test set is generated by simultaneously varying systolic blood pressure, diastolic blood pressure, blood glucose level, respiratory rate, heart rate, and body temperature and values are randomly selected from the critical zone. The graph shows the mortality risk difference (ΔMR) calculated by subtracting the predicted mortality risk of the seed (Class 0) from the predicted mortality risk of its corresponding critical case. The X-axis represents the case numbers and the Y-axis represents ΔMR. It is expected to get a positive MR difference and the negative ΔMR cases represent failed test cases.
Fig. 4 Multi-attribute test results, including the mortality risk prediction under double-attribute variation tests and the mortality risk difference (ΔMR) between the seed and 6-attribute variation test cases. Risk prediction under varying respiratory rate and heart rate by a logistic regression, b LSTM model, and c CW-LSTM model. Risk prediction under varying systolic and diastolic blood pressure by d logistic regression, e LSTM model, and f CW-LSTM model. Risk prediction under varying glucose and diastolic blood pressure by g logistic regression, h LSTM model, and i CW-LSTM model. j–l represent ΔMR for high critical range cases and m–o represent ΔMR for low critical range cases. The test set is generated by simultaneously varying systolic blood pressure, diastolic blood pressure, blood glucose level, respiratory rate, heart rate, and body temperature and values are randomly selected from the critical zone. The graph shows the mortality risk difference (ΔMR) calculated by subtracting the predicted mortality risk of the seed (Class 0) from the predicted mortality risk of its corresponding critical case. The X-axis represents the case numbers and the Y-axis represents ΔMR. It is expected to get a positive MR difference and the negative ΔMR cases represent failed test cases.CC BY-NC-ND · PubMed Central
Fig. 5 Gradient-generated deteriorating test cases and machine learning models’ mortality risk predictions by LR, CW-LSTM, and LSTM models. a–c show the average time series of the generated abnormal test cases (in red area curves) and the normal seed cases used (in blue area curves) for each of the 3 attributes. d Models’ predicted average mortality risks for each deteriorating attribute. The standard deviation is indicated by the error bar and the number of detected cases out of 3 is shown in red.
Fig. 5 Gradient-generated deteriorating test cases and machine learning models’ mortality risk predictions by LR, CW-LSTM, and LSTM models. a–c show the average time series of the generated abnormal test cases (in red area curves) and the normal seed cases used (in blue area curves) for each of the 3 attributes. d Models’ predicted average mortality risks for each deteriorating attribute. The standard deviation is indicated by the error bar and the number of detected cases out of 3 is shown in red.CC BY-NC-ND · PubMed Central
Fig. 6 Predicted 5-year breast cancer survivability results of a multi-layer perceptron (MLP) model on test cases. Four major breast cancer screen attributes are involved, including CS tumor size, number of positive lymph nodes, number of lymph nodes examined, and grade. a–c, g Predicted survivability results on single-attribute varying test cases. The blue shaded area of a–c represents the standard deviation calculated from 3 independent experimental trials (models). d–f Predicted survivability results on double-attribute varying test cases. h Predicted survivability results on triple-attribute varying test cases involving CS tumor size, number of positive lymph nodes, and grade. In the boxplots (g) and (h) the horizontal line within the box represents the median value, while the box itself encompasses the interquartile range (IQR), containing the middle 50% of the data. The whiskers extend to the values within 1.5 times the IQR from the box (upper and lower quartiles). The green triangle point on the box represents the mean of the distribution. The points on boxplots (g) and (h) represent 21,723 from class 0 (C0) and 3152 samples from class 1 (C1).
Fig. 6 Predicted 5-year breast cancer survivability results of a multi-layer perceptron (MLP) model on test cases. Four major breast cancer screen attributes are involved, including CS tumor size, number of positive lymph nodes, number of lymph nodes examined, and grade. a–c, g Predicted survivability results on single-attribute varying test cases. The blue shaded area of a–c represents the standard deviation calculated from 3 independent experimental trials (models). d–f Predicted survivability results on double-attribute varying test cases. h Predicted survivability results on triple-attribute varying test cases involving CS tumor size, number of positive lymph nodes, and grade. In the boxplots (g) and (h) the horizontal line within the box represents the median value, while the box itself encompasses the interquartile range (IQR), containing the middle 50% of the data. The whiskers extend to the values within 1.5 times the IQR from the box (upper and lower quartiles). The green triangle point on the box represents the mean of the distribution. The points on boxplots (g) and (h) represent 21,723 from class 0 (C0) and 3152 samples from class 1 (C1).CC BY-NC-ND · PubMed Central
Fig. 7 Performance comparison between the original machine learning models and the resampled (SMOTE or AdaSyn) or reweighted models under single-attribute varying tests. a–c and d–f Mortality risk prediction results by the original LSTM model and the resampled or reweighted LSTM models under MIMIC-III and eICU test cases for respiratory rate, temperature, and systolic blood pressure, respectively. g–i and j–l 5-year cancer survivability prediction results by the original MLP model and the resampled or reweighted MLP models under SEER BCS and LCS test cases for CS tumor size, the number of positive lymph nodes, and the number of lymph nodes examined, respectively. Horizontal dashed lines represent model-specific thresholds. The shading represents the standard deviation calculated from 3 independent experimental trials (models).
Fig. 7 Performance comparison between the original machine learning models and the resampled (SMOTE or AdaSyn) or reweighted models under single-attribute varying tests. a–c and d–f Mortality risk prediction results by the original LSTM model and the resampled or reweighted LSTM models under MIMIC-III and eICU test cases for respiratory rate, temperature, and systolic blood pressure, respectively. g–i and j–l 5-year cancer survivability prediction results by the original MLP model and the resampled or reweighted MLP models under SEER BCS and LCS test cases for CS tumor size, the number of positive lymph nodes, and the number of lymph nodes examined, respectively. Horizontal dashed lines represent model-specific thresholds. The shading represents the standard deviation calculated from 3 independent experimental trials (models).CC BY-NC-ND · PubMed Central
Fig. 8 Performance and responsiveness of the transformer model compared with LSTM. Figures (a) and (b) show the various Class 1 (death) and Class 0 (survival) performance of the transformer model trained and tested on the original MIMIC-III and eICU datasets, respectively, with error bars indicating the standard deviation from three experimental trials. The dashed line represents the LSTM model’s performance. Figure (c–e) shows the predicted mortality risk by the transformer model for respiratory rate, temperature, and systolic blood pressure on MIMIC-III single-attribute test cases, while (f–h) shows the same for eICU test cases. Horizontal dashed lines denote model-specific thresholds for mortality risk prediction. Rec_C1, Pre_C1, F1_C1, AU_PRC_C1, Rec_C0, Pre_C0, F1_C0, AU_PRC_C0, Accuracy, Bal_Acc, and AUROC stand for Recall Class 1, Precision Class 1, F1 score Class 1, Area Under the Precision-Recall Curve Class 1, Recall Class 0, Precision Class 0, F1 score Class 0, Area Under the Precision-Recall Curve Class 0, Accuracy, Balanced Accuracy, and Area under the Receiver Operating Curve, respectively. The shading represents the standard deviation calculated from 3 independent experimental trials (models).
Fig. 8 Performance and responsiveness of the transformer model compared with LSTM. Figures (a) and (b) show the various Class 1 (death) and Class 0 (survival) performance of the transformer model trained and tested on the original MIMIC-III and eICU datasets, respectively, with error bars indicating the standard deviation from three experimental trials. The dashed line represents the LSTM model’s performance. Figure (c–e) shows the predicted mortality risk by the transformer model for respiratory rate, temperature, and systolic blood pressure on MIMIC-III single-attribute test cases, while (f–h) shows the same for eICU test cases. Horizontal dashed lines denote model-specific thresholds for mortality risk prediction. Rec_C1, Pre_C1, F1_C1, AU_PRC_C1, Rec_C0, Pre_C0, F1_C0, AU_PRC_C0, Accuracy, Bal_Acc, and AUROC stand for Recall Class 1, Precision Class 1, F1 score Class 1, Area Under the Precision-Recall Curve Class 1, Recall Class 0, Precision Class 0, F1 score Class 0, Area Under the Precision-Recall Curve Class 0, Accuracy, Balanced Accuracy, and Area under the Receiver Operating Curve, respectively. The shading represents the standard deviation calculated from 3 independent experimental trials (models).CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 48 references

Research works printed but not linked: 3

  1. Johnson, A., Pollard, T. & Mark, R. MIMIC-III clinical database (version 1.4). PhysioNet 10, 2 (2016).
  2. Finlayson, S. G., Chung, H. W., Kohane, I. S. & Beam, A. L. Adversarial attacks against medical deep learning systems. arXiv preprint arXiv:1804.05296 (2018). · arXiv
  3. Lundberg, S. M., & Lee, S. I. A unified approach to interpreting model predictions. Proceedings of the Advances in Neural Information Processing Systems 30 (Curran Associates, Inc. 2017).

Cited by

See all 10

Metadata

Authors
Pias, T. S., Afrose, S., Tuli, M. D., Trisha, I. H., Deng, X., Nemeroff, C. B., & Yao, D. D.
Affiliations
  • Virginia Tech (US)
  • Oak Ridge National Laboratory (US)
  • Green Life Medical College (BD)
  • University of Arizona (US)
  • The University of Texas at Austin (US)
Corresponding author
Yao, Danfeng Daphne
Venue
Communications Medicine
Published
2025-03-11
Volume
5
Issue
1
Pages
62-62
ISSN
2730-664X (ISSN-L)
OpenAlex
W4408341851
Cited by
10
References
48
Citations by year
2026: 5 · 2025: 5
FWCI
4.01
Type
journal article
Language
en
Open access
Licence
Locations
  • Communications Medicine (publishedVersion, cc-by-nc-nd)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • VTechWorks (Virginia Tech) (publishedVersion, cc-by-nc-nd)
  • OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
machine learning model evaluation, mortality prediction, intensive care unit, gradient ascent, synthetic test cases, in-hospital mortality, breast cancer prediction, lung cancer prediction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Low responsiveness of machine learning models to critical or deteriorating health conditions. https://app.cochat.ai/papers/10.1038/s43856-025-00775-0
The paper
Pias, T. S., Afrose, S., Tuli, M. D., Trisha, I. H., Deng, X., Nemeroff, C. B., & Yao, D. D. (2025). Low responsiveness of machine learning models to critical or deteriorating health conditions. Communications Medicine. https://doi.org/10.1038/s43856-025-00775-0

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.