The DOI matches its Crossref record; 45 of the 48 references it cites were found in a registry or an index.
Low responsiveness of machine learning models to critical or deteriorating health conditions
Open accessPublished version of a preprint
Abstract
BACKGROUND: Machine learning (ML) based mortality prediction models can be immensely useful in intensive care units. Such a model should generate warnings to alert physicians when a patient's condition rapidly deteriorates, or their vitals are in highly abnormal ranges. Before clinical deployment, it is important to comprehensively assess a model's ability to recognize critical patient conditions. METHODS: We develop multiple medical ML testing approaches, including a gradient ascent method and neural activation map. We systematically assess these machine learning models' ability to respond to serious medical conditions using additional test cases, some of which are time series. Guided by medical doctors, our evaluation involves multiple machine learning models, resampling techniques, and four datasets for two clinical prediction tasks. RESULTS: We identify serious deficiencies in the models' responsiveness, with the models being unable to recognize severely impaired medical conditions or rapidly deteriorating health. For in-hospital mortality prediction, the models tested using our synthesized cases fail to recognize 66% of the injuries. In some instances, the models fail to generate adequate mortality risk scores for all test cases. Our study identifies similar kinds of deficiencies in the responsiveness of 5-year breast and lung cancer prediction models. CONCLUSIONS: Using generated test cases, we find that statistical machine-learning models trained solely from patient data are grossly insufficient and have many dangerous blind spots. Most of the ML models tested fail to respond adequately to critically ill patients. How to incorporate medical knowledge into clinical machine learning models is an important future research direction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A recent study clearly demonstrated this issue, revealing that AI models missed approximately 66% of life-threatening injuries in hospitalized patients, underscoring the urgent need for robust calibration procedures [5].”
From Enhancing Clinician Trust in AI Diagnostics: A Dynamic Framework for Confidence Calibration and Transparency (2025) · cites this paper as [5]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 1×
“13 out of the 17 attributes have negative coef fi cients, e.g., the temperature is strongly inversely correlated with the predicted probability (Supplementary Fig.”
Topics in Optimal Transportation
Villani, C. (2003) · Graduate Studies in Mathematics · doi:10.1090/gsm/058 · Cited by 4,412
Pollard, T. J., Johnson, A. E. W., Raffa, J. D., Celi, L. A., Mark, R. G., & Badawi, O. (2018) · Scientific Data · doi:10.1038/sdata.2018.178 · Cited by 1,993
Abràmoff, M. D., Lavin, P. T., Birch, M., Shah, N., & Folk, J. C. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0040-6 · Cited by 1,638
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217 · cited 1×
“Recently, generative technologies have been proposed to produce curated manmade images for testing self-driving vehicles 9,10 .”
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217 · cited 1×
“Recently, generative technologies have been proposed to produce curated manmade images for testing self-driving vehicles 9,10 .”
Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194 · cited 1×
“In image and natural language domains, a common testing approach is adversarial attacks 39 – 42 .”
Research works printed but not linked: 3
- Johnson, A., Pollard, T. & Mark, R. MIMIC-III clinical database (version 1.4). PhysioNet 10, 2 (2016).
- Finlayson, S. G., Chung, H. W., Kohane, I. S. & Beam, A. L. Adversarial attacks against medical deep learning systems. arXiv preprint arXiv:1804.05296 (2018). · arXiv
- Lundberg, S. M., & Lee, S. I. A unified approach to interpreting model predictions. Proceedings of the Advances in Neural Information Processing Systems 30 (Curran Associates, Inc. 2017).
Cited by
Yu, Y., Gomez-Cabello, C. A., Haider, S. A., Genovese, A., Prabha, S., Trabilsy, M., Collaco, B. G., Wood, N. G., Bagaria, S., Tao, C., et al. (2025) · Diagnostics · doi:10.3390/diagnostics15172204 · Cited by 32
“A recent study clearly demonstrated this issue, revealing that AI models missed approximately 66% of life-threatening injuries in hospitalized patients, underscoring the urgent need for robust calibration procedures [5].”
Imaezue, G. C., & Marampelly, H. (2025) · Journal of Speech Language and Hearing Research · doi:10.1044/2025_jslhr-25-00003 · Cited by 9
“However, AI-driven clinical care also poses risks such as inaccurate or biased outputs, reduced human oversight, and potential loss of human connection in therapy (Botha et al., 2024; Karimian et al., 2022; Pias et al., 2025).”
Enhancing Fairness and Accuracy in Diagnosing Type 2 Diabetes in Young Adult Population
Pias, T. S., Su, Y., Tang, X., Wang, H., Faghani, S., & Yao, D. D. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3616312 · Cited by 17
“However, health data can be imbalanced, which could potentially lead machine learning models to learn patterns with existing bias from the provided data [19], [20], [21], [22] causing low performance [23].”
Self-citationWang, Y., Islam, M. M., Okere, A. N., Poly, T. N., & Lin, M. (2026) · Endocrine · doi:10.1007/s12020-025-04514-2 · Cited by 2
“Indeed, missing high-risk patients (false negatives) could delay critical care, so future work should focus on optimizing recall through cost-sensitive learning or threshold adjustments [65].”
Safeguarding AI-driven digital health – An adaptation of the Swiss cheese model for safety
Denecke, K. (2026) · Digital Health · doi:10.1177/20552076261459551 · Cited by 2
Metadata
- Authors
- Pias, T. S., Afrose, S., Tuli, M. D., Trisha, I. H., Deng, X., Nemeroff, C. B., & Yao, D. D.
- Affiliations
- Virginia Tech (US)
- Oak Ridge National Laboratory (US)
- Green Life Medical College (BD)
- University of Arizona (US)
- The University of Texas at Austin (US)
- Corresponding author
- Yao, Danfeng Daphne
- Venue
- Communications Medicine
- Published
- 2025-03-11
- Volume
- 5
- Issue
- 1
- Pages
- 62-62
- ISSN
- 2730-664X (ISSN-L)
- PMC
- PMC11897252
- OpenAlex
- W4408341851
- Cited by
- 10
- References
- 48
- Citations by year
- 2026: 5 · 2025: 5
- FWCI
- 4.01
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Communications Medicine (publishedVersion, cc-by-nc-nd)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- VTechWorks (Virginia Tech) (publishedVersion, cc-by-nc-nd)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- machine learning model evaluation, mortality prediction, intensive care unit, gradient ascent, synthetic test cases, in-hospital mortality, breast cancer prediction, lung cancer prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Low responsiveness of machine learning models to critical or deteriorating health conditions. https://app.cochat.ai/papers/10.1038/s43856-025-00775-0
- The paper
- Pias, T. S., Afrose, S., Tuli, M. D., Trisha, I. H., Deng, X., Nemeroff, C. B., & Yao, D. D. (2025). Low responsiveness of machine learning models to critical or deteriorating health conditions. Communications Medicine. https://doi.org/10.1038/s43856-025-00775-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.