Evaluating a Widely Implemented Proprietary Deterioration Index Model among Hospitalized Patients with COVID-19
Open accessPublished version of a preprint
Abstract
Rationale: The Epic Deterioration Index (EDI) is a proprietary prediction model implemented in over 100 U.S. hospitals that was widely used to support medical decision-making during the coronavirus disease (COVID-19) pandemic. The EDI has not been independently evaluated, and other proprietary models have been shown to be biased against vulnerable populations. Objectives: To independently evaluate the EDI in hospitalized patients with COVID-19 overall and in disproportionately affected subgroups. Methods: We studied adult patients admitted with COVID-19 to units other than the intensive care unit at a large academic medical center from March 9 through May 20, 2020. We used the EDI, calculated at 15-minute intervals, to predict a composite outcome of intensive care unit–level care, mechanical ventilation, or in-hospital death. In a subset of patients hospitalized for at least 48 hours, we also evaluated the ability of the EDI to identify patients at low risk of experiencing this composite outcome during their remaining hospitalization. Results: Among 392 COVID-19 hospitalizations meeting inclusion criteria, 103 (26%) met the composite outcome. The median age of the cohort was 64 (interquartile range, 53–75) with 168 (43%) Black patients and 169 (43%) women. The area under the receiver-operating characteristic curve of the EDI was 0.79 (95% confidence interval, 0.74–0.84). EDI predictions did not differ by race or sex. When exploring clinically relevant thresholds of the EDI, we found patients who met or exceeded an EDI of 68.8 made up 14% of the study cohort and had a 74% probability of experiencing the composite outcome during their hospitalization with a sensitivity of 39% and a median lead time of 24 hours from when this threshold was first exceeded. Among the 286 patients hospitalized for at least 48 hours who had not experienced the composite outcome, 14 (13%) never exceeded an EDI of 37.9, with a negative predictive value of 90% and a sensitivity above this threshold of 91%. Conclusions: We found the EDI identifies small subsets of high-risk and low-risk patients with COVID-19 with good discrimination, although its clinical use as an early warning system is limited by low sensitivity. These findings highlight the importance of independent evaluation of proprietary models before widespread operational use among patients with COVID-19.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Vendors increasingly provide proprietary "turnkey" CDSS solutions for identifying clinical deterioration and sepsis.
“Vendors increasingly provide proprietary “turnkey” CDSS solutions for identifying clinical deterioration and sepsis [42–44].”
From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [44]
The Methods section (‘The Epic Deterioration Index Model’) describes a proprietary model developed by a healthcare software vendor to detect clinical deterioration, and the Abstract says it is implemented in hundreds of U.S. hospitals. The paper does not establish that vendors are increasingly offering such solutions, or that this model identifies sepsis; it also does not characterize it as ‘turnkey.’ (LLM, conf=0.85)
Experimental — accuracy not yet measuredSingh 2021 had a high risk of selection bias due to restricting validation to COVID-19 patients.
“Singh 2021 21 had a high risk of selection bias due to restricting validation to COVID-19 patients.”
From A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools (2026) · cites this paper as [21]
The Methods (Study Cohort) restrict the cohort to adults hospitalized with COVID-19. The Discussion says thresholds may differ for patients hospitalized with other conditions and calls for broader validation, but it does not assess this restriction as a high risk of selection bias. (LLM, conf=0.83)
Experimental — accuracy not yet measuredMore than a quarter of hospitalized patients in the first wave of the COVID-19 pandemic in 2020 required a transfer to an intensive care unit or died.
“For example, more than a quarter of hospitalized patients in the first wave of the COVID-19 pandemic in 2020 required a transfer to an intensive care unit or died.6”
From Quantification of Sepsis Model Alerts in 24 US Hospitals Before and During the COVID-19 Pandemic (2021) · cites this paper as [6]
The Results report that 103 of 392 hospitalizations (26%) had a composite adverse outcome, and the Methods define that outcome to include ICU-level care or in-hospital death. But the study covered eligible admissions at one Michigan academic medical center from March to May 2020, excluded patients admitted directly to the ICU, and also included mechanical ventilation in the composite. Its 26% figure therefore does not establish the stated rate for hospitalized patients generally or precisely for ICU transfer or death alone. (LLM, conf=0.87)
Experimental — accuracy not yet measuredEarlier predictions provide clinicians with greater lead time for action, ultimately enabling the care team to identify patients who may be entering a risky state with enough time to intervene.
“Earlier predictions provide clinicians with greater lead time for action24, ultimately enabling the care team to identify patients who may be entering a risky state with enough time to intervene10.”
From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Richardson, S., Hirsch, J. S., Narasimhan, M., Crawford, J. M., McGinn, T., Davidson, K. W., and the Northwell COVID-19 Research Consortium, Barnaby, D. P., Becker, L. B., Chelico, J. D., et al. (2020) · JAMA · doi:10.1001/jama.2020.6775 · Cited by 9,254 · see citing works
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · see citing works
Fair Allocation of Scarce Medical Resources in the Time of Covid-19
Emanuel, E., Persad, G., Upshur, R. E. G., Thome, B. D. C., Parker, M., Glickman, A., Zhang, C., Boyle, C., Smith, M. J., & Phillips, J. P. (2020) · New England Journal of Medicine · doi:10.1056/nejmsb2005114 · Cited by 3,143 · see citing works
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
A targeted real-time early warning score (TREWScore) for septic shock.
Henry, K. E., Hager, D. N., Pronovost, P. J., & Saria, S. (2015) · Science Translational Medicine · doi:10.1126/scitranslmed.aab3719 · Cited by 598 · see citing works
Face recognition vendor test part 3:
Grother, P. J., Ngan, M. L., & Hanaoka, K. (2019) · doi:10.6028/nist.ir.8280 · Cited by 332 · see citing works
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312 · see citing works
Gu, T., Mack, J. A., Salvatore, M., Sankar, S. P., Valley, T. S., Singh, K., Nallamothu, B. K., Kheterpal, S., Lisabeth, L. D., Fritsche, L. G., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.25197 · Cited by 263 · see citing works
Self-citationPredictive analytics in health care: how can we know it works?
Van Calster, B., Wynants, L., Timmerman, D., Steyerberg, E. W., & Collins, G. S. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz130 · Cited by 223 · see citing works
Kipnis, P., Turk, B. J., Wulf, D. A., LaGuardia, J. C., Liu, V. X., Churpek, M. M., Brufau, S. R., & Escobar, G. J. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.09.013 · Cited by 167 · see citing works
Green, M., Lander, H., Snyder, A. M., Hudson, P., Churpek, M. M., & Edelson, D. P. (2017) · Resuscitation · doi:10.1016/j.resuscitation.2017.10.028 · Cited by 149 · see citing works
Why the C-statistic is not informative to evaluate early warning scores and what metrics to use
Brufau, S. R., Huddleston, J. M., Escobar, G. J., & Liebow, M. (2015) · Critical Care · doi:10.1186/s13054-015-0999-1 · Cited by 141 · see citing works
Oh, J., Makar, M., Fusco, C., McCaffrey, R., Rao, K. K., Ryan, E. E., Washer, L., West, L. R., Young, V. B., Guttag, J. V., et al. (2018) · Infection Control and Hospital Epidemiology · doi:10.1017/ice.2018.16 · Cited by 124 · see citing works
Self-citationPredicting COVID-19 Malignant Progression with AI TechniquesPreprint
Bai, X., Fang, C., Zhou, Y., Bai, S., Liu, Z., Xia, L., Chen, Q., Xu, Y., Xia, T., Gong, S., et al. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3557984 · Cited by 97 · see citing works
Rothman Index variability predicts clinical deterioration and rapid response activation
Wengerter, B. C., Pei, K. Y., Asuzu, D., & Davis, K. A. (2017) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2017.07.031 · Cited by 30 · see citing works
Machine Learning in Clinical Journals
Singh, K., Beam, A. L., & Nallamothu, B. K. (2020) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.120.007491 · Cited by 10 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- [6]Strickland E AI can help hospitals triage COVID-19 patients IEEE Spectrum 2020 [updated 2020 Apr 21; accessed 2020 May 1]. Available from: https://spectrum.ieee.org/the-human-os/artificial-intelligence/medical-ai/ai-can-help-hospitals-triage-covid19-patients
Cited by
Sorted by citations
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995 · see citing works
Self-citationArtificial Intelligence -based technologies in nursing: A scoping literature review of the evidence
von Gerich, H., Moen, H., Block, L. J., Chu, C. H., DeForest, H., Hobensack, M., Michalowski, M., Mitchell, J., Nibber, R., Olalia, M. A., et al. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2021.104153 · Cited by 360 · see citing works
Armoundas, A. A., Narayan, S. M., Arnett, D. K., Spector-Bagdady, K., Bennett, D. A., Celi, L. A., Friedman, P. A., Gollob, M. H., Hall, J. L., Kwitek, A. E., et al. (2024) · Circulation · doi:10.1161/cir.0000000000001201 · Cited by 347 · see citing works
Development of a prognostic model for mortality in COVID-19 infection using machine learning
Booth, A. L., Abels, E. A., & McCaffrey, P. (2020) · Modern Pathology · doi:10.1038/s41379-020-00700-x · Cited by 182 · see citing works
Does “ AI ” stand for augmenting inequality in the era of covid-19 healthcare?
Leslie, D., Mazumder, A., Peppin, A., Wolters, M. K., & Hagerty, A. (2021) · BMJ · doi:10.1136/bmj.n304 · Cited by 177 · see citing works
Use of artificial intelligence in critical care: opportunities and obstacles
Pinsky, M. R., Bedoya, A., Bihorac, A., Celi, L., Churpek, M., Economou-Zavlanos, N. J., Elbers, P., Saria, S., Liu, V., Lyons, P. G., et al. (2024) · Critical Care · doi:10.1186/s13054-024-04860-z · Cited by 160 · see citing works
“Vendors increasingly provide proprietary “turnkey” CDSS solutions for identifying clinical deterioration and sepsis [42–44].”
BackgroundBurdick, H., Lam, C., Mataraso, S. J., Siefkas, A. C., Braden, G. L., Dellinger, R. P., McCoy, A., Vincent, J., Green‐Saxena, A., Barnes, G., et al. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103949 · Cited by 149 · see citing works
Syrowatka, A., Kuznetsova, M., Alsubai, A., Beckman, A. L., Bain, P. A., Craig, K. J. T., Hu, J., Jackson, G. P., Rhee, K., & Bates, D. W. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00459-8 · Cited by 130 · see citing works
Artificial intelligence-enabled decision support in nephrology
Loftus, T. J., Shickel, B., Ozrazgat‐Baslanti, T., Ren, Y., Glicksberg, B. S., Cao, J., Singh, K., Chan, L., Nadkarni, G. N., & Bihorac, A. (2022) · Nature Reviews Nephrology · doi:10.1038/s41581-022-00562-3 · Cited by 109 · see citing works
Self-citationThe Internet of Federated Things (IoFT)
Kontar, R. A., Shi, N., Yue, X., Chung, S., Byon, E., Chowdhury, M., Jin, J. J., Kontar, W., Masoud, N., Nouiehed, M., et al. (2021) · IEEE Access · doi:10.1109/access.2021.3127448 · Cited by 67 · see citing works
Self-citationKamran, F., Tang, S., Otles, E., McEvoy, D. S., Saleh, S. N., Gong, J., Li, B. Y., Dutta, S., Liu, X., Medford, R. J., et al. (2022) · BMJ · doi:10.1136/bmj-2021-068576 · Cited by 60 · see citing works
Self-citationQuantification of Sepsis Model Alerts in 24 US Hospitals Before and During the COVID-19 Pandemic
Wong, A., Cao, J., Lyons, P. G., Dutta, S., Major, V. J., Ötleş, E., & Singh, K. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.35286 · Cited by 57 · see citing works
Self-citationRossetti, S. C., Dykes, P. C., Knaplund, C., Cho, S., Withall, J., Lowenthal, G., Albers, D., Lee, R. Y., Jia, H., Bakken, S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03609-7 · Cited by 54 · see citing works
“Earlier predictions provide clinicians with greater lead time for action 24 , ultimately enabling the care team to identify patients who may be entering a risky state with enough time to intervene 10 .”
van der Vegt, A. H., Campbell, V. K., Mitchell, I., Malycha, J., Simpson, J., Flenady, T., Flabouris, A., Lane, P. J., Mehta, N., Kalke, V. R., et al. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad220 · Cited by 53 · see citing works
Generalizability of an acute kidney injury prediction model across health systems
Cao, J., Zhang, X., Shahinian, V. B., Yin, H., Steffick, D., Saran, R., Crowley, S. T., Mathis, M. R., Nadkarni, G. N., Heung, M., et al. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00563-8 · Cited by 49 · see citing works
Self-citationSmart Diagnostics: Combining Artificial Intelligence and In Vitro Diagnostics
McRae, M. P., Rajsri, K. S., Alcorn, T. M., & McDevitt, J. T. (2022) · Sensors · doi:10.3390/s22176355 · Cited by 45 · see citing works
A proposal for developing a platform that evaluates algorithmic equity and accuracy
Cerrato, P. L., Halamka, J. D., & Pencina, M. J. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100423 · Cited by 43 · see citing works
McRae, M. P., Dapkins, I. P., Sharif, I., Anderman, J., Fenyö, D., Sinokrot, O., Kang, S. K., Christodoulides, N. J., Vurmaz, D., Simmons, G. W., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/22033 · Cited by 40 · see citing works
Putting explainable AI in context: institutional explanations for medical AI
Theunissen, M., & Browning, J. (2022) · Ethics and Information Technology · doi:10.1007/s10676-022-09649-8 · Cited by 40 · see citing works
Li, R., Smith, M. A., Lu, J., Avati, A., Wang, S. X. Y., Teuteberg, W., Shum, K., Hong, G., Seevaratnam, B., Westphal, J., et al. (2022) · NEJM Catalyst · doi:10.1056/cat.21.0457 · Cited by 37 · see citing works
Cummings, B. C., Ansari, S., Motyka, J. R., Wang, G., Medlin Jr, R. P., Kronick, S. L., Singh, K., Park, P. K., Napolitano, L. M., Dickson, R. P., et al. (2021) · JMIR Medical Informatics · doi:10.2196/25066 · Cited by 35 · see citing works
Self-citationArtificial intelligence in rheumatoid arthritis: potential applications and future implications
Gilvaz, V. J., & Reginato, A. M. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1280312 · Cited by 36 · see citing works
Bridging the gap: Towards an expanded toolkit for AI-driven decision-making in the public sector
Fischer-Abaigar, U., Kern, C. A., Barda, N., & Kreuter, F. (2024) · Government Information Quarterly · doi:10.1016/j.giq.2024.101976 · Cited by 49 · see citing works
Validation of a Proprietary Deterioration Index Model and Performance in Hospitalized Adults
Byrd, T. F., Southwell, B. J., Ravishankar, A., Tran, T., Kc, A., Phelan, T., Melton, G. B., Usher, M. G., Scheppmann, D., Switzer, S., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.24176 · Cited by 31 · see citing works
Mou, Z., Godat, L. N., El-Kareh, R. E., Berndtson, A. E., Doucet, J. J., & Costantini, T. W. (2021) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0000000000003431 · Cited by 27 · see citing works
Velasco, F. T., Yang, D. M., Zhang, M., Nelson, T. L., Sheffield, T. Y., Keller, T., Wang, Y., Walker, C., Katterapalli, C., Zimmerman, K., et al. (2021) · Journal of Hospital Medicine · doi:10.12788/jhm.3717 · Cited by 26 · see citing works
Lupei, M. I., Li, D., Ingraham, N. E., Baum, K. D., Benson, B., Puskarich, M. A., Milbrandt, D., Melton, G. B., Scheppmann, D., Usher, M. G., et al. (2022) · PLoS ONE · doi:10.1371/journal.pone.0262193 · Cited by 26 · see citing works
Rong, R., Gu, Z., Lai, H., Nelson, T. L., Keller, T., Walker, C., Jin, K. W., Chen, C., Návar, A. M., Velasco, F. T., et al. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf026 · Cited by 30 · see citing works
Addressing ethical issues in healthcare artificial intelligence using a lifecycle-informed process
Collins, B. X., Bélisle-Pipon, J.-C., Evans, B. J., Ferryman, K., Jiang, X., Nebeker, C., Novak, L., Roberts, K., Were, M., Yin, Z., et al. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae108 · Cited by 21 · see citing works
Hui, D., Nortjé, N., George, M. C., Wilson, K. J., Urbauer, D. L., Lenz, C., Wallace, S. K., Andersen, C. R., Mendoza, T. R., Haque, S. A., et al. (2022) · Journal of Clinical Oncology · doi:10.1200/jco.22.00849 · Cited by 21 · see citing works
The future of automated infection detection: Innovation to transform practice (Part III/III)
Branch‐Elliman, W., Sundermann, A. J., Wiens, J., & Shenoy, E. S. (2023) · Antimicrobial Stewardship & Healthcare Epidemiology · doi:10.1017/ash.2022.333 · Cited by 19 · see citing works
Self-citationBraun, E. J., Singh, S., Penlesky, A. C., Strong, E. A., Holt, J. M., Fletcher, K. E., Stadler, M., Nattinger, A. B., & Crotty, B. H. (2022) · BMJ Quality & Safety · doi:10.1136/bmjqs-2021-014498 · Cited by 22 · see citing works
Early Warning Scores and Their Application in the Inpatient Oncology Settings
Nagarajah, S., Krzyzanowska, M. K., & Murphy, T. (2022) · JCO Oncology Practice · doi:10.1200/op.21.00532 · Cited by 19 · see citing works
Cummings, B. C., Blackmer, J. M., Motyka, J. R., Farzaneh, N., Cao, L., Bisco, E. L., Glassbrook, J. D., Roebuck, M. D., Gillies, C. E., Admon, A. J., et al. (2023) · Critical Care Medicine · doi:10.1097/ccm.0000000000005837 · Cited by 19 · see citing works
Self-citationAn Analysis COVID-19 in Mexico: a Prediction of Severity
Martínez‐Martínez, M. U., Alpizar-Rodriguez, D., Flores‐Ramírez, R., Portales‐Pérez, D. P., Soria‐Guerra, R. E., Pérez‐Vázquez, F. J., & Martínez‐Gutiérrez, F. (2022) · Journal of General Internal Medicine · doi:10.1007/s11606-021-07235-0 · Cited by 17 · see citing works
Harish, K. B., Price, W. N., & Aphinyanaphongs, Y. (2022) · JMIR Formative Research · doi:10.2196/33970 · Cited by 17 · see citing works
Steitz, B. D., McCoy, A. B., Reese, T. J., Liu, S., Weavind, L., Shipley, K., Russo, E. M., & Wright, A. T. (2023) · Journal of General Internal Medicine · doi:10.1007/s11606-023-08349-3 · Cited by 19 · see citing works
Learning From Past Respiratory Infections to Predict COVID-19 Outcomes: Retrospective Study
Sang, S., Sun, R., Coquet, J., Carmichael, H. L., Seto, T., & Hernandez‐Boussard, T. (2021) · Journal of Medical Internet Research · doi:10.2196/23026 · Cited by 14 · see citing works
Adamuz, J., González‐Samartino, M., Jiménez‐Martínez, E., Tàpia‐Pérez, M., López‐Jiménez, M., Rodríguez‐Fernández, H., Castro-Navarro, T., Zuriguel‐Pérez, E., Carratalà, J., & Juvé‐Udina, M. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-041726 · Cited by 16 · see citing works
Nuevo-Ortega, P., Reina-Artacho, C., Dominguez-Moreno, F., Becerra‐Muñoz, V. M., Ruiz-Del-Fresno, L., Estecha-Foncea, M. A., Aguilar-Galvez, A. M., Barrera-Serrano, R., Becerra‐Muñoz, V. M., Cabrera-César, E., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-06199-3 · Cited by 16 · see citing works
Salehinejad, H., Meehan, A. M., Rahman, P., Core, M. A., Borah, B. J., & Caraballo, P. J. (2023) · EClinicalMedicine · doi:10.1016/j.eclinm.2023.102312 · Cited by 20 · see citing works
Effects of hydroxychloroquine and its metabolites in patients with connective tissue diseases
Onmaz, D. E., Tezcan, D., Abuşoğlu, S., Yilmaz, S., Yerlikaya, F. H., Onmaz, M., Abuşoğlu, G., & Ünlü, A. (2021) · Inflammopharmacology · doi:10.1007/s10787-021-00887-8 · Cited by 14 · see citing works
Çelik, N. Ç., & Kılınç, E. A. (2025) · Clinical Rheumatology · doi:10.1007/s10067-025-07417-9 · Cited by 14 · see citing works
A new tool for evaluating health equity in academic journals; the Diversity Factor
Gallifant, J., Zhang, J., Whebell, S. F., Quion, J., Escobar, B., Gichoya, J. W., Herrera, K. V., Jina, R., Chidambaram, S., Mehndiratta, A., et al. (2023) · PLOS Global Public Health · doi:10.1371/journal.pgph.0002252 · Cited by 14 · see citing works
How US law will evaluate artificial intelligence for covid-19
Krass, M., Henderson, P., Mello, M. M., Studdert, D. M., & Ho, D. E. (2021) · BMJ · doi:10.1136/bmj.n234 · Cited by 13 · see citing works
Harnessing AI in critical care: opportunities, challenges and key steps for success
Kalimouttou, A., Stevens, R. D., & Pirracchio, R. (2026) · Thorax · doi:10.1136/thorax-2024-222125 · Cited by 12 · see citing works
Artificial Intelligence in migrant health: a critical perspective on opportunities and risks
Matlin, S. A., Claron, I. M. M., Merone, J., Netto, G., Takian, A., Zaman, M. H., & Saso, L. (2025) · The Lancet Regional Health - Europe · doi:10.1016/j.lanepe.2025.101421 · Cited by 13 · see citing works
Off-label use of artificial intelligence models in healthcare
Krishnamoorthy, M., Sjoding, M. W., & Wiens, J. (2024) · Nature Medicine · doi:10.1038/s41591-024-02870-6 · Cited by 12 · see citing works
Self-citationDechgummarn, Y., Fuangfoo, P., & Kampeerawat, W. (2023) · IEEE Access · doi:10.1109/access.2023.3340515 · Cited by 11 · see citing works
Simons, L. M., Lorenzo-Redondo, R., Gibson, M., Kinch, S. L., Vandervaart, J. P., Reiser, N. L., Eren, M., Lux, E., McNally, E. M., Tambur, A. R., et al. (2022) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofac027 · Cited by 11 · see citing works
Dadabhoy, F. Z., Driver, L., McEvoy, D. S., Stevens, R., Rubins, D., & Dutta, S. (2023) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2022.11.012 · Cited by 10 · see citing works
Smith, A. J., Patterson, B. W., Pulia, M. S., Mayer, J. G., Schwei, R. J., Nagarajan, R., Liao, F., Shah, M. N., & Boutilier, J. J. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac214 · Cited by 9 · see citing works
Contrastive Transfer Learning for Prediction of Adverse Events in Hospitalized Patients
Salehinejad, H., Meehan, A. M., Caraballo, P. J., & Borah, B. J. (2023) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2023.3344035 · Cited by 9 · see citing works
Li, D., Hu, L., Peng, X., Xiao, N., Zhao, H., Liu, G., Liu, H., Li, K., Ai, B., Xia, H., et al. (2022) · iScience · doi:10.1016/j.isci.2022.103961 · Cited by 10 · see citing works
Kulkarni, N. S., Landler, M. P., Cohen, E. R., Wayne, D. B., & Szmuilowicz, E. (2025) · Journal of Hospital Medicine · doi:10.1002/jhm.70012 · Cited by 8 · see citing works
Weissman, G. E., & Liu, V. X. (2021) · Current Opinion in Critical Care · doi:10.1097/mcc.0000000000000855 · Cited by 8 · see citing works
Benson, B. E., Belle, A., Lee, S., Bassin, B. S., Medlin, R. P., Sjoding, M. W., & Ward, K. R. (2023) · BMC Anesthesiology · doi:10.1186/s12871-023-02283-x · Cited by 8 · see citing works
Self-citationToward Realizing the Promise of AI in Precision Health Across the Spectrum of Care
Wiens, J., Spector‐Bagdady, K., & Mukherjee, B. (2024) · Annual Review of Genomics and Human Genetics · doi:10.1146/annurev-genom-010323-010230 · Cited by 7 · see citing works
Self-citationThe missing value of medical artificial intelligence
Kohane, I. S., & Manrai, A. K. (2025) · Nature Medicine · doi:10.1038/s41591-025-04050-6 · Cited by 7 · see citing works
Singh, K., Murali, A., Stevens, H. E., Vydiswaran, V. G. V., Bohnert, A. S. B., Brummett, C. M., & Fernandez, A. C. (2022) · Surgery · doi:10.1016/j.surg.2022.01.008 · Cited by 7 · see citing works
Self-citationClinical prediction rules for adverse evolution in patients with COVID-19 by the Omicron variant
Portuondo, J., Barrio, I., España, P. P., Garcia-Asensio, J., Villanueva, A., Gascón, M., Rodriguez-Idiazabal, L., Larrea, N., García‐Gutiérrez, S., Quintana, J. M., et al. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105039 · Cited by 8 · see citing works
Beyond Unplanned ICU Transfers: Linking a Revised Definition of Deterioration to Patient Outcomes
Byrd, T. F., Phelan, T., Ingraham, N. E., Langworthy, B., Bhasin, A., Kc, A., Melton, G. B., & Tignanelli, C. J. (2024) · Critical Care Medicine · doi:10.1097/ccm.0000000000006333 · Cited by 7 · see citing works
Portuondo, J., Bilbao‐González, A., Tíscar-González, V., Gutiérrez, I., García‐Gutiérrez, S., Martínez-Mejuto, A., Santiago-Garín, J., Arribas-García, S., García-Asensio, J., Chart-Pascual, J., et al. (2022) · Internal and Emergency Medicine · doi:10.1007/s11739-022-02931-z · Cited by 7 · see citing works
Augmenting existing deterioration indices with chest radiographs to predict clinical deterioration
Mu, E., Jabbour, S., Dalca, A. V., Guttag, J. V., Wiens, J., & Sjoding, M. W. (2022) · PLoS ONE · doi:10.1371/journal.pone.0263922 · Cited by 7 · see citing works
Self-citationArtificial Intelligence Systems in CKD: Where Do We Stand and What Will the Future Bring?
Padmanabhan, A. A., Balczewski, E. A., & Singh, K. (2022) · Advances in Chronic Kidney Disease · doi:10.1053/j.ackd.2022.06.004 · Cited by 6 · see citing works
Self-citationArtificial intelligence in obstetric anaesthesiology – the future of patient care?
Kong, A. Y. H., Liu, N., Tan, H. S., Sia, A. T. H., & Sng, B. L. (2024) · International Journal of Obstetric Anesthesia · doi:10.1016/j.ijoa.2024.104288 · Cited by 6 · see citing works
Major, V. J., Jones, S. A., Razavian, N., Bagheri, A., Mendoza, F., Stadelman, J., Horwitz, L. I., Austrian, J. S., & Aphinyanaphongs, Y. (2022) · Applied Clinical Informatics · doi:10.1055/s-0042-1750416 · Cited by 6 · see citing works
Settling the Score on Algorithmic Discrimination in Health Care
Ghassemi, M., Hightower, M., & Nsoesie, E. O. (2024) · NEJM AI · doi:10.1056/aip2400583 · Cited by 6 · see citing works
Logaras, E., Billis, A. S., Kyparissidis-Kokkinidis, I., Ketseridou, S. N., Fourlis, A., Tzotzis, A., Imprialos, K. P., Doumas, M. N., & Bamidis, P. D. (2022) · JMIR Formative Research · doi:10.2196/36933 · Cited by 5 · see citing works
Weigand, L., Viers, T., & Tipton, E. (2025) · Critical Care Nurse · doi:10.4037/ccn2025131 · Cited by 5 · see citing works
Deterioration Index in Critically Injured Patients: A Feasibility Analysis
Wu, R., Smith, A. A., Brown, T., Hunt, J. P., Greiffenstein, P. P., Taghavi, S., Tatum, D. M., Jackson‐Weaver, O., & Duchesne, J. C. (2022) · Journal of Surgical Research · doi:10.1016/j.jss.2022.08.019 · Cited by 5 · see citing works
Multisite Pragmatic Cluster-Randomized Controlled Trial of the CONCERN Early Warning SystemPreprint
Rossetti, S. C., Dykes, P. C., Knaplund, C., Cho, S., Withall, J. B., Lowenthal, G., Albers, D. J., Lee, R. Y., Jia, H., Henry, S. B., et al. (2024) · medRxiv · doi:10.1101/2024.06.04.24308436 · Cited by 5 · see citing works
A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools
Patel, H., Crusco, S., Hansen, D., Morales, J., Patale, T., Chiuzan, C., Sands, M., Naidich, J., Myers, K., Davidson, K. W., et al. (2026) · Journal of General Internal Medicine · doi:10.1007/s11606-026-10381-y · Cited by 4 · see citing works
van der Vegt, A. H., Campbell, V. K., Webb, R. T., Venkatesh, B., Lane, P. J., Wilks, K. M., McPhail, S., Rice, M., Isaacs, T., Abdel‐Hafez, A., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf192 · Cited by 4 · see citing works
Validated tool for early prediction of intensive care unit admission in COVID-19 patients
Huang, H., Liu, Y. H., Li, J., Dong, H., Gao, S., Huang, Z.-Y., Fu, S., Yang, L., Lu, H., Xia, L.-Y., et al. (2021) · World Journal of Clinical Cases · doi:10.12998/wjcc.v9.i28.8388 · Cited by 4 · see citing works
Trustworthiness of a machine learning early warning model in medical and surgical inpatients
Caraballo, P. J., Meehan, A. M., Fischer, K. M., Rahman, P., Simon, G., Melton, G. B., Salehinejad, H., & Borah, B. J. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae156 · Cited by 4 · see citing works
Dynamic modeling of hospitalized COVID-19 patients reveals disease state–dependent risk factors
Soper, B., Cadena, J., Nguyen, S., Chan, K. H. R., Kiszka, P., Womack, L., Work, M., Duggan, J. M., Haller, S. T., Hanrahan, J., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac012 · Cited by 4 · see citing works
Stanevich, O. V., Bakin, E. A., Korshunova, A. A., Gudkova, A. Y., Afanasev, A. A., Shlyk, I. V., Lioznov, D. A., Polushin, Y. S., & Куликов, А. Н. (2022) · Terapevticheskii arkhiv · doi:10.26442/00403660.2022.11.201941 · Cited by 4 · see citing works
Langer-Gould, A. M., Xu, S., Myers, L. C., Chen, A., Greene, J., Creekmur, B., Bruxvoort, K. J., Adams, J., Liu, V. X., & Gould, M. K. (2022) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2022.10.023 · Cited by 4 · see citing works
Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems
Kopanitsa, G. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001534 · Cited by 3 · see citing works
Prosepe, I., Groenwold, R. H. H., Knevel, R., Pajouheshnia, R., & van Geloven, N. (2022) · Frontiers in Epidemiology · doi:10.3389/fepid.2022.899589 · Cited by 3 · see citing works
Ecosystem of Patient-Centered Research and Information System Design
Hsueh, P.-Y. S. (2022) · Cognitive informatics in biomedicine and healthcare · doi:10.1007/978-3-031-07696-1_16 · Cited by 3 · see citing works
Lyons, P. G., Chen, V., Sekhar, T. C., McEvoy, C. A., Kollef, M. H., Govindan, R., Westervelt, P., Vranas, K. C., Maddox, T. M., Geng, E. H., et al. (2023) · JCO Clinical Cancer Informatics · doi:10.1200/cci.22.00104 · Cited by 3 · see citing works
Development and Validation of the COVID-19 Hospitalized Patient Deterioration Index
Nau, C., Butler, R. K., Huang, C., Khang, V. K., Chen, A., Creekmur, B., Broder, B. I., Subject, C. C., Sharp, A. L., Moreta-Sainz, L. M., et al. (2023) · The American Journal of Managed Care · doi:10.37765/ajmc.2023.89470 · Cited by 3 · see citing works
Ling-Hu, T., Simons, L. M., Dean, T. J., Rios-Guzman, E., Caputo, M., Alisoltani, A., Qi, C., Malczynski, M., Blanke, T., Jennings, L. J., et al. (2024) · Cell Reports Medicine · doi:10.1016/j.xcrm.2023.101361 · Cited by 2 · see citing works
How patients distinguish between clinical and administrative predictive models in health care
Nong, P. (2024) · The American Journal of Managed Care · doi:10.37765/ajmc.2024.89484 · Cited by 3 · see citing works
Ma, S. P., Rohatgi, N., & Chen, J. H. (2024) · Journal of Hospital Medicine · doi:10.1002/jhm.13404 · Cited by 3 · see citing works
A Bold First Toe into the Uncharted Waters of Evaluating Proprietary Clinical Prediction Models
Weissman, G. E. (2021) · Annals of the American Thoracic Society · doi:10.1513/annalsats.202103-332ed · Cited by 3 · see citing works
Continuous predicted risks should be retained when deploying clinical prediction models
Blythe, R. D., Parsons, R., Ong, M. E. H., & Barnett, A. G. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.112009 · Cited by 2 · see citing works
Gao, S., Zhang, Y., Deng, M., Liu, H., Xu, W., Luo, M., Tian, Y., & Zhang, B. (2026) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2025.1694001 · Cited by 3 · see citing works
Brooten, J. K., Speiser, J. L., Gabbard, J., MILLER, D. P., Mahler, S. A., Turner, A. S., Omlor, R. L., Mielke, M. M., & Cline, D. M. (2025) · Academic Emergency Medicine · doi:10.1111/acem.15096 · Cited by 2 · see citing works
Myers, L. C., Xu, S., Chen, A., Greene, J., Creekmur, B., Bruxvoort, K. J., Escobar, G. J., Adams, J., Langer-Gould, A. M., Liu, V. X., et al. (2022) · Journal of Hospital Medicine · doi:10.1002/jhm.13007 · Cited by 2 · see citing works
Predictive Utility of Epic’s Health Composite Score for Same-Day Cancellation in Cataract Surgery
Phan, C., Amiri, I., Levi, M., & Fleischman, D. (2026) · Clinical ophthalmology · doi:10.2147/opth.s601482 · Cited by 1 · see citing works
Langer-Gould, A. M., Xu, S., Myers, L. C., Chen, A., Greene, J., Creekmur, B., Bruxvoort, K. J., Adams, J., Liu, V. X., & Gould, M. K. (2022) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2022.11.017 · Cited by 1 · see citing works
Lee, S., Benson, B. E., Belle, A., Medlin, R. P., Jerkins, D., Goss, F. R., Khanna, A. K., DeVita, M. A., & Ward, K. R. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000465 · Cited by 1 · see citing works
Hoy, O. K., Walker, A. R., Sonday, C., Carmona, A., & Pellegrino, A. N. (2025) · Journal of Surgical Research · doi:10.1016/j.jss.2025.01.024 · Cited by 1 · see citing works
Hislop, M. (2023) · Research Square · doi:10.21203/rs.3.rs-3249460/v1 · Cited by 1 · see citing works
Logaras, E., Billis, A. S., Kyparissidis-Kokkinidis, I., Ketseridou, S. N., Fourlis, A., Imprialos, K. P., Tzotzas, A., Doumas, M. N., & Bamidis, P. D. (2022) · doi:10.2196/preprints.36933
Barton, H. J., Ötleş, E., Salwei, M. E., Montague, E. N. H., Benda, N. C., & Werner, N. E. (2025) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813251369399
- Authors
- Singh, K., Valley, T. S., Tang, S., Li, B. Y., Kamran, F., Sjoding, M. W., Wiens, J., Otles, E., Donnelly, J. P., Wei, M. Y., McBride, J. P., Cao, J., Penoza, C., Ayanian, J. Z., & Nallamothu, B. K.
- Affiliations
- University of Michigan (US)
- Department of Industrial and Operations Engineering, University of Michigan College of Engineering, Ann Arbor, Michigan; and
- Michigan Medicine (US)
- Corresponding author
- Singh, Karandeep
- ORCID
- Venue
- Annals of the American Thoracic Society
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-07-01
- Volume
- 18
- Issue
- 7
- Pages
- 1129-1137
- ISSN
- 2325-6621 (ISSN-L)
- 2329-6933
- PMC
- PMC8328366
- OpenAlex
- W3114626119
- Cited by
- 111
- References
- 16
- Citations by year
- 2026: 12 · 2025: 15 · 2024: 16 · 2023: 16 · 2022: 32 · 2021: 16 · 2020: 4
- FWCI
- 2.23
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Annals of the American Thoracic Society (publishedVersion, cc-by-nc-nd)
- medRxiv (acceptedVersion)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion)
- Funders
- National Institutes of Health
- Michigan Institute for Data Science, University of Michigan
- National Institute on Aging (K23AG056638)
- National Heart, Lung, and Blood Institute (K01HL136687, L30 HL133988, K23HL140165, K12HL138039)
- National Institute of General Medical Sciences (T32GM007863)
- U.S. National Library of Medicine (R01LM013325)
- Topics
- Health Sciences › Medicine › Infectious Diseases › COVID-19 Clinical Research Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
- Keywords
- COVID-19 pandemic, model evaluation, early warning systems, hospitalized patients, intensive care unit admission, mechanical ventilation, in-hospital mortality, racial and sex disparities, model discrimination, model bias, clinical decision support, health equity, Black patients, women, vulnerable populations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
