Early identification of patients admitted to hospital for covid-19 at risk of clinical deterioration: model development and multisite external validation study
Open accessAbstract
OBJECTIVE: To create and validate a simple and transferable machine learning model from electronic health record data to accurately predict clinical deterioration in patients with covid-19 across institutions, through use of a novel paradigm for model development and code sharing. DESIGN: Retrospective cohort study. SETTING: One US hospital during 2015-21 was used for model training and internal validation. External validation was conducted on patients admitted to hospital with covid-19 at 12 other US medical centers during 2020-21. PARTICIPANTS: 33 119 adults (≥18 years) admitted to hospital with respiratory distress or covid-19. MAIN OUTCOME MEASURES: An ensemble of linear models was trained on the development cohort to predict a composite outcome of clinical deterioration within the first five days of hospital admission, defined as in-hospital mortality or any of three treatments indicating severe illness: mechanical ventilation, heated high flow nasal cannula, or intravenous vasopressors. The model was based on nine clinical and personal characteristic variables selected from 2686 variables available in the electronic health record. Internal and external validation performance was measured using the area under the receiver operating characteristic curve (AUROC) and the expected calibration error-the difference between predicted risk and actual risk. Potential bed day savings were estimated by calculating how many bed days hospitals could save per patient if low risk patients identified by the model were discharged early. RESULTS: 9291 covid-19 related hospital admissions at 13 medical centers were used for model validation, of which 1510 (16.3%) were related to the primary outcome. When the model was applied to the internal validation cohort, it achieved an AUROC of 0.80 (95% confidence interval 0.77 to 0.84) and an expected calibration error of 0.01 (95% confidence interval 0.00 to 0.02). Performance was consistent when validated in the 12 external medical centers (AUROC range 0.77-0.84), across subgroups of sex, age, race, and ethnicity (AUROC range 0.78-0.84), and across quarters (AUROC range 0.73-0.83). Using the model to triage low risk patients could potentially save up to 7.8 bed days per patient resulting from early discharge. CONCLUSION: A model to predict clinical deterioration was developed rapidly in response to the covid-19 pandemic at a single hospital, was applied externally without the sharing of data, and performed well across multiple medical centers, patient subgroups, and time periods, showing its potential as a tool for use in optimizing healthcare resources.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Models using readily available EHR data can accurately predict clinical deterioration and sepsis hours before they occur.
“Two prominent examples relevant to critical care are models using readily available EHR data that can accurately predict clinical deterioration and sepsis hours before they occur [28–33]; these models exemplify real-time CDSS that alert clinicians and prompt evaluation, testing, and interventions [33].”
From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [30]
The Discussion reports that a model using nine routinely collected EHR variables predicted clinical deterioration in patients hospitalized with COVID-19 and performed well across validation cohorts. The Methods describe predictions updated every four hours, but do not establish a specific number of hours of advance warning. The paper does not report a model predicting sepsis. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020) · Nature · doi:10.1038/s41586-020-2649-2 · Cited by 23,826 · see citing works
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works
Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013) · Wiley Series in Probability and Statistics · doi:10.1002/9781118548387 · Cited by 13,861 · see citing works
Data Structures for Statistical Computing in Python
McKinney, W. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-00a · Cited by 11,511 · see citing works
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · see citing works
NWChem: A comprehensive and scalable open-source solution for large scale molecular simulations
Valiev, M., Bylaska, E. J., Govind, N., Kowalski, K., Straatsma, T. P., van Dam, H. J. J., Wang, D., Nieplocha, J., Aprà, E., Windus, T. L., et al. (2010) · Computer Physics Communications · doi:10.1016/j.cpc.2010.04.018 · Cited by 5,794 · see citing works
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · see citing works
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579 · see citing works
Reback, J., McKinney, W., Bossche, J. V. D., Jbrockmendel, Augspurger, T., Cloud, P., Gfyoung, Sinhrks, Klein, A., Tratner, J., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3509134 · Cited by 4,025 · see citing works
Fair Allocation of Scarce Medical Resources in the Time of Covid-19
Emanuel, E., Persad, G., Upshur, R. E. G., Thome, B. D. C., Parker, M., Glickman, A., Zhang, C., Boyle, C., Smith, M. J., & Phillips, J. P. (2020) · New England Journal of Medicine · doi:10.1056/nejmsb2005114 · Cited by 3,143 · see citing works
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · see citing works
Privacy in the age of medical big data
Price, W. N., & Cohen, I. G. (2019) · Nature Medicine · doi:10.1038/s41591-018-0272-7 · Cited by 1,522 · see citing works
Collaborative, C. (2020) · British journal of surgery · doi:10.1002/bjs.11746 · Cited by 1,423 · see citing works
Prediction models need appropriate internal, internal-external, and external validation
Steyerberg, E. W., & Harrell, F. E. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.005 · Cited by 1,304 · see citing works
Do no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266 · see citing works
Self-citationRosenberg, E. S., Dufort, E. M., Udo, T., Wilberschied, L. A., Kumar, J., Tesoriero, J. M., Weinberg, P., Kirkwood, J., Muse, A., DeHovitz, J. A., et al. (2020) · JAMA · doi:10.1001/jama.2020.8630 · Cited by 1,162 · see citing works
Measuring Diagnoses: ICD Code Accuracy
O’Malley, K. J., Cook, K. F., Price, M. D., Wildes, K. R., Hurdle, J. F., & Ashton, C. M. (2005) · Health Services Research · doi:10.1111/j.1475-6773.2005.00444.x · Cited by 1,121 · see citing works
Knight, S. R., Ho, A. Y. W., Pius, R., Buchan, I. E., Carson, G., Drake, T. M., Dunning, J. W., Fairfield, C. J., Gamble, C. L., Green, C., et al. (2020) · BMJ · doi:10.1136/bmj.m3339 · Cited by 1,020 · see citing works
Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995 · see citing works
Self-citationObtaining well calibrated probabilities using Bayesian binning
Pakdaman Naeini, M., Cooper, G., & Hauskrecht, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9602 · Cited by 991 · see citing works
External validation of prognostic models: what, why, how, when and where?
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C., & van Diepen, M. (2021) · Clinical Kidney Journal · doi:10.1093/ckj/sfaa188 · Cited by 887 · see citing works
A Framework for Rationing Ventilators and Critical Care Beds During the COVID-19 Pandemic
White, D. B., & Lo, B. (2020) · JAMA · doi:10.1001/jama.2020.5046 · Cited by 826 · see citing works
Leakage in data mining: Formulation, detection, and avoidance
Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/2382577.2382579 · Cited by 791 · see citing works
Acute renal failure in the ICU: risk factors and outcome evaluated by the SOFA score
de Mendonça, A., Vincent, J., Suter, P. M., Moreno, R. P., Dearden, N. M., Antonelli, M. M., Takala, J. A., Sprung, C. L., & Cantraine, F. R. L. (2000) · Intensive Care Medicine · doi:10.1007/s001340051281 · Cited by 738 · see citing works
A tutorial on calibration measurements and calibration models for clinical prediction models
Huang, Y., Li, W., Macheret, F., Gabriel, R. A., & Ohno-Machado, L. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz228 · Cited by 486 · see citing works
The “inconvenient truth” about AI in healthcare
Panch, T., Mattie, H., & Celi, L. A. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0155-4 · Cited by 473 · see citing works
Siontis, G. C. M., Tzoulaki, I., Castaldi, P. J., & Ioannidis, J. P. A. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.007 · Cited by 462 · see citing works
Automated Identification of Adults at Risk for In-Hospital Clinical Deterioration
Escobar, G. J., Liu, V. X., Schuler, A., Lawson, B., Greene, J. D., & Kipnis, P. (2020) · New England Journal of Medicine · doi:10.1056/nejmsa2001090 · Cited by 314 · see citing works
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312 · see citing works
Carenzo, L., Costantini, E., Greco, M., Barra, F. L., Rendiniello, V., Mainetti, M., Bui, R., Zanella, A., Grasselli, G., Lagioia, M., et al. (2020) · Anaesthesia · doi:10.1111/anae.15072 · Cited by 311 · see citing works
Comparative study of techniques for large-scale feature selection
Ferri, F. J., Pudil, P., Hatef, M., & Kittler, J. (1994) · Machine intelligence and pattern recognition · doi:10.1016/b978-0-444-81892-8.50040-7 · Cited by 307 · see citing works
Making Machine Learning Models Clinically Useful
Shah, N. H., Milstein, A., & Bagley, P. (2019) · JAMA · doi:10.1001/jama.2019.10306 · Cited by 294 · see citing works
Dave, D., Friedson, A., Matsuzawa, K., & Sabia, J. J. (2020) · Economic Inquiry · doi:10.1111/ecin.12944 · Cited by 265 · see citing works
Vergano, M., Bertolini, G., Giannini, A., Gristina, G. R., Livigni, S., Mistraletti, G., Riccioni, L., & Petrini, F. (2020) · Critical Care · doi:10.1186/s13054-020-02891-w · Cited by 253 · see citing works
Predictive analytics in health care: how can we know it works?
Van Calster, B., Wynants, L., Timmerman, D., Steyerberg, E. W., & Collins, G. S. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz130 · Cited by 223 · see citing works
Gupta, R. K., Harrison, E. M., Ho, A. Y. W., Docherty, A. B., Knight, S. R., van Smeden, M., Abubakar, I., Lipman, M. C. I., Quartagno, M., Pius, R., et al. (2021) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(20)30559-2 · Cited by 217 · see citing works
Eriksson, C. O., Stoner, R. C., Eden, K., Newgard, C. D., & Guise, J. (2016) · Journal of General Internal Medicine · doi:10.1007/s11606-016-3936-3 · Cited by 209 · see citing works
Preferences for Home vs Hospital Care Among Low-Risk Patients With Community-Acquired Pneumonia
Coley, C. M. (1996) · Archives of Internal Medicine · doi:10.1001/archinte.156.14.1565 · Cited by 161 · see citing works
Xie, J., Hungerford, D., Chen, H., Abrams, S. T., Li, S., Wang, G., Wang, Y., Kang, H., Bonnett, L. J., Zheng, R., et al. (2020) · SSRN Electronic Journal · doi:10.1101/2020.03.28.20045997 · Cited by 157 · see citing works
Shamout, F. E., Shen, Y., Wu, N., Kaku, A., Park, J., Makino, T., Jastrzȩbski, S., Witowski, J. S., Wang, D., Zhang, B., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00453-0 · Cited by 153 · see citing works
Artificial intelligence cooperation to support the global response to COVID-19
Luengo-Oroz, M. A., Pham, K. H., Aylett-Bullock, J., Kirkpatrick, R., Luccioni, A. S., Rubel, S., Wachholz, C., Chakchouk, M., Biggs, P., Nguyen, T., et al. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0184-3 · Cited by 133 · see citing works
Wiens, J., Guttag, J., & Horvitz, E. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002162 · Cited by 123 · see citing works
Self-citationRandomized clinical trials of artificial intelligence
Angus, D. C. (2020) · JAMA · doi:10.1001/jama.2020.1039 · Cited by 112 · see citing works
Singh, K., Valley, T. S., Tang, S., Li, B. Y., Kamran, F., Sjoding, M. W., Wiens, J., Otles, E., Donnelly, J. P., Wei, M. Y., et al. (2021) · Annals of the American Thoracic Society · doi:10.1513/annalsats.202006-698oc · Cited by 111 · see citing works
Self-citationOutcomes and Mortality Among Adults Hospitalized With COVID-19 at US Medical Centers
Nguyen, N. T., Chinn, J. O., Nahmias, J. T., Yuen, S. W., Kirby, K. A., Hohmann, S., & Amin, A. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.0417 · Cited by 111 · see citing works
The epic sepsis model falls short-the importance of external validation
Habib, A. R., Lin, A. L., & Grant, R. W. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.3333 · Cited by 101 · see citing works
Equitably Allocating Resources during Crises: Racial Differences in Mortality Prediction Models
Ashana, D. C., Anesi, G. L., Liu, V. X., Escobar, G. J., Chesley, C., Eneanya, N. D., Weissman, G. E., Miller, W. D., Harhay, M. O., & Halpern, S. D. (2021) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.202012-4383oc · Cited by 98 · see citing works
Tang, S., Davarmanesh, P., Song, Y., Koutra, D., Sjoding, M. W., & Wiens, J. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa139 · Cited by 88 · see citing works
Self-citationMachine Learning, Predictive Analytics, and Clinical Practice: Can the Past Inform the Present?
Peterson, E. D. (2019) · JAMA · doi:10.1001/jama.2019.17831 · Cited by 79 · see citing works
Machine Learning for COVID-19 needs global collaboration and data-sharing
Peiffer-Smadja, N., Maatoug, R., Lescure, F.-X., D’Ortenzio, E., Pineau, J., & King, J.-R. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0181-6 · Cited by 70 · see citing works
Hospital at home for the management of COVID-19: preliminary experience with 63 patients
Pericàs, J. M., Cucchiari, D., Torrallardona‐Murphy, O., Calvo, J., Serralabós, J., Alvés, E., Agelet, A., Hidalgo, J., Alves, E., Castells, E., et al. (2020) · Infection · doi:10.1007/s15010-020-01527-z · Cited by 63 · see citing works
What is a hospital bed day worth? A contingent valuation study of hospital Chief Executive Officers
Page, K., Barnett, A. G., & Graves, N. (2017) · BMC Health Services Research · doi:10.1186/s12913-017-2079-5 · Cited by 63 · see citing works
Barish, M., Bolourani, S., Lau, L. F., Shah, S., & Zanos, T. P. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00254-2 · Cited by 58 · see citing works
A validated, real-time prediction model for favorable outcomes in hospitalized COVID-19 patients
Razavian, N., Major, V. J., Sudarshan, M., Burk‐Rafel, J., Stella, P., Randhawa, H., Bilaloglu, S., Chen, J., Nguy, V., Wang, W., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00343-x · Cited by 57 · see citing works
The impact of lockdown timing on COVID-19 transmission across US counties
Huang, X., Shao, X., Xing, L., Hu, Y., Sin, D. D., & Zhang, X. (2021) · EClinicalMedicine · doi:10.1016/j.eclinm.2021.101035 · Cited by 54 · see citing works
Calibration Drift Among Regression and Machine Learning Models for Hospital Mortality.
Davis, S. E., Lasko, T. A., Chen, G., & Matheny, M. E. (2017) · PubMed Central · Cited by 50 · see citing works
COVID-19 in the United States: Trajectories and second surge behavior
James, N., & Menzies, M. (2020) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/5.0024204 · Cited by 47 · see citing works
Cummings, B. C., Ansari, S., Motyka, J. R., Wang, G., Medlin Jr, R. P., Kronick, S. L., Singh, K., Park, P. K., Napolitano, L. M., Dickson, R. P., et al. (2021) · JMIR Medical Informatics · doi:10.2196/25066 · Cited by 35 · see citing works
Self-citationDevelopment and external validation of a prognostic tool for COVID-19 critical disease
Chow, D. S., Glavis‐Bloom, J., Soun, J. E., Weinberg, B. D., Loveless, T. B., Xie, X. S., Mutasa, S., Monuki, E. S., Park, J. I., Bota, D. A., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0242953 · Cited by 26 · see citing works
Carmichael, H. L., Coquet, J., Sun, R., Sang, S., Groat, D., Asch, S. M., Bledsoe, J. R., Peltan, I. D., Jacobs, J. R., & Hernandez‐Boussard, T. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103802 · Cited by 18 · see citing works
Wang, M., Zhang, J., Ye, D., Wang, Z., Liu, J., He, H., Shen, B., Luo, Z., Ye, J., Xu, Y. Q., et al. (2020) · Clinica Chimica Acta · doi:10.1016/j.cca.2020.06.051 · Cited by 11 · see citing works
Temporal changes in co-morbidities and mortality in patients hospitalized for COVID-19 in Italy
Angeli, F., & Bachetti, T. (2020) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2020.10.019 · Cited by 9 · see citing works
Sands, K., Wenzel, R. P., McLean, L. E., Korwek, K. M., Roach, J. D., Poland, R. E., Miller, K. M., Burgess, L. H., Jackson, E. S., & Perlin, J. B. (2020) · Infection Control and Hospital Epidemiology · doi:10.1017/ice.2020.1264 · Cited by 9 · see citing works
Kip, K. E., Snyder, G. M., Yealy, D. M., Mellors, J. W., Minnier, T. E., Donahoe, M. P., McKibben, J. C., Collins, K., & Marroquin, O. C. (2020) · medRxiv · doi:10.1101/2020.09.29.20203802 · Cited by 6 · see citing works
Flagged first, then by citations
2 of the 68 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- [35]Hooker G Mentch L. Please Stop Permuting Features: An Explanation and Alternatives. arXiv. 2019. https://arxiv.org/abs/1905.03151 · arXiv
- [44]Strickland E. AI May Help Hospitals Decide Which COVID-19 Patients Live or Die. IEEE Spectrum. 2020. https://spectrum.ieee.org/ai-can-help-hospitals-triage-covid19-patients (accessed 1 Jan 2022).
- [50]Pedregosa F Varoquaux G Gramfort A. Scikit-learn: Machine Learning in Python. J Mach Learn Res 2011; 12: 2825- 30.
- [62]Buolamwini J, Gebru T. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In: Friedler SA, Wilson C, eds. Proceedings of the 1st Conference on Fairness, Accountability and Transparency. New York, NY, USA: PMLR 2018. 77-91. https://proceedings.mlr.press/v81/buolamwini18a.html
Cited by
Sorted by citations
Transforming Cardiovascular Care With Artificial Intelligence: From Discovery to Practice
Khera, R., Oikonomou, E. K., Nadkarni, G. N., Morley, J., Wiens, J., Butte, A. J., & Topol, E. J. (2024) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2024.05.003 · Cited by 309 · see citing works
Self-citationCestonaro, C., Delicati, A., Marcante, B., Caenazzo, L., & Tozzo, P. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1305756 · Cited by 231 · see citing works
Use of artificial intelligence in critical care: opportunities and obstacles
Pinsky, M. R., Bedoya, A., Bihorac, A., Celi, L., Churpek, M., Economou-Zavlanos, N. J., Elbers, P., Saria, S., Liu, V., Lyons, P. G., et al. (2024) · Critical Care · doi:10.1186/s13054-024-04860-z · Cited by 160 · see citing works
Prognostic models in COVID-19 infection that predict severity: a systematic review
Buttia, C., Llanaj, E., Raeisi‐Dehkordi, H., Kastrati, L., Amiri, M., Meçani, R., Taneri, P. E., Gómez‐Ochoa, S. A., Raguindin, P. F., Wehrli, F., et al. (2023) · European Journal of Epidemiology · doi:10.1007/s10654-023-00973-x · Cited by 54 · see citing works
Pfaffenlehner, M., Behrens, M., Zöller, D., Ungethüm, K., Günther, K., Rücker, V., Reese, J., Heuschmann, P. U., Kesselmeier, M., Remo, F., et al. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02440-x · Cited by 41 · see citing works
Machine Learning Techniques Applied to COVID-19 Prediction: A Systematic Literature Review
Cheng, Y., Cheng, R. L., Xu, T., Tan, X., & Bai, Y. (2025) · Bioengineering · doi:10.3390/bioengineering12050514 · Cited by 21 · see citing works
Wichmann, R. M., Fernandes, F. T., Filho, A. D. P. C., IACOV-BR Network, Ciconelle, A. C. M., de Brito, A. M. E. S., Nunes, B. P., Silva, D. L. E., Anschau, F., Rodrigues, H. D. C., et al. (2023) · Scientific Reports · doi:10.1038/s41598-022-26467-6 · Cited by 19 · see citing works
The future of automated infection detection: Innovation to transform practice (Part III/III)
Branch‐Elliman, W., Sundermann, A. J., Wiens, J., & Shenoy, E. S. (2023) · Antimicrobial Stewardship & Healthcare Epidemiology · doi:10.1017/ash.2022.333 · Cited by 19 · see citing works
Self-citationGuerreiro, J., Garriga, R., Bagén, T. L., Sharma, B., Karnik, N. S., & Matic, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01203-8 · Cited by 15 · see citing works
Zheng, L., Lin, Y., Fang, K., Wu, J., & Zheng, M. (2023) · Hepatology Research · doi:10.1111/hepr.13907 · Cited by 13 · see citing works
Zysman, M., Asselineau, J., Saut, O., Frison, É., Oranger, M., Maurac, A., Charriot, J., Achkir, R., Regueme, S. C., Klein, É., et al. (2023) · European Radiology · doi:10.1007/s00330-023-09759-x · Cited by 13 · see citing works
Viderman, D., Kotov, A., Popov, M., & Abdildin, Y. G. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105308 · Cited by 13 · see citing works
Bittmann, F. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.923361 · Cited by 12 · see citing works
Matsumoto, K., Nohara, Y., Sakaguchi, M., Takayama, Y., Fukushige, S., Soejima, H., Nakashima, N., & Kamouchi, M. (2023) · JMIR Perioperative Medicine · doi:10.2196/50895 · Cited by 12 · see citing works
AI models in health care are not colour blind and we should not be either
Wiens, J., Creary, M., & Sjoding, M. W. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00092-9 · Cited by 11 · see citing works
Self-citationMasuda, S., Fukasawa, T., Inokuchi, S., Otsuki, B., Murata, K., Shimizu, T., Sono, T., Honda, S., Shima, K., Sakamoto, M., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-82359-x · Cited by 10 · see citing works
Dadabhoy, F. Z., Driver, L., McEvoy, D. S., Stevens, R., Rubins, D., & Dutta, S. (2023) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2022.11.012 · Cited by 10 · see citing works
Self-citationFu, Y., Zhong, W., Liu, T., Li, J., Xiao, K., Ma, X., Xie, L., Jiang, J., Zhou, H., Liu, R., et al. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.880999 · Cited by 10 · see citing works
Toward Realizing the Promise of AI in Precision Health Across the Spectrum of Care
Wiens, J., Spector‐Bagdady, K., & Mukherjee, B. (2024) · Annual Review of Genomics and Human Genetics · doi:10.1146/annurev-genom-010323-010230 · Cited by 7 · see citing works
Self-citationAugmenting existing deterioration indices with chest radiographs to predict clinical deterioration
Mu, E., Jabbour, S., Dalca, A. V., Guttag, J. V., Wiens, J., & Sjoding, M. W. (2022) · PLoS ONE · doi:10.1371/journal.pone.0263922 · Cited by 7 · see citing works
Self-citationAfnan, M., Afnan, M. A. M., Liu, Y., Savulescu, J., Mishra, A., Conitzer, V., & Rudin, C. D. (2022) · Reproductive BioMedicine Online · doi:10.1016/j.rbmo.2022.03.015 · Cited by 7 · see citing works
Performance of a Sepsis Prediction Model Across Different Sepsis Definitions
Dutta, S., McMurry, R., Tasi, M. C., Dunham, L. N., McEvoy, D. S., Stump, T., Filbin, M. R., & Rhee, C. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.5599 · Cited by 6 · see citing works
Self-citationChen, N., Li, S., Kuang, Z., Gong, T., Zhou, W., & Wang, Y. (2024) · Health care science · doi:10.1002/hcs2.81 · Cited by 6 · see citing works
Chang, T., Nuppnau, M., He, Y., Kocher, K. E., Valley, T. S., Sjoding, M. W., & Wiens, J. (2024) · PLOS Global Public Health · doi:10.1371/journal.pgph.0003555 · Cited by 6 · see citing works
Self-citationAchilonu, O., Obaido, G., Ogbuokiri, B., Aruleba, K., Musenge, E., & Fabian, J. (2024) · Frontiers in Digital Health · doi:10.3389/fdgth.2024.1427845 · Cited by 5 · see citing works
COVID-19: An evaluation of predictive scoring systems in South Africa
Prim, B. A., Kalla, I., Zamparini, J. M., & Mohamed, F. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e21733 · Cited by 5 · see citing works
Li, C., Liang, S., Wang, X., Lui, S., & Wang, C. (2025) · Precision Clinical Medicine · doi:10.1093/pcmedi/pbaf008 · Cited by 4 · see citing works
Prediction models based on machine learning algorithms for COVID-19 severity risk
Zhang, H., Wang, Y., Xie, Y., Wang, C., Ma, Y., & Jin, X. (2025) · BMC Public Health · doi:10.1186/s12889-025-22976-x · Cited by 4 · see citing works
Progress against health inequities is slow and obstacles are numerous
Loder, E. (2022) · BMJ · doi:10.1136/bmj.o470 · Cited by 4 · see citing works
Ovcharenko, E. A., Кутихин, А. Г., Gruzdeva, O. V., Kuzmina, A. A., Александровна, С. Т., Брусина, Е. Б., Владимировна, К. С., Евгеньевна, Б. Т., Kuzmenko, S., Osyaev, N. Y., et al. (2023) · Journal of Cardiovascular Development and Disease · doi:10.3390/jcdd10020039 · Cited by 4 · see citing works
National Multi-Institutional Validation of a Surgical Transfusion Risk Prediction Model
Lou, S. S., Liu, Y., Cohen, M. E., Ko, C. Y., Hall, B. L., & Kannampallil, T. (2023) · Journal of the American College of Surgeons · doi:10.1097/xcs.0000000000000874 · Cited by 4 · see citing works
Nonadverse COVID-19 evolution predictors: the CoNAE scale
Pulido, E., Larrea, N., García‐Gutiérrez, S., Gallardo, M. S., Rio, J. J. G.-D., Gascón, M., Legarreta, M. J., Villanueva, A., & Quintana, J. M. (2024) · Emergencias · doi:10.55633/s3me/e024.2023 · Cited by 3 · see citing works
Prognostic Modeling and Major Dataset Shifts During the COVID-19 Pandemic
Knaus, W. A. (2022) · JAMA Health Forum · doi:10.1001/jamahealthforum.2022.1103 · Cited by 3 · see citing works
Lyons, P. G., Chen, V., Sekhar, T. C., McEvoy, C. A., Kollef, M. H., Govindan, R., Westervelt, P., Vranas, K. C., Maddox, T. M., Geng, E. H., et al. (2023) · JCO Clinical Cancer Informatics · doi:10.1200/cci.22.00104 · Cited by 3 · see citing works
Predicting Severe Respiratory Failure in Patients with COVID-19: A Machine Learning Approach
Ceylan, B., Olmuşçelik, O., Karaalioğlu, B., Ceylan, Ş., Şahin, M., Aydın, S., Yılmaz, E., Dumlu, R., Kapmaz, M., Çiçek, Y., et al. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13237386 · Cited by 3 · see citing works
Swinnerton, K. N., Fillmore, N. R., Vo, A. D., La, J., Elbers, D. C., Brophy, M. T., Do, N., Monach, P. A., & Branch‐Elliman, W. (2025) · EClinicalMedicine · doi:10.1016/j.eclinm.2025.103114 · Cited by 3 · see citing works
Habib, A. R., & Lo, N. C. (2022) · BMJ · doi:10.1136/bmj.o354 · Cited by 3 · see citing works
Xu, Z., Yang, Q., Peng, D., Jiang, Y., Su, Y., Wu, B., Chen, Z., Huang, J., & Liu, X. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1759988 · Cited by 2 · see citing works
Self-citationZhao, Q., Jiang, W., Zhang, C., Ouyang, M., Wu, N., & Guo, Y. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1863050 · Cited by 2 · see citing works
Machine learning model evaluation on healthcare applications: A Review
de Souza, C. M. P., Barreto, C. A. D. S., de Macedo, L. V., de Brito, B. A. O., Targino, V. V., Betcel, E. C., de Almeida, F. G., Rodrigues, A. A. G., Malaquias, R. S., & Filho, I. D. M. B. (2023) · Seven Editora eBooks · doi:10.56238/interdiinovationscrese-013 · Cited by 2 · see citing works
Muto, R., Fukuta, S., Watanabe, T., Shindo, Y., Kanemitsu, Y., Kajikawa, S., Yonezawa, T., Inoue, T., Ichihashi, T., Shiratori, Y., et al. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.1042067 · Cited by 2 · see citing works
A systematic literature review on Machine Learning Model evaluation on healthcare applications
de Souza, C. M. P., Barreto, C. A. D. S., Macedo, L., de Brito, B. A. O., Targino, V. V., Betcel, E. C., de Almeida, F. G., Rodrigues, A. A. G., Malaquias, R. S., & Filho, I. D. M. B. (2023) · Research Society and Development · doi:10.33448/rsd-v12i6.42042 · Cited by 2 · see citing works
Kamran, F., Tjandra, D., Valley, T. S., Prescott, H. C., Shah, N. H., Liu, V. X., Horvitz, E., & Wiens, J. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf036 · Cited by 1 · see citing works
Self-citationLaake, J. H., Hagen, M., Aavitsland, P., Buanes, E. A., Fjone, K. S., Kvåle, R., Olsen, B. F., & Hofsø, K. (2025) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.70027 · Cited by 1 · see citing works
Chen, G., Lee, T., Tsang, J. L. Y., Binnie, A., McCarthy, A., Cowan, J., Archambault, P. M., Lellouche, F., Turgeon, A. F., Yoon, J., et al. (2025) · Critical Care Explorations · doi:10.1097/cce.0000000000001262 · Cited by 1 · see citing works
Predictive Modeling for Clostridioides difficile Infection
Rao, K. K., & Wiens, J. (2025) · Infectious Disease Clinics of North America · doi:10.1016/j.idc.2025.07.015 · Cited by 1 · see citing works
Self-citationByrd, T. F., Mattson, M., Polt, M., Pint, K., Dimitrov, K., Willis, A., Lister, J., Beacom, E., & Tignanelli, C. (2025) · Applied Clinical Informatics · doi:10.1055/a-2648-4817 · Cited by 1 · see citing works
Ceylan, B., Ceylan, Ş., Olmuşçelik, O., Karaalioğlu, B., Akan, M., Şahin, M., Muğlu, M., Aydın, S., Yılmaz, E., Dumlu, R., et al. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1724947 · Cited by 1 · see citing works
Kubiliute, I., Zaboras, E., Majauskaitė, F., Urbonienė, J., Zablockienė, B., Gefenaitė, G., Mickienė, A., & Jančorienė, L. (2026) · PLoS ONE · doi:10.1371/journal.pone.0350112 · Cited by 1 · see citing works
AI and Dynamic Prediction of Deterioration in Covid-19
Danker, C., Murzabekov, M., Forsberg, D., Lidströmer, N., Honoré, A., Rautiainen, S., & Herlenius, E. (2022) · Springer eBooks · doi:10.1007/978-3-031-08506-2_12 · Cited by 1 · see citing works
Predicting the prognosis of patients with mild COVID-19 by chest CT based on machine learning
Ji, B., Kong, L., Wang, J., Liu, C., Yuan, K., Zhu, L., & Liang, H. (2024) · Chinese Journal of Academic Radiology · doi:10.1007/s42058-024-00150-3 · Cited by 1 · see citing works
Tang, S., Man, Q., Zhu, D., Yu, X., Chen, R., Wang, S., Lu, Y., Shi, Q., Suo, C., & Xiong, L. (2024) · Frontiers in Microbiology · doi:10.3389/fmicb.2024.1361197 · Cited by 1 · see citing works
Self-citationMind the Gap: Wearable Lactate and Glucose Monitors for Hospitalized Patients
Guzzi, J., Falter, F., Kumar, A. B., & Perrino, A. C. (2025) · Cureus · doi:10.7759/cureus.78536 · Cited by 1 · see citing works
Wiegand, M., Cowan, S. L., Waddington, C. S., Halsall, D. J., Keevil, V. L., Tom, B. D. M., Taylor, V., Gkrania‐Klotsas, E., Preller, J., & Goudie, R. J. B. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-060026 · Cited by 1 · see citing works
Dong, G., Jin, F.-F., Huang, Q., Wu, C., Zhu, J., & Wang, T. (2022) · World Journal of Emergency Medicine · doi:10.5847/wjem.j.1920-8642.2022.103 · Cited by 1 · see citing works
Artificial intelligence in respiratory medicine
Gonem, S. (2023) · Elsevier eBooks · doi:10.1016/b978-0-443-15688-5.00027-9
Tazegül, G., Aydın, V., Tigen, E. T., Şengel, B. E., Köksal, K., Doğan, B., Karakurt, S., Altıkardeş, Z. A., Mülazımoğlu, L., Fak, A. S., et al. (2024) · Turkish Journal of Internal Medicine · doi:10.46310/tjim.1502238
Cross-Validation for Longitudinal Datasets with Unstable Correlations
Krishnamoorthy, M., Sjoding, M. W., & Wiens, J. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736898
Self-citationBarton, H. J., Ötleş, E., Salwei, M. E., Montague, E. N. H., Benda, N. C., & Werner, N. E. (2025) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813251369399
Jaafar, S. T. (2026) · Bilad Alrafidain Journal of Humanities and Social Science · doi:10.54720/bajhss/2026.080120
Thibaut, J., Geenen, C., Hosten, E., Libin, P., Dyck, K. V., & André, E. (2026) · Archives of Public Health · doi:10.1186/s13690-026-01958-4
Alaskar, B. S., AlQahtani, N. M., Alhussain, O. M., Almutairi, S. N., Alrashidi, N., & Alabdulwahab, A. O. (2026) · Medical Forum Monthly · doi:10.60110/medforum.370732
- Authors
- Kamran, F., Tang, S., Otles, E., McEvoy, D. S., Saleh, S. N., Gong, J., Li, B. Y., Dutta, S., Liu, X., Medford, R. J., Valley, T. S., West, L. R., Singh, K., Blumberg, S., Donnelly, J. P., Shenoy, E. S., Ayanian, J. Z., Nallamothu, B. K., Sjoding, M. W., & Wiens, J.
- Affiliations
- University of Michigan (US)
- Department of Industrial and Operations Engineering, University of Michigan College of Engineering, Ann Arbor, MI, USA
- Mass General Brigham (US)
- The University of Texas Southwestern Medical Center (US)
- University of California, San Francisco (US)
- Harvard University (US)
- Massachusetts General Hospital (US)
- Corresponding author
- Wiens, Jenna
- ORCID
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2022-02-17
- Volume
- 376
- Pages
- e068576
- ISSN
- 1756-1833 (ISSN-L)
- PMC
- PMC8850910
- OpenAlex
- W4212881260
- Cited by
- 60
- References
- 68
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 13 · 2023: 16 · 2022: 12
- FWCI
- 4.44
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- BMJ (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by-nc)
- Funders
- Agency for Healthcare Research and Quality
- National Science Foundation
- National Heart, Lung, and Blood Institute
- National Center for Emerging and Zoonotic Infectious Diseases
- U.S. National Library of Medicine
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Infectious Diseases › COVID-19 Clinical Research Studies
- Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
- Keywords
- COVID-19 pandemic, clinical deterioration prediction, risk prediction model, early discharge, electronic health record data, external validation, mechanical ventilation, vasopressors, high-flow nasal cannula, in-hospital mortality, triage, COVID-19 pneumonia, COVID-19 hospitalization
- MeSH terms
- Machine Learning, Clinical Deterioration, Clinical Decision Rules, COVID-19, SARS-CoV-2, Adolescent, Adult, Aged, Aged, 80 and over, Female, Hospitalization, Hospitals, Humans, Male, Middle Aged, Predictive Value of Tests, Prognosis, Retrospective Studies, ROC Curve, Linear Models, Risk Assessment, Area Under Curve, Young Adult, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
