Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisions
Open accessAbstract
Demand has outstripped healthcare supply during the coronavirus disease 2019 (COVID-19) pandemic. Emergency departments (EDs) are tasked with distinguishing patients who require hospital resources from those who may be safely discharged to the community. The novelty and high variability of COVID-19 have made these determinations challenging. In this study, we developed, implemented and evaluated an electronic health record (EHR) embedded clinical decision support (CDS) system that leverages machine learning (ML) to estimate short-term risk for clinical deterioration in patients with or under investigation for COVID-19. The system translates model-generated risk for critical care needs within 24 h and inpatient care needs within 72 h into rapidly interpretable COVID-19 Deterioration Risk Levels made viewable within ED clinician workflow. ML models were derived in a retrospective cohort of 21,452 ED patients who visited one of five ED study sites and were prospectively validated in 15,670 ED visits that occurred before ( n = 4322) or after ( n = 11,348) CDS implementation; model performance and numerous patient-oriented outcomes including in-hospital mortality were measured across study periods. Incidence of critical care needs within 24 h and inpatient care needs within 72 h were 10.7% and 22.5%, respectively and were similar across study periods. ML model performance was excellent under all conditions, with AUC ranging from 0.85 to 0.91 for prediction of critical care needs and 0.80–0.90 for inpatient care needs. Total mortality was unchanged across study periods but was reduced among high-risk patients after CDS implementation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Hinson et al. (2022) developed a ML algorithm to predict near-term clinical deterioration in emergency patients with real-time EHR data.
“In response, Hinson et al. (2022) developed a ML algorithm to predict near-term clinical deterioration in emergency patients with real-time EHR data [25].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Fair Allocation of Scarce Medical Resources in the Time of Covid-19
Emanuel, E., Persad, G., Upshur, R. E. G., Thome, B. D. C., Parker, M., Glickman, A., Zhang, C., Boyle, C., Smith, M. J., & Phillips, J. P. (2020) · New England Journal of Medicine · doi:10.1056/nejmsb2005114 · Cited by 3,143 · see citing works
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Zheng, Z., Peng, F., Xu, B., Zhao, J., Liu, H., Peng, J., Li, Q., Jiang, C., Zhou, Y., Liu, S., et al. (2020) · Journal of Infection · doi:10.1016/j.jinf.2020.04.021 · Cited by 2,074 · see citing works
Ranney, M. L., Griffeth, V., & Jha, A. K. (2020) · New England Journal of Medicine · doi:10.1056/nejmp2006141 · Cited by 2,018 · see citing works
Kucirka, L. M., Lauer, S. A., Laeyendecker, O. B., Boon, D., & Lessler, J. (2020) · Annals of Internal Medicine · doi:10.7326/m20-1495 · Cited by 1,414 · see citing works
Dessie, Z. G., & Zewotir, T. T. (2021) · BMC Infectious Diseases · doi:10.1186/s12879-021-06536-3 · Cited by 875 · see citing works
A reason for visit classification for ambulatory care.
Schneider, D., Appleton, L., & McLemore, T. (1979) · PubMed · Cited by 790 · see citing works
Sun, X., & Xu, W. (2014) · IEEE Signal Processing Letters · doi:10.1109/lsp.2014.2337313 · Cited by 638 · see citing works
Artificial Intelligence in Health Care: A Report From the National Academy of Medicine
Matheny, M. E., Whicher, D., & Thadaney Israni, S. (2020) · JAMA · doi:10.1001/jama.2019.21579 · Cited by 628 · see citing works
Pijls, B. G. C. W., Jolani, S., Atherley, A., Derckx, R. T., Dijkstra, J. I. R., Franssen, G. H. L. M., Hendriks, S., Richters, A., Jellema, A., Zalpuri, S., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044640 · Cited by 580 · see citing works
Levin, S., Toerper, M., Hamrock, E., Hinson, J. S., Barnes, S., Gardner, H., Dugas, A., Linton, B., Kirsch, T., & Kelen, G. (2018) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2017.08.005 · Cited by 424 · see citing works
Self-citationChallenges and Opportunities of Big Data in Health Care: A Systematic Review
Kruse, C. S., Goswamy, R., Raval, Y., & Marawi, S. (2016) · JMIR Medical Informatics · doi:10.2196/medinform.5359 · Cited by 395 · see citing works
SARS-CoV-2 B.1.1.529 (Omicron) Variant — United States, December 1–8, 2021
CDC COVID-19 Response Team (2021) · MMWR Morbidity and Mortality Weekly Report · doi:10.15585/mmwr.mm7050e1 · Cited by 322 · see citing works
Haimovich, A. D., Ravindra, N. G., Stoytchev, S., Young, H. P., Wilson, F. P., van Dijk, D., Schulz, W. L., & Taylor, R. A. (2020) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2020.07.022 · Cited by 289 · see citing works
Predicting Disease Severity and Outcome in COVID-19 Patients: A Review of Multiple Biomarkers
Tjendra, Y., Mana, A. F. A., Freire, A. P. E., Akgun, Y., Millan, N. C., Gomez-Fernandez, C. R., & Cray, C. (2020) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2020-0471-sa · Cited by 274 · see citing works
Barfod, C., Lauritzen, M. M. P., Danker, J. K., Sölétormos, G., Forberg, J. L., Berlac, P. A., Lippert, F. K., Lundstrøm, L. H., Antonsen, K., & Lange, K. H. W. (2012) · Scandinavian Journal of Trauma Resuscitation and Emergency Medicine · doi:10.1186/1757-7241-20-28 · Cited by 194 · see citing works
Pre-existing traits associated with Covid-19 illness severity
Ebinger, J. E., Achamallah, N., Ji, H., Claggett, B. L., Sun, N., Botting, P. G., Nguyen, T., Luong, E., Kim, E. H., Park, E. Y., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0236240 · Cited by 171 · see citing works
Guan, X., Zhang, B., Fu, M., Li, M., Xu, Y., Zhu, Y., Peng, J., Guo, H., & Lu, Y. (2021) · Annals of Medicine · doi:10.1080/07853890.2020.1868564 · Cited by 134 · see citing works
An Electronic Emergency Triage System to Improve Patient Distribution by Critical Outcomes
Dugas, A. F., Kirsch, T. D., Toerper, M., Korley, F., Yenokyan, G., France, D., Hager, D., & Levin, S. (2016) · The Journal of Emergency Medicine · doi:10.1016/j.jemermed.2016.02.026 · Cited by 132 · see citing works
Self-citationSong, X., Yu, A. S. L., Kellum, J. A., Waitman, L. R., Matheny, M. E., Simpson, S. Q., Hu, Y., & Liu, M. (2020) · Nature Communications · doi:10.1038/s41467-020-19551-w · Cited by 120 · see citing works
Marcozzi, D. E., Carr, B. G., Liferidge, A. T., Baehr, N. E., & Browne, B. J. (2017) · International Journal of Health Services · doi:10.1177/0020731417734498 · Cited by 112 · see citing works
Bellou, V., Tzoulaki, I., van Smeden, M., Moons, K. G. M., Εvangelou, Ε., & Belbasis, L. (2021) · European Respiratory Journal · doi:10.1183/13993003.02964-2020 · Cited by 68 · see citing works
Martínez, D. A., Levin, S., Klein, E. Y., Parikh, C. R., Menez, S. P., Taylor, R. A., & Hinson, J. S. (2020) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2020.05.026 · Cited by 61 · see citing works
Self-citationBreiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017) · Routledge eBooks · doi:10.1201/9781315139470-1 · Cited by 61 · see citing works
Wongvibulsin, S., Garibaldi, B. T., Antar, A. A. R., Wen, J., Wang, M., Gupta, A., Bollinger, R. C., Xu, Y., Wang, K., Betz, J., et al. (2021) · Annals of Internal Medicine · doi:10.7326/m20-6754 · Cited by 53 · see citing works
Walston, J. M., Cabrera, D., Bellew, S. D., Olive, M., Lohse, C. M., & Bellolio, F. (2016) · Western Journal of Emergency Medicine · doi:10.5811/westjem.2016.2.28501 · Cited by 42 · see citing works
Considine, J., Jones, D. A., Pilcher, D. V., & Currey, J. (2016) · European Journal of Emergency Medicine · doi:10.1097/mej.0000000000000375 · Cited by 38 · see citing works
Payán‐Pernía, S., Pérez, L. G., Remacha, A. F., Gil, J. S., & Novelli, S. (2020) · Laboratory Medicine · doi:10.1093/labmed/lmaa105 · Cited by 36 · see citing works
Pinto, D., Lunet, N., & Azevedo, A. (2010) · PubMed · Cited by 31 · see citing works
Development and external validation of a prognostic tool for COVID-19 critical disease
Chow, D. S., Glavis‐Bloom, J., Soun, J. E., Weinberg, B. D., Loveless, T. B., Xie, X. S., Mutasa, S., Monuki, E. S., Park, J. I., Bota, D. A., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0242953 · Cited by 26 · see citing works
Bellou, V., Tzoulaki, I., Εvangelou, Ε., et al. (2020) · medRxiv · doi:10.1101/2020.05.13.20100495 · Cited by 12 · see citing works
Factors Associated with Worsening Oxygenation in Patients with Non-severe COVID-19 Pneumonia
Hahm, C. R., Lee, Y. K., Oh, D. H., Ahn, M. Y., Choi, J., Kang, N. R., Oh, J., Choi, H. Z., & Kim, S. (2021) · Tuberculosis & respiratory diseases · doi:10.4046/trd.2020.0139 · Cited by 9 · see citing works
The many faces of COVID-19: managing uncertainty
Sederer, L. I. (2021) · The Lancet Psychiatry · doi:10.1016/s2215-0366(21)00031-6 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [9]COVID PRECISE consortium. COVID PRECISE: Precise Risk Estimation to optimise COVID-19 Care for Infected or Suspected patients in diverse settings. COVID PRECISE consortium. https://www.covprecise.org/living-review/ (accessed 15 Oct 2021).
- [37]Breiman, L. Classification and Regression Trees 1st edn. (Routledge, 2017).
- [41]Lundberg, S. M. & Lee, S.-I. In Advances in Neural Information Processing Systems (eds. Guyon, I., et al.) (Curran Associates, Inc., 2017). https://proceedings.neurips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf.
Cited by
Sorted by citations
Chen, S., Yu, J., Chamouni, S., Wang, Y., & Li, Y. (2024) · BMC Medicine · doi:10.1186/s12916-024-03566-x · Cited by 73 · see citing works
“In response, Hinson et al. (2022) developed a ML algo-rithm to predict near-term clinical deterioration in emergency patients with real-time EHR data [25].”
BackgroundPreti, L. M., Ardito, V., Compagni, A., Petracca, F., & Cappellaro, G. (2024) · Journal of Medical Internet Research · doi:10.2196/55897 · Cited by 62 · see citing works
Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings
Magrabi, F., Lyell, D., & Coiera, E. W. (2023) · Yearbook of Medical Informatics · doi:10.1055/s-0043-1768733 · Cited by 25 · see citing works
Wang, M., Sushil, M., Miao, B. Y., & Butte, A. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad085 · Cited by 20 · see citing works
Artificial intelligence and infectious diseases: Scope and perspectives
Abbara, S., Crabol, Y., de Bouillé, J. G., Dinh, A., & Morquin, D. (2025) · Infectious Diseases Now · doi:10.1016/j.idnow.2025.105131 · Cited by 16 · see citing works
Advanced Artificial Intelligence Methods for Medical Applications
Siriborvornratanakul, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35748-0_24 · Cited by 9 · see citing works
Yip, R., Sun, Y. J., Bassuk, A. G., & Mahajan, V. B. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000849 · Cited by 8 · see citing works
Maynard, S., Farrington, J., Raza, S., & Stanworth, S. J. (2026) · Transfusion Medicine Reviews · doi:10.1016/j.tmrv.2026.150961 · Cited by 7 · see citing works
Liew, C., Ong, S., & Ng, D. C.-E. (2025) · Scientific Reports · doi:10.1038/s41598-024-80538-4 · Cited by 6 · see citing works
Lopes, J., Faria, M., & Santos, M. F. (2024) · PLoS ONE · doi:10.1371/journal.pone.0302697 · Cited by 5 · see citing works
Implementation of machine learning in emergency departments: A systematic review
Hosseini, B., Patel, A., Landes, M., Vaillancourt, S., Mamdani, M. M., Maruthananth, K., Matharu, N., Pathan, Z., Sivapragasam, K., Ruangsomboon, O., et al. (2026) · Digital Health · doi:10.1177/20552076251411209 · Cited by 4 · see citing works
Levin, S., Sarani, N., Hinson, J. S., Naiman, M., Cannon, C. M., Smith, A., Steinhart, B., Debraine, A., Kehoe, S. M., Immhoff, B., et al. (2025) · Critical Care Explorations · doi:10.1097/cce.0000000000001194 · Cited by 3 · see citing works
Self-citationCherian, J. P., Cosgrove, S. E., Haghpanah, F., & Klein, E. Y. (2023) · Infection Control and Hospital Epidemiology · doi:10.1017/ice.2023.76 · Cited by 2 · see citing works
Self-citationHinson, J. S., Zhao, X., Klein, E. Y., Badaki‐Makun, O., Rothman, R. E., Copenhaver, M. S., Smith, A., Fenstermacher, K., Toerper, M. F., Pekosz, A. S., et al. (2024) · Journal of the American College of Emergency Physicians Open · doi:10.1002/emp2.13117 · Cited by 2 · see citing works
Self-citationCloud-Based Predictive Analytics for Hospital Resource Optimization
Gupta, S., Gupta, M., Pahuja, P., Geeta, & Kumar, R. (2025) · International Conference on Computational Intelligence and Computing Applications (ICCICA) · doi:10.1109/iccica67008.2025.11337754 · Cited by 1 · see citing works
Freire, M. P., Chioro, A., Tureck, F., Tofani, L. F. N., Silva, L., Bigal, A. L., Vieira, A. D. C. S., & Louvison, M. C. P. (2025) · Cadernos de Saúde Pública · doi:10.1590/0102-311xen044824 · Cited by 1 · see citing works
Rhoten, B. A., Jones, A. C., Maxwell, C. A., & Stolldorf, D. P. (2023) · Journal for Healthcare Quality · doi:10.1097/jhq.0000000000000406
Freire, M. P., Chioro, A., Tureck, F., Tofani, L. F. N., Silva, L., Bigal, A. L., Vieira, A. D. C. S., & Louvison, M. C. P. (2025) · Cadernos de Saúde Pública · doi:10.1590/0102-311xpt044824
Arya, N., Krishnan, S. B., Mamodiya, U., & Chakrabarti, P. (2025) · International Conference on Sustainability, Innovation & Technology (ICSIT) · doi:10.1109/icsit65336.2025.11294645
Bogner, E., Thomas, A., Bajgain, B. B., Sauro, K. M., & Lee, J. (2025) · doi:10.2196/preprints.89834
Application of Foundation Models in Emergency and Critical Care: A Scoping ReviewPreprint
Kong, Y., Mai, X., Meyer, A., FANG, Y. C., Xu, Z., Song, Z., Gong, C., Wacker, D. A., Heneghan, J. A., Wang, Q., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-8338830/v1
The portability paradox of foundation models for clinical decision support
Rosen, K. L., Sui, M., Ong, A. Y., & Kvedar, J. C. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02615-4
Sneha, S., Awofisayo, R. M., Nino, V., Rischie, I., & Kumah, L. I. (2026) · JMIR AI · doi:10.2196/preprints.98274
Mallet, M. C., Redah, I., Sahnoun, Y. G., Nabi, H., Geneviève, L. D., Rousseau, E., Duvvuri, V. R. S. K., Berthelot, S., & Isabel, S. (2026) · Research Square · doi:10.21203/rs.3.rs-9521303/v1
Bogner, E., Thomas, A., Bajgain, B. B., van Rassel, C. R., Sauro, K. M., & Lee, J. (2026) · JMIR Medical Informatics · doi:10.2196/89834
Causality-Informed Machine Learning for ICU Length of Stay Prediction: A Healthcare Case Study
Chen, X., Xu, R., Liu, Y., Huang, W., Huang, M., Chen, W., & Xie, S. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-8447-5_12
Fischer, D., Meier, L., Braun, T., Koch, S., & Roth, F. (2022) · Journal of Health Informatics and Digital Systems · doi:10.68159/o013818674
Saleem, M., Bidoki, M. Z., Alsuraihi, W., Al-Garadi, M. A., & Ahmed, A. (2026) · Healthcare · doi:10.3390/healthcare14182922
- Authors
- Hinson, J. S., Klein, E., Smith, A., Toerper, M., Dungarani, T., Hager, D., Hill, P., Kelen, G., Niforatos, J. D., Stephens, R. S., Strauss, A. T., & Levin, S.
- Affiliations
- Johns Hopkins University (US)
- Center for Disease Dynamics, Economics & Policy (US)
- Howard County General Hospital (US)
- Corresponding author
- Hinson, Jeremiah S.
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2022-07-16
- Volume
- 5
- Issue
- 1
- Pages
- 94-94
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC9287691
- OpenAlex
- W4285588267
- Cited by
- 30
- References
- 36
- Citations by year
- 2026: 11 · 2025: 9 · 2024: 4 · 2023: 5 · 2022: 1
- FWCI
- 1.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- U.S. Department of Health & Human Services | Centers for Disease Control and Prevention (U01CK000589)
- U.S. Department of Health & Human Services | Agency for Healthcare Research and Quality (R18 HS026640-02)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- COVID-19 pandemic, clinical decision support, electronic health records, emergency department, hospital admission decisions, discharge decisions, in-hospital mortality, workflow integration, machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
