Deep-Learning Approaches to Identify Critically Ill Patients at Emergency Department Triage Using Limited Information
Open accessAbstract
Importance: Triage quickly identifies critically ill patients, helping to facilitate timely interventions. Many emergency departments use the emergency severity index (ESI) or abnormal vital sign thresholds to identify critically ill patients. However, both rely on fixed thresholds, and false activations detract from efficient care. Prior research suggests that machine-learning approaches may improve triage accuracy, but have relied on information that is often unavailable during the triage process. Objective: We examined whether deep-learning approaches could identify critically ill patients using data immediately available at triage with greater discriminative power than ESI or abnormal vital sign thresholds. Design: Retrospective, cross-sectional study. Setting: An urban tertiary care hospital in the Northeastern United States. Participants: Adult patients presenting to the emergency department from 1/1/2012 - 1/1/2020 were included. Deidentified triage information included structured data (age, sex, initial vital signs, ESI score, and clinical trigger activation due to abnormal vital signs), and textual data (chief complaint) with critical illness (defined as mortality or ICU admission within 24 hours) as the outcome. Interventions: Three progressively complex deep-learning models were trained (logistic regression on structured data, neural network on structured data, and neural network on structured and textual data), and applied to triage information from all patients. Main Outcomes and Measures: The primary outcome was the accuracy of the model for predicting whether patients were critically ill using area under the receiver-operator curve (AUC), as compared to ESI, utilizing a 10-fold cross-validation. Results: 445,925 patients were included, with 60,901 (13.7%) critically ill. Vital sign thresholds identified critically ill patients with AUC 0.521 (95% CI 0.519 -- 0.522), and ESI less than 3 demonstrated AUC 0.672 (95% CI 0.671 -- 0.674), logistic regression classified patients with AUC 0.803 (95% CI 0.802 -- 0.804), neural network with structured data with 0.811 (95% CI 0.807 - 0.815), and the neural network model with textual data with AUC 0.851 (95% CI 0.849 -- 0.852). Conclusions and Relevance: Deep-learning techniques represent a promising method of enhancing the triage process, even when working from limited information. Further research is needed to determine if improved predictions can be translated into meaningful clinical and operational benefits.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The most important variables for predicting admission risk include the number of comorbidities, arrival mode, and patient age.
“The most important variables include the number of comorbidities, arrival mode, and patient age which all latently encode a patient’s fraility; highly predictive of admission risk21.”
From Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during COVID-19 (2021) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · see citing works
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430 · see citing works
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2019) · Apollo - University of Cambridge Repository · doi:10.17863/cam.64134 · Cited by 10,460 · see citing works
NWChem: A comprehensive and scalable open-source solution for large scale molecular simulations
Valiev, M., Bylaska, E. J., Govind, N., Kowalski, K., Straatsma, T. P., van Dam, H. J. J., Wang, D., Nieplocha, J., Aprà, E., Windus, T. L., et al. (2010) · Computer Physics Communications · doi:10.1016/j.cpc.2010.04.018 · Cited by 5,794 · see citing works
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388 · see citing works
Collins, P. G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,312 · see citing works
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · see citing works
Cognitive biases associated with medical decisions: a systematic review
Saposnik, G., Redelmeier, D., Ruff, C. C., & Tobler, P. N. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0377-1 · Cited by 1,001 · see citing works
Sun, X., & Xu, W. (2014) · IEEE Signal Processing Letters · doi:10.1109/lsp.2014.2337313 · Cited by 638 · see citing works
Emergency department triage prediction of clinical outcomes using machine learning models
Raita, Y., Goto, T., Faridi, M. K., Brown, D., Camargo, C. A., & Hasegawa, K. (2019) · Critical Care · doi:10.1186/s13054-019-2351-7 · Cited by 531 · see citing works
Deep learning in clinical natural language processing: a methodical review
Wu, S., Roberts, K. E., Datta, S., Du, J., Ji, Z., Si, Y., Soni, S., Wang, Q., Wei, Q., Xiang, Y., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz200 · Cited by 495 · see citing works
Data Recombination for Neural Semantic Parsing
Jia, R., & Liang, P. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1002 · Cited by 441 · see citing works
Levin, S., Toerper, M., Hamrock, E., Hinson, J. S., Barnes, S., Gardner, H., Dugas, A., Linton, B., Kirsch, T., & Kelen, G. (2018) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2017.08.005 · Cited by 424 · see citing works
Predicting hospital admission at emergency department triage using machine learning
Hong, W. S., Haimovich, A. D., & Taylor, R. A. (2018) · PLOS ONE · doi:10.1371/journal.pone.0201016 · Cited by 326 · see citing works
Fernandes, M. P., Vieira, S. M., Leite, F., Palos, C., Finkelstein, S. N., & Sousa, J. M. C. (2019) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2019.101762 · Cited by 314 · see citing works
Horng, S., Sontag, D. A., Halpern, Y., Jernite, Y., Shapiro, N. I., & Nathanson, L. A. (2017) · PLOS ONE · doi:10.1371/journal.pone.0174708 · Cited by 310 · see citing works
Self-citationPlatts‐Mills, T. F., Travers, D. A., Biese, K., McCall, B., Kizer, S., LaMantia, M. A., Busby‐Whitehead, J., & Cairns, C. B. (2010) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2010.00670.x · Cited by 172 · see citing works
Hinson, J. S., Martínez, D. A., Schmitz, P. S. K., et al. (2018) · International Journal of Emergency Medicine · doi:10.1186/s12245-017-0161-8 · Cited by 171 · see citing works
Mistry, B., de Ramirez, S. S., Kelen, G. D., Schmitz, P. S. K., Balhara, K. S., Levin, S., Martínez, D. A., Psoter, K. J., Anton, X. P., & Hinson, J. S. (2017) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2017.09.036 · Cited by 160 · see citing works
Luten, R. C., Wears, R. L., Broselow, J., Croskerry, P., Joseph, M. M., & Frush, K. S. (2002) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2002.tb02175.x · Cited by 128 · see citing works
Differences in Vital Signs Between Elderly and Nonelderly Patients Prior to Ward Cardiac Arrest
Churpek, M. M., Yuen, T. C., Winslow, C. J., Hall, J. B., & Edelson, D. P. (2015) · Critical Care Medicine · doi:10.1097/ccm.0000000000000818 · Cited by 100 · see citing works
Redefining Hypotension in the Elderly
Oyetunji, T. A. (2011) · Archives of Surgery · doi:10.1001/archsurg.2011.154 · Cited by 88 · see citing works
Luten, R. C., Wears, R. L., Broselow, J., et al. (2002) · Academic Emergency Medicine · doi:10.1197/aemj.9.8.840 · Cited by 87 · see citing works
Behavioral Heuristics in Coronary-Artery Bypass Graft Surgery
Olenski, A. R., Zimerman, A., Coussens, S., & Jena, A. B. (2020) · New England Journal of Medicine · doi:10.1056/nejmc1911289 · Cited by 77 · see citing works
Emergency Department Abnormal Vital Sign “Triggers” Program Improves Time to Therapy
McGillicuddy, D. C., O’Connell, F., Shapiro, N. I., Calder, S. A., Mottley, L., Roberts, J. C., & Sánchez, L. D. (2011) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2011.01056.x · Cited by 37 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [11]Abadi M, Barham P, Chen J, et al. Tensorflow: A system for large‐scale machine learning In 12th USENIX symposium on operating systems design and implementation (OSDI 16). 2016, pp. 265‐ 283.
Cited by
Sorted by citations
Duckworth, C., Chmiel, F. P., Burns, D. K., Zlatev, Z. D., White, N. M., Daniels, T. W. V., Kiuber, M., & Boniface, M. J. (2021) · Scientific Reports · doi:10.1038/s41598-021-02481-y · Cited by 109 · see citing works
Akhlaghi, H., Freeman, S., Vari, C., McKenna, B., Braitberg, G., Karro, J., & Tahayori, B. (2024) · Emergency Medicine Australasia · doi:10.1111/1742-6723.14325 · Cited by 35 · see citing works
Chmiel, F. P., Burns, D. K., Azor, M., Borca, F., Boniface, M. J., Zlatev, Z. D., White, N. M., Daniels, T. W. V., & Kiuber, M. (2020) · doi:10.1101/2020.12.02.20239194 · Cited by 30 · see citing works
Sitthiprawiat, P., Wittayachamnankul, B., Sirikul, W., & Laohavisudhi, K. (2025) · Scientific Reports · doi:10.1038/s41598-025-17180-1 · Cited by 12 · see citing works
An ensemble model for predicting dispositions of emergency department patients
Kuo, K.-M., Lin, Y.-L., Chang, C. S., & Kuo, T. J. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02503-5 · Cited by 10 · see citing works
- Authors
- Joseph, J. W., Leventhal, E. L., Grossestreuer, A. V., Wong, M. L., Joseph, L. J., Nathanson, L. A., Donnino, M. W., Elhadad, N., & Sanchez, L. D.
- Affiliations
- Beth Israel Deaconess Medical Center (US)
- Columbia University (US)
- Corresponding author
- Joseph, Joshua W.
- ORCID
- Venue
- Journal of the American College of Emergency Physicians Open
- Publisher
- openRxiv
- Published
- 2020-05-06
- Volume
- 1
- Issue
- 5
- Pages
- 773-781
- ISSN
- 2688-1152 (ISSN-L)
- PMC
- PMC7593422
- OpenAlex
- W3082613412
- Cited by
- 57
- References
- 29
- Citations by year
- 2026: 8 · 2025: 12 · 2024: 14 · 2023: 5 · 2022: 9 · 2021: 9
- FWCI
- 2.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- Journal of the American College of Emergency Physicians Open (publishedVersion, cc-by-nc-nd)
- medRxiv (acceptedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute of General Medical Sciences (R01GM114355)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Emergency Medicine › Emergency and Acute Care Studies
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- emergency department triage, critical illness prediction, deep learning, Emergency Severity Index, mortality prediction, intensive care unit admission, artificial neural networks, gradient boosting decision tree, logistic regression, emergency department overcrowding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
