Cited by 424Emergency and Acute Care StudiesMachine Learning in HealthcareArtificial Intelligence in Healthcare

The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.

Machine-Learning-Based Electronic Triage More Accurately Differentiates Patients With Respect to Clinical Outcomes Compared With the Emergency Severity Index

Levin, Scott, Toerper, Matthew, Hamrock, Eric, Hinson, Jeremiah S., Barnes, Sean, Gardner, Heather, Dugas, Andrea, Linton, Bob, Kirsch, Tom, Kelen, Gabor·2018·Annals of Emergency Medicine

DOI 10.1016/j.annemergmed.2017.08.005

Open this paper in CoChat — free account

Abstract

Study objective Standards for emergency department (ED) triage in the United States rely heavily on subjective assessment and are limited in their ability to risk-stratify patients. This study seeks to evaluate an electronic triage system (e-triage) based on machine learning that predicts likelihood of acute outcomes enabling improved patient differentiation.Methods A multisite, retrospective, cross-sectional study of 172,726 ED visits from urban and community EDs was conducted. E-triage is composed of a random forest model applied to triage data (vital signs, chief complaint, and active medical history) that predicts the need for critical care, an emergency procedure, and inpatient hospitalization in parallel and translates risk to triage level designations. Predicted outcomes and secondary outcomes of elevated troponin and lactate levels were evaluated and compared with the Emergency Severity Index (ESI).Results E-triage predictions had an area under the curve ranging from 0.73 to 0.92 and demonstrated equivalent or improved identification of clinical patient outcomes compared with ESI at both EDs. E-triage provided rationale for risk-based differentiation of the more than 65% of ED visits triaged to ESI level 3. Matching the ESI patient distribution for comparisons, e-triage identified more than 10% (14,326 patients) of ESI level 3 patients requiring up triage who had substantially increased risk of critical care or emergency procedure (1.7% ESI level 3 versus 6.2% up triaged) and hospitalization (18.9% versus 45.4%) across EDs.Conclusion E-triage more accurately classifies ESI level 3 patients and highlights opportunities to use predictive analytics to support triage decisionmaking. Further prospective validation is needed.

Integrity signals

Registry

Crossref · 10.1016/j.annemergmed.2017.08.005

Cites

52 of 52 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Random Forests
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Seventh Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure<SUBTITLE>The JNC 7 Report</SUBTITLE>
Chobanian, A. V. (2003) · JAMA · doi:10.1001/jama.289.19.2560 · Cited by 25,386
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
An Introduction to Statistical Learning
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) · Springer Texts in Statistics · doi:10.1007/978-1-4614-7138-7 · Cited by 12,106
Surviving Sepsis Campaign: International Guidelines for Management of Severe Sepsis and Septic Shock
Dellinger, R. P., Levy, M. M., Rhodes, A., Annane, D., Gerlach, H., Opal, S. M., Sevransky, J. E., Sprung, C. L., Douglas, I. S., Jaeschke, R., et al. (2013) · Critical Care Medicine · doi:10.1097/ccm.0b013e31827e83af · Cited by 9,694
Surviving Sepsis Campaign: International Guidelines for Management of Severe Sepsis and Septic Shock, 2012
Dellinger, R. P., Levy, M. M., Rhodes, A., Annane, D., Gerlach, H., Opal, S. M., Sevransky, J., Sprung, C. L., Douglas, I. S., Jaeschke, R., et al. (2013) · Intensive Care Medicine · doi:10.1007/s00134-012-2769-8 · Cited by 7,371
Third universal definition of myocardial infarction
the Writing Group on behalf of the Joint ESC/ACCF/AHA/WHF Task Force for the Universal Definition of Myocardial Infarction, Thygesen, K., Alpert, J. S., Jaffe, A. S., Simoons, M. L., CHAITMAN, B. R., & White, H. D. (2012) · Nature Reviews Cardiology · doi:10.1038/nrcardio.2012.122 · Cited by 6,585
Research guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951
Some Studies in Machine Learning Using the Game of Checkers
Samuel, A. L. (1959) · IBM Journal of Research and Development · doi:10.1147/rd.33.0210 · Cited by 4,443
Ensemble-based classifiers
Rokach, L. (2010) · Artificial Intelligence Review · doi:10.1007/s10462-009-9124-7 · Cited by 2,657
Third Universal Definition of Myocardial Infarction
Thygesen, K. A., Alpert, J. S., Jaffe, A. S., Simoons, M. L., CHAITMAN, B. R., White, H. D., Thygesen, K. A., Alpert, J. S., White, H. D., Jaffe, A. S., et al. (2012) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2012.08.001 · Cited by 2,619
The Effect of Emergency Department Crowding on Clinically Oriented Outcomes
Bernstein, S. L., Aronsky, D., Duseja, R., Epstein, S., Handel, D., Hwang, U., McCarthy, M., John McConnell, K., Pines, J. M., Rathlev, N., et al. (2009) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2008.00295.x · Cited by 1,147
Impact of delayed transfer of critically ill patients from the emergency department to the intensive care unit.
Chalfin, D. B., Trzeciak, S., Likourezos, A., Baumann, B. M., & Dellinger, R. P. (2007) · Critical Care Medicine · doi:10.1097/01.ccm.0000266585.74905.5a · Cited by 1,022
Early stroke treatment associated with better outcome
Marler, J. R., Tilley, B. C., Lü, M., Brott, T. G., Lyden, P., Grotta, J. C., Broderick, J. P., Levine, S. R., Frankel, M., Horowitz, S. H., et al. (2000) · Neurology · doi:10.1212/wnl.55.11.1649 · Cited by 856
National Hospital Ambulatory Medical Care Survey: 2019 Emergency Department Summary Tables
Cairns, C., & Kang, K. (2022) · doi:10.15620/cdc:115748 · Cited by 725
A roadmap for national action on clinical decision support
Osheroff, J. A., Teich, J. M., Middleton, B., Steen, E. B., Wright, A., & Detmer, D. E. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2334 · Cited by 707
Glossary of Terms
(1998) · Machine Learning · doi:10.1023/a:1017181826899 · Cited by 638
Reliability and Validity of a New Five‐level Triage Instrument
Wuerz, R. C., Milne, L. W., Eitel, D. R., Travers, D., & Gilboy, N. (2000) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2000.tb01066.x · Cited by 554
Emergency Department Crowding Is Associated With Poor Care for Patients With Severe Pain
Pines, J. M., & Hollander, J. E. (2007) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2007.07.008 · Cited by 515
Modern Triage in the Emergency Department
Christ, M., Grossmann, F. F., Winter, D., Bingisser, R. M., & Platz, E. (2010) · Deutsches Ärzteblatt international · doi:10.3238/arztebl.2010.0892 · Cited by 511
Emergency department crowding and thrombolysis delays in acute myocardial infarction
Schull, M. J., Vermeulen, M. J., Slaughter, G., Morrison, L. J., & Daly, P. (2004) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2004.05.004 · Cited by 402
Using methods from the data-mining and machine-learning literature for disease classification and prediction: a case study examining classification of heart failure subtypes
Austin, P. C., Tu, J. V., Ho, J. E., Levy, D. A., & Lee, D. S. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.11.008 · Cited by 340
Optimizing Emergency Department Front-End Operations
Wiler, J. L., Gentle, C., Halfpenny, J. M., Heins, A., Mehrotra, A., Mikhail, M. G., & Fite, D. (2010) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2009.05.021 · Cited by 339
The Emergency Severity Index Triage Algorithm Version 2 Is Reliable and Valid
Eitel, D. R., Travers, D. A., Rosenau, A. M., Gilboy, N., & Wuerz, R. C. (2003) · Academic Emergency Medicine · doi:10.1197/s1069-6563(03)00350-6 · Cited by 291
The Impact of Emergency Department Crowding Measures on Time to Antibiotics for Patients With Community-Acquired Pneumonia
Pines, J. M., Localio, A. R., Hollander, J. E., Baxt, W. G., Lee, H., Phillips, C., & Metlay, J. P. (2007) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2007.07.021 · Cited by 287
Effect of Emergency Department Crowding on Time to Antibiotics in Patients Admitted With Community-Acquired Pneumonia
Fee, C. R., Weber, E. J., Maak, C. A., & Bacchetti, P. (2007) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2007.08.003 · Cited by 257
A prospective, multicenter derivation of a biomarker panel to assess risk of organ dysfunction, shock, and death in emergency department patients with suspected sepsis
Shapiro, N. I., Trzeciak, S., Hollander, J. E., Birkhahn, R., Otero, R., Osborn, T. M., Moretti, E., Nguyen, H. B., Gunnerson, K. J., MILZMAN, D. P., et al. (2008) · Critical Care Medicine · doi:10.1097/ccm.0b013e318192fd9d · Cited by 243
The Association between Emergency Department Crowding and Hospital Performance on Antibiotic Timing for Pneumonia and Percutaneous Intervention for Myocardial Infarction
Pines, J. M., Hollander, J. E., Localio, A. R., & Metlay, J. P. (2006) · Academic Emergency Medicine · doi:10.1197/j.aem.2006.03.568 · Cited by 231
Abnormal vital signs are strong predictors for intensive care unit admission and in-hospital mortality in adults triaged in the emergency department - a prospective cohort study
Barfod, C., Lauritzen, M. M. P., Danker, J. K., Sölétormos, G., Forberg, J. L., Berlac, P. A., Lippert, F. K., Lundstrøm, L. H., Antonsen, K., & Lange, K. H. W. (2012) · Scandinavian Journal of Trauma Resuscitation and Emergency Medicine · doi:10.1186/1757-7241-20-28 · Cited by 194
Emergency Department Physician Activation of the Catheterization Laboratory and Immediate Transfer to an Immediately Available Catheterization Laboratory Reduce Door-to-Balloon Time in ST-Elevation Myocardial Infarction
Khot, U. N., Johnson, M. L., Ramsey, C., Khot, M. B., Todd, R., Shaikh, S. R., & Berg, W. J. (2007) · Circulation · doi:10.1161/circulationaha.106.677401 · Cited by 167
More Patients Are Triaged Using the Emergency Severity Index Than Any Other Triage Acuity System in the United States
McHugh, M., Tanabe, P. J., McClelland, M. S., & Khare, R. K. (2011) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2011.01240.x · Cited by 159
Impact of Delayed Transfer of Critically Ill Stroke Patients from the Emergency Department to the Neuro-ICU
Rincón, F., Mayer, S. A., Rivolta, J., Stillman, J. I., Boden-Albala, B. M., Elkind, M. S. V., Marshall, R. S., & Chong, J. Y. (2010) · Neurocritical Care · doi:10.1007/s12028-010-9347-0 · Cited by 140
An Electronic Emergency Triage System to Improve Patient Distribution by Critical Outcomes
Dugas, A. F., Kirsch, T. D., Toerper, M., Korley, F., Yenokyan, G., France, D., Hager, D., & Levin, S. (2016) · The Journal of Emergency Medicine · doi:10.1016/j.jemermed.2016.02.026 · Cited by 132
Self-citation
The impact of a delay in intensive care unit admission for community-acquired pneumonia
Phua, J., Ngerng, W. J., & Lim, T. K. (2010) · European Respiratory Journal · doi:10.1183/09031936.00154209 · Cited by 104
Overtesting and the Downstream Consequences of Overtreatment: Implications of “Preventing Overdiagnosis” for Emergency Medicine
Carpenter, C. R., Raja, A. S., & Brown, M. D. (2015) · Academic Emergency Medicine · doi:10.1111/acem.12820 · Cited by 99
Reliability of Computerized Emergency Triage
Dong, S. L., Bullard, M. J., Meurer, D. P., Blitz, S., Ohinmaa, A. E., Holroyd, B. R., & Rowe, B. H. (2006) · Academic Emergency Medicine · doi:10.1197/j.aem.2005.10.014 · Cited by 95
Mapping Out the Emergency Department Disposition Decision for High-Acuity Patients
Calder, L., Forster, A. J., Stiell, I. G., Carr, L. K., Perry, J. J., Vaillancourt, C., & Brehaut, J. (2012) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2012.04.013 · Cited by 88
Overcrowding is associated with delays in percutaneous coronary intervention for acute myocardial infarction
Kulstad, E. B., & Kelley, K. M. (2009) · International Journal of Emergency Medicine · doi:10.1007/s12245-009-0107-x · Cited by 81
Triage: an investigation of the process and potential vulnerabilities
Hitchcock, M., Gillespie, B. M., Crilly, J. L., & Chaboyer, W. P. (2013) · Journal of Advanced Nursing · doi:10.1111/jan.12304 · Cited by 79
Decreasing Length of Stay in the Emergency Department With a Split Emergency Severity Index 3 Patient Flow Model
Arya, R., Wei, G., McCoy, J. V., Crane, J., Ohman-Strickland, P., & Eisenstein, R. M. (2013) · Academic Emergency Medicine · doi:10.1111/acem.12249 · Cited by 78
Under-triage as a Significant Factor Affecting Transfer Time between the Emergency Department and the Intensive Care Unit
Yurkova, I., & Wolf, L. A. (2011) · Journal of Emergency Nursing · doi:10.1016/j.jen.2011.01.016 · Cited by 72
Undertriage in the Manchester triage system: an assessment of severity and options for improvement
Seiger, N., van Veen, M., Steyerberg, E. W., Ruige, M., van Meurs, A. H. J., & Moll, H. A. (2011) · Archives of Disease in Childhood · doi:10.1136/adc.2010.206797 · Cited by 60
Placing Physician Orders at Triage: The Effect on Length of Stay
Russ, S. E., Jones, I. R., Aronsky, D., Dittus, R. S., & Slovis, C. M. (2010) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2010.02.006 · Cited by 58
The Impact of ED Boarding Time, Severity of Illness, and Discharge Destination on Outcomes of Critically Ill ED Patients
Intas, G., Stergiannis, P. I., Chalari, E., Tsoumakas, K., & Fildissis, G. (2012) · Advanced Emergency Nursing Journal · doi:10.1097/tme.0b013e318251515f · Cited by 47
Principles of Anatomy and Physiology, 13th Edition
(2014) · Medicine & Science in Sports & Exercise · doi:10.1249/01.mss.0000446521.17381.6e · Cited by 39
Emergency Department Abnormal Vital Sign “Triggers” Program Improves Time to Therapy
McGillicuddy, D. C., O’Connell, F., Shapiro, N. I., Calder, S. A., Mottley, L., Roberts, J. C., & Sánchez, L. D. (2011) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2011.01056.x · Cited by 37
An integrated computerized triage system in the emergency department.
Aronsky, D., Jones, I. R., Raines, B., Hemphill, R. R., Mayberry, S. R., Luther, M. A., & Slusser, T. (2008) · PubMed Central · Cited by 31
An Interdepartmental Care Model to Expedite Admission from the Emergency Department to the Medical ICU
Elliott, D. J., Williams, K. D., Wu, P., Kher, H. V., Michalec, B., Reinbold, N., Coletti, C., Patel, B., & Dressler, R. M. (2015) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(15)41071-2 · Cited by 24
The effects of computerized triage on nurse work behavior.
Levin, S., France, D. J., Mayberry, R. S., Stonemetz, S., Jones, I. R., & Aronsky, D. (2006) · Europe PMC (PubMed Central) · Cited by 14
Self-citation
Factors Influencing Time-Dependent Quality Indicators for Patients With Suspected Acute Coronary Syndrome
France, D. J., Levin, S., Ding, R., Hemphill, R. R., Han, J. H., Russ, S. E., Aronsky, D., & Weinger, M. (2016) · Journal of Patient Safety · doi:10.1097/pts.0000000000000242 · Cited by 10
Self-citation

Research works printed but not linked: 1

Gilboy N, Tanabe T, Travers D, et al. Emergency Severity Index (ESI): A Triage Tool for Emergency Department Care, Version 4. Implementation Handbook 2012 Edition. Rockville, MD: Agency for Healthcare Research and Quality; 2012.

Sorted by citations

Artificial Intelligence in Pharmaceutical Technology and Drug Delivery Design
Vora, L. K., Gholap, A. D., Jetha, K. L., Singh, T. R. R., Solanki, H. K., & Chavda, V. P. (2023) · Pharmaceutics · doi:10.3390/pharmaceutics15071916 · Cited by 1,193
Wearables and the Medical Revolution
Dunn, J., Runge, R., & Snyder, M. (2018) · Personalized Medicine · doi:10.2217/pme-2018-0044 · Cited by 654
Emergency department triage prediction of clinical outcomes using machine learning models
Raita, Y., Goto, T., Faridi, M. K., Brown, D., Camargo, C. A., & Hasegawa, K. (2019) · Critical Care · doi:10.1186/s13054-019-2351-7 · Cited by 556
What is Machine Learning? A Primer for the Epidemiologist
Bi, Q., Goodman, K. E., Kaminsky, J., & Lessler, J. (2019) · American Journal of Epidemiology · doi:10.1093/aje/kwz189 · Cited by 579
Predicting hospital admission at emergency department triage using machine learning
Hong, W. S., Haimovich, A. D., & Taylor, R. A. (2018) · PLOS ONE · doi:10.1371/journal.pone.0201016 · Cited by 326

See all 424

Authors: Levin, S., Toerper, M., Hamrock, E., Hinson, J. S., Barnes, S., Gardner, H., Dugas, A., Linton, B., Kirsch, T., & Kelen, G.
Affiliations: Johns Hopkins University (US)
University of Maryland, College Park (US)
Uniformed Services University of the Health Sciences (US)
Corresponding author: Levin, Scott
ORCID: Levin, Scott
Toerper, Matthew
Hamrock, Eric
Hinson, Jeremiah S.
Barnes, Sean
Gardner, Heather
Dugas, Andrea
Kelen, Gabor
Venue: Annals of Emergency Medicine
Publisher: Elsevier BV
Published: 2018-05
Volume: 71
Issue: 5
Pages: 565-574.e2
ISSN: 0196-0644 (ISSN-L)
DOI: 10.1016/j.annemergmed.2017.08.005
OpenAlex: W2752349109
Cited by: 424
References: 52
Citations by year: 2026: 71 · 2025: 60 · 2024: 64 · 2023: 53 · 2022: 54 · 2021: 37 · 2020: 39 · 2019: 29 · 2018: 16
FWCI: 29.57
Type: journal article
Language: en
Open access: closed
Licence: www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations: Annals of Emergency Medicine (publishedVersion)
PubMed (publishedVersion)
Funders: Agency for Healthcare Research and Quality (R21HS023641)
Topics: Health Sciences › Medicine › Emergency Medicine › Emergency and Acute Care Studies
Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords: emergency department triage, Emergency Severity Index, risk stratification, random forest, machine learning prediction, hospitalization prediction, troponin, lactate, triage decision support
MeSH terms: Machine Learning, Adult, Algorithms, Cross-Sectional Studies, Emergency Service, Hospital, Female, Humans, Male, Retrospective Studies, Triage, United States, Area Under Curve, Vital Signs

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Machine-Learning-Based Electronic Triage More Accurately Differentiates Patients With Respect to Clinical Outcomes Compared With the Emergency Severity Index

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks