Artificial intelligence algorithm to predict the need for critical care in prehospital emergency medical services
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Background: In emergency medical services (EMSs), accurately predicting the severity of a patient’s medical condition is important for the early identification of those who are vulnerable and at high-risk. In this study, we developed and validated an artificial intelligence (AI) algorithm based on deep learning to predict the need for critical care during EMS. Methods: We conducted a retrospective observation cohort study. The algorithm was established using development data from the Korean national emergency department information system, which were collected during visits in real time from 151 emergency departments (EDs). We validated the algorithm using EMS run sheets from two EDs. The study subjects comprised adult patients who visited EDs. The endpoint was critical care, and we used age, sex, chief complaint, symptom onset to arrival time, trauma, and initial vital signs as the predicted variables. Results: The number of patients in the development data was 8,981,181, and the validation data comprised 2604 EMS run sheets from two hospitals. The area under the receiver operating characteristic curve of the algorithm to predict the critical care was 0.867 (95% confidence interval, [0.864–0.871]). This result outperformed the Emergency Severity Index (0.839 [0.831–0.846]), Korean Triage and Acuity System (0.824 [0.815–0.832]), National Early Warning Score (0.741 [0.734–0.748]), and Modified Early Warning Score (0.696 [0.691–0.699]). Conclusions: The AI algorithm accurately predicted the need for the critical care of patients using information during EMS and outperformed the conventional triage tools and early warning scores.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
No free lunch theorems for optimization
Wolpert, D. H., & Macready, W. G. (1997) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.585893 · Cited by 14,237
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340 · cited 2×
“From the previous study, we found that conventional statistical methods such as logistic regression may have difficulty in determining the relationship among input variables [10, 28, 29].”
BackgroundPrevalence of Diabetic Retinopathy in the United States, 2005-2008
Zhang, X., Saaddine, J. B., Chou, C.-F., Cotch, M. F., Cheng, Y. J., Geiss, L. S., Gregg, E. W., Albright, A. L., Klein, B. E. K., & Klein, R. (2010) · JAMA · doi:10.1001/jama.2010.1111 · Cited by 1,196
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097
Smith, G. B., Prytherch, D. R., Meredith, P., Schmidt, P. E., & Featherstone, P. I. (2013) · Resuscitation · doi:10.1016/j.resuscitation.2012.12.016 · Cited by 1,077
Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis
Sun, G.-W., Shook, T. L., & Kay, G. L. (1996) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(96)00025-x · Cited by 933 · cited 1×
“From the previous study, we found that conventional statistical methods such as logistic regression may have difficulty in determining the relationship among input variables [10, 28, 29].”
Logistic regression in the medical literature:
Bagley, S. C., White, H., & Golomb, B. A. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00372-9 · Cited by 616 · cited 1×
“From the previous study, we found that conventional statistical methods such as logistic regression may have difficulty in determining the relationship among input variables [10, 28, 29].”
Research works printed but not linked: 26
- Moore L. Measuring quality and effectiveness of prehospital ems. Prehospital Emerg Care. 1999;3:325–31.. · DOI
- Baxt WG, Jones G, Fortlage D. The trauma triage rule: a new, resource-based approach to the prehospital identification of major trauma victims. Ann Emerg Med. 1990;19:1401–6. · DOI
- Lidal IB, Holte HH, Vist GE. Triage systems for pre-hospital emergency medical services - a systematic review. Scand J Trauma Resusc Emerg Med. 2013;21:28. · DOI
- Seymour CW. Prediction of critical illness during out-of-hospital emergency care. JAMA. 2010;304:747. · DOI · Paper page
- Kahn JM, Branas CC, Schwab CW, Asch DA. Regionalization of medical critical care: what can we learn from the trauma experience? Crit Care Med. 2008;36:3085–8. · DOI
- Evans SM, Murray A, Patrick I, Fitzgerald M, Smith S, Andrianopoulos N, et al. Assessing clinical handover between paramedics and the trauma team. Injury. 2010;41:460–4. · DOI
- Dojmi Di Delupis F, Pisanelli P, Di Luccio G, Kennedy M, Tellini S, Nenci N, et al. Communication during handover in the pre-hospital/hospital interface in Italy: from evaluation to implementation of multidisciplinary training through high-fidelity simulation. Intern Emerg Med. 2014;9:575–82. · DOI
- Kwon J, Jeon K-H, Kim HM, Kim MJ, Lim S, Kim K-H, et al. Deep-learning-based out-of-hospital cardiac arrest prognostic system to predict clinical outcomes. Resuscitation. 2019;139:84–91. · DOI
- Kwon J, Lee Y, Lee Y, Lee S, Park H, Park J. Validation of deep-learning-based triage and acuity score using a large national dataset. PLoS One. 2018;13(10):e0205836. https://doi.org/10.1371/journal.pone.0205836. eCollection 2018. · DOI
- Schalkoff RJ. Pattern recognition - statistical, structural and neural approaches; 1992.
- Ioffe S, Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; 2015.
- Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, et al. TensorFlow: A System for Large-Scale Machine Learning TensorFlow: A system for large-scale machine learning. In: 12th USENIX Symp Oper Syst Des Implement (OSDI ‘16); 2016. p. 265–84.
- Gilboy N, Tanabe P, Travers D. Rosenau AM. Emergency Severity Index (ESI): A Triage Tool for Emergency Department Care, Version 4: Implementation Handbook 2012 Edition; 2012.
- Mistry B, Stewart De Ramirez S, Kelen G, PSK S, Balhara KS, Levin S, et al. Accuracy and Reliability of Emergency Department Triage Using the Emergency Severity Index: An International Multicenter Assessment. Ann Emerg Med. 2018;71:581–7 e3. · DOI · Paper page
- Kwon H, Kim YJ, Jo YH, Lee JH, Lee JH, Kim J, et al. The Korean triage and acuity scale: associations with admission, disposition, mortality and length of stay in the emergency department. Int J Qual Heal Care. 2018. · DOI
- Lee B, Kim DK, Park JD, Kwak YH. Clinical considerations when applying vital signs in pediatric Korean triage and acuity scale. J Korean Med Sci. 2017;32:1702. · doi:10.3346/jkms.2017.32.10.1702
- Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified early warning score in medical admissions. QJM. 2001;94:521–6. · DOI · Paper page
- Williams TA, Tohira H, Finn J, Perkins GD, Ho KM. The ability of early warning scores (EWS) to detect critical illness in the prehospital setting: a systematic review. Resuscitation. 2016;102:35–43. · DOI
- Jouffroy R, Saade A, Ellouze S, Carpentier A, Michaloux M, Carli P, et al. Prehospital triage of septic patients at the SAMU regulation: comparison of qSOFA, MRST, MEWS and PRESEP scores. Am J Emerg Med. 2018;36:820–4. · DOI
- Carpenter J, Bithell J. Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat Med. 2000;19:1141–64. · doi:10.1002/(sici)1097-0258(20000515)19:9<1141::aid-sim479>3.0.co;2-f
- Shu E, Ives Tallman C, Frye W, Boyajian JG, Farshidpour L, Young M, et al. Pre-hospital qSOFA as a predictor of sepsis and mortality. Am J Emerg Med. 2019;37(7):1273-8. https://doi.org/10.1016/j.ajem.2018.09.025. Epub 2018 Sep 18. · DOI
- Buschhorn HM, Strout TD, Sholl JM, Baumann MR. Emergency medical services triage using the emergency severity index: is it reliable and valid? J Emerg Nurs. 2013;39:e55–63. · DOI
- Leeies M, Ffrench C, Strome T, Weldon E, Bullard M, Grierson R. Prehospital application of the Canadian triage and acuity scale by emergency medical services. CJEM. 2017;19:26–31. · doi:10.1017/cem.2016.345
- Christ M, Grossmann F, Winter D, Bingisser R, Platz E. Modern triage in the emergency department. Dtsch Arztebl Int. 2010;107:892–8.
- Checucci E, Autorino R, Cacciamani GE, Amparore D, De Cillis S, Piana A, et al. Artificial intelligence and neural networks in urology: current clinical applications. Minerva Urol Nefrol. 2019. https://doi.org/10.23736/S0393-2249.19.03613-0. · DOI · Paper page
- Chen X, Duan Y, Houthooft R, Schulman J, Sutskever I, Abbeel P. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets; 2016.
Cited by
A short guide for medical professionals in the era of artificial intelligence
Meskó, B., & Görög, M. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00333-z · Cited by 409
Maassen, O., Fritsch, S., Palm, J., Deffge, S., Kunze, J., Marx, G., Riedel, M., Schuppert, A., & Bickenbach, J. (2021) · Journal of Medical Internet Research · doi:10.2196/26646 · Cited by 125
Factors influencing clinicians' willingness to use an AI-based clinical decision support system
Choudhury, A. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.920662 · Cited by 64
“Another study proposed a deep learning model that could precisely predict patients’ needs in the critical care department (3).”
BackgroundThe aspects of running artificial intelligence in emergency care; a scoping review
(2023) · Archives of Academic Emergency Medicine · doi:10.22037/aaem.v11i1.1974 · Cited by 45
Lee, S., Park, H. J., Hwang, J., Lee, S. W., Han, K. S., Kim, W. Y., Jeong, J., Kang, H., Kim, A., Lee, C., et al. (2023) · Emergency Medicine International · doi:10.1155/2023/1221704 · Cited by 8
Self-citation
Metadata
- Authors
- Kang, D.-Y., Cho, K.-J., Kwon, O., Kwon, J.-M., Jeon, K.-H., Park, H., Lee, Y., Park, J., & Oh, B.-H.
- Venue
- Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine
- Published
- 2020-03-04
- Volume
- 28
- Issue
- 1
- Pages
- 17-17
- ISSN
- 1757-7241 (ISSN-L)
- OpenAlex
- W3011770729
- Cited by
- 129
- References
- 12
- FWCI
- 15.47
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Funders
- Ministry of Trade, industry & Energy (2000486,)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Artificial intelligence algorithm to predict the need for critical care in prehospital emergency medical services. https://app.cochat.ai/papers/10.1186/s13049-020-0713-4
- The paper
- Kang, D.-Y., Cho, K.-J., Kwon, O., Kwon, J.-M., Jeon, K.-H., Park, H., Lee, Y., Park, J., & Oh, B.-H. (2020). Artificial intelligence algorithm to predict the need for critical care in prehospital emergency medical services. Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine. https://doi.org/10.1186/s13049-020-0713-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.