A validation of machine learning-based risk scores in the prehospital setting
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The triage of patients in prehospital care is a difficult task, and improved risk assessment tools are needed both at the dispatch center and on the ambulance to differentiate between low- and high-risk patients. This study validates a machine learning-based approach to generating risk scores based on hospital outcomes using routinely collected prehospital data. METHODS: Dispatch, ambulance, and hospital data were collected in one Swedish region from 2016-2017. Dispatch center and ambulance records were used to develop gradient boosting models predicting hospital admission, critical care (defined as admission to an intensive care unit or in-hospital mortality), and two-day mortality. Composite risk scores were generated based on the models and compared to National Early Warning Scores (NEWS) and actual dispatched priorities in a prospectively gathered dataset from 2018. RESULTS: A total of 38203 patients were included from 2016-2018. Concordance indexes (or areas under the receiver operating characteristics curve) for dispatched priorities ranged from 0.51-0.66, while those for NEWS ranged from 0.66-0.85. Concordance ranged from 0.70-0.79 for risk scores based only on dispatch data, and 0.79-0.89 for risk scores including ambulance data. Dispatch data-based risk scores consistently outperformed dispatched priorities in predicting hospital outcomes, while models including ambulance data also consistently outperformed NEWS. Model performance in the prospective test dataset was similar to that found using cross-validation, and calibration was comparable to that of NEWS. CONCLUSIONS: Machine learning-based risk scores outperformed a widely-used rule-based triage algorithm and human prioritization decisions in predicting hospital outcomes. Performance was robust in a prospectively gathered dataset, and scores demonstrated adequate calibration. Future research should explore the robustness of these methods when applied to other settings, establish appropriate outcome measures for use in determining the need for prehospital care, and investigate the clinical impact of interventions based on these methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
mice: Multivariate Imputation by Chained Equations in R
Buuren, S. V., & Groothuis-Oudshoorn, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i03 · Cited by 13,870 · cited 1×
“Missing vitals were multiply imputed five times using predictive mean matching over 20 iterations as implemented in the ‘mice’ R package [38].”
MethodBootstrap Methods and their Application
Davison, A. C., & Hinkley, D. V. (1997) · doi:10.1017/cbo9780511802843 · Cited by 7,350
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · cited 1×
“Perhaps most concerningly, ML methods readily integrate undesirable systemic biases present in the data they are trained on into the prediction model [31].”
MethodLevin, S., Toerper, M., Hamrock, E., Hinson, J. S., Barnes, S., Gardner, H., Dugas, A., Linton, B., Kirsch, T., & Kelen, G. (2018) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2017.08.005 · Cited by 424
Predicting hospital admission at emergency department triage using machine learning
Hong, W. S., Haimovich, A. D., & Taylor, R. A. (2018) · PLOS ONE · doi:10.1371/journal.pone.0201016 · Cited by 326 · cited 3×
“Such data have been found to provide substantial improvements 418 to patient outcome predictions [27,29].”
BackgroundAn Electronic Emergency Triage System to Improve Patient Distribution by Critical Outcomes
Dugas, A. F., Kirsch, T. D., Toerper, M., Korley, F., Yenokyan, G., France, D., Hager, D., & Levin, S. (2016) · The Journal of Emergency Medicine · doi:10.1016/j.jemermed.2016.02.026 · Cited by 132 · cited 3×
“We considered the performance of the models in the prospective dataset to be the best metric of future model performance, though results in this field have previously been reported based on cross-validation [26,32] or randomly selected hold-out samples [28].”
MethodResult
Cited by
Artificial intelligence in emergency medicine: A scoping review
Kirubarajan, A., Taher, A., Khan, S., & Masood, S. (2020) · JACEP Open · doi:10.1002/emp2.12277 · Cited by 127
The AI Future of Emergency Medicine
Petrella, R. J. (2024) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2024.01.031 · Cited by 43
Lee, S., Park, H. J., Hwang, J., Lee, S. W., Han, K. S., Kim, W. Y., Jeong, J., Kang, H., Kim, A., Lee, C., et al. (2023) · Emergency Medicine International · doi:10.1155/2023/1221704 · Cited by 8
Metadata
- Authors
- Spangler, D., Hermansson, T., Smekal, D., & Blomberg, H.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2019-12-13
- Volume
- 14
- Issue
- 12
- Pages
- e0226518
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2981937324
- Cited by
- 72
- References
- 8
- FWCI
- 8.79
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A validation of machine learning-based risk scores in the prehospital setting. https://app.cochat.ai/papers/10.1371/journal.pone.0226518
- The paper
- Spangler, D., Hermansson, T., Smekal, D., & Blomberg, H. (2019). A validation of machine learning-based risk scores in the prehospital setting. PLOS ONE. https://doi.org/10.1371/journal.pone.0226518
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.