The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Prediction of In‐hospital Mortality in Emergency Department Patients With Sepsis: A Local Big Data–Driven, Machine Learning Approach
Open accessAbstract
Objectives: Predictive analytics in emergency care has mostly been limited to the use of clinical decision rules (CDR s) in the form of simple heuristics and scoring systems. In the development of CDR s, limitations in analytic methods and concerns with usability have generally constrained models to a preselected small set of variables judged to be clinically relevant and to rules that are easily calculated. Furthermore, CDR s frequently suffer from questions of generalizability, take years to develop, and lack the ability to be updated as new information becomes available. Newer analytic and machine learning techniques capable of harnessing the large number of variables that are already available through electronic health records (EHR s) may better predict patient outcomes and facilitate automation and deployment within clinical decision support systems. In this proof‐of‐concept study, a local, big data–driven, machine learning approach is compared to existing CDR s and traditional analytic methods using the prediction of sepsis in‐hospital mortality as the use case. Methods: This was a retrospective study of adult ED visits admitted to the hospital meeting criteria for sepsis from October 2013 to October 2014. Sepsis was defined as meeting criteria for systemic inflammatory response syndrome with an infectious admitting diagnosis in the ED. ED visits were randomly partitioned into an 80%/20% split for training and validation. A random forest model (machine learning approach) was constructed using over 500 clinical variables from data available within the EHR s of four hospitals to predict in‐hospital mortality. The machine learning prediction model was then compared to a classification and regression tree (CART) model, logistic regression model, and previously developed prediction tools on the validation data set using area under the receiver operating characteristic curve (AUC) and chi‐square statistics. Results: There were 5,278 visits among 4,676 unique patients who met criteria for sepsis. Of the 4,222 patients in the training group, 210 (5.0%) died during hospitalization, and of the 1,056 patients in the validation group, 50 (4.7%) died during hospitalization. The AUC s with 95% confidence intervals (CI s) for the different models were as follows: random forest model, 0.86 (95% CI = 0.82 to 0.90); CART model, 0.69 (95% CI = 0.62 to 0.77); logistic regression model, 0.76 (95% CI = 0.69 to 0.82); CURB‐65, 0.73 (95% CI = 0.67 to 0.80); MEDS, 0.71 (95% CI = 0.63 to 0.77); and mREMS, 0.72 (95% CI = 0.65 to 0.79). The random forest model AUC was statistically different from all other models (p ≤ 0.003 for all comparisons). Conclusions: In this proof‐of‐concept study, a local big data–driven, machine learning approach outperformed existing CDR s as well as traditional analytic techniques for predicting in‐hospital mortality of ED patients with sepsis. Future research should prospectively evaluate the effectiveness of this approach and whether it translates into improved clinical outcomes for high‐risk sepsis patients. The methods developed serve as an example of a new model for predictive analytics in emergency care that can be automated, applied to other clinical outcomes of interest, and deployed in EHR s to enable locally relevant clinical predictions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
APACHE II: A severity of disease classification system
KNAUS, W. A., DRAPER, E. A., WAGNER, D. P., & ZIMMERMAN, J. E. (1985) · Critical Care Medicine · doi:10.1097/00003246-198510000-00009 · Cited by 13,664
APACHE II-A Severity of Disease Classification System: Reply
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1986) · Critical Care Medicine · doi:10.1097/00003246-198608000-00028 · Cited by 13,376
Vincent, J. L., Moreno, R., Takala, J., Willatts, S., De Mendonça, A., Bruining, H., Reinhart, C. K., Suter, P. M., & Thijs, L. G. (1996) · Intensive Care Medicine · doi:10.1007/bf01709751 · Cited by 11,674
Early goal-directed therapy in the treatment of severe sepsis and septic shock
Rivers, E., Nguyen, B., Havstad, S., Ressler, J., Muzzin, A., Knoblich, B., Peterson, E., & Tomlanovich, M. (2001) · New England Journal of Medicine · doi:10.1056/nejmoa010307 · Cited by 10,743
Finding groups in data: an introduction to cluster analysis
(1991) · Choice Reviews Online · doi:10.5860/choice.28-4558 · Cited by 10,474
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Angus, D. C., Linde-Zwirble, W. T., Lidicker, J., Clermont, G., Carcillo, J., & Pinsky, M. R. (2001) · Critical Care Medicine · doi:10.1097/00003246-200107000-00002 · Cited by 8,503
APACHE II: a severity of disease classification system.
Knaus, W. A., Draper, E. S., Wagner, D. P., & Zimmerman, J. E. (1985) · PubMed · Cited by 6,034
An efficient k-means clustering algorithm: analysis and implementation
Kanungo, T., Mount, D. M., Netanyahu, N. S., Piatko, C. D., Silverman, R., & Wu, A. Y. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017616 · Cited by 5,660
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Bias in random forest variable importance measures: Illustrations, sources and a solution
Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-25 · Cited by 3,688
Finding Groups in Data: An Introduction to Cluster Analysis.
Sarle, W. S., Kaufman, L., & Rousseeuw, P. J. (1991) · Journal of the American Statistical Association · doi:10.2307/2290430 · Cited by 3,539
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Lim, W. S. (2003) · Thorax · doi:10.1136/thorax.58.5.377 · Cited by 3,070
Variable selection using random forests
Genuer, R., Poggi, J.-M., & Tuleau-Malot, C. (2010) · Pattern Recognition Letters · doi:10.1016/j.patrec.2010.03.014 · Cited by 2,619
The Inevitable Application of Big Data to Health Care
Murdoch, T. B., & Detsky, A. S. (2013) · JAMA · doi:10.1001/jama.2013.393 · Cited by 1,664
Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models
Zou, K. H., O’Malley, A. J., & Mauri, L. (2007) · Circulation · doi:10.1161/circulationaha.105.594929 · Cited by 1,473
Big Data In Health Care: Using Analytics To Identify And Manage High-Risk And High-Cost Patients
Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A., & Escobar, G. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0041 · Cited by 1,235
Risk prediction models: II. External validation, model updating, and impact assessment
Moons, K. G. M., Kengne, A. P., Grobbee, D. E., Royston, P., Vergouwe, Y., Altman, D. G., & Woodward, M. (2012) · Heart · doi:10.1136/heartjnl-2011-301247 · Cited by 1,195
Systemic Inflammatory Response Syndrome Criteria in Defining Severe Sepsis
Kaukonen, K.-M., Bailey, M., Pilcher, D., Cooper, D. J., & Bellomo, R. (2015) · New England Journal of Medicine · doi:10.1056/nejmoa1415236 · Cited by 1,166
Discretization: An Enabling Technique
Liu, H., Hussain, F., Tan, C. L., & Dash, M. (2002) · Data Mining and Knowledge Discovery · doi:10.1023/a:1016304305535 · Cited by 981
Krumholz, H. M. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0053 · Cited by 589
Validation, updating and impact of clinical prediction rules: A review
Toll, D. B., Janssen, K. J. M., Vergouwe, Y., & Moons, K. G. M. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.04.008 · Cited by 580
Methodologic Standards for the Development of Clinical Decision Rules in Emergency Medicine
STIELL, I., & WELLS, G. (1999) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(99)70309-4 · Cited by 571
From Mindless to Mindful Practice — Cognitive Bias and Clinical Decision Making
Croskerry, P. (2013) · New England Journal of Medicine · doi:10.1056/nejmp1303712 · Cited by 571
Shapiro, N. I., Wolfe, R. E., Moore, R. B., Smith, E., Burdick, E., & Bates, D. W. (2003) · Critical Care Medicine · doi:10.1097/01.ccm.0000054867.01688.d1 · Cited by 499
Siontis, G. C. M., Tzoulaki, I., Castaldi, P. J., & Ioannidis, J. P. A. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.007 · Cited by 459
The legal and ethical concerns that arise from using complex predictive analytics in health care
Cohen, I. G., Amarasingham, R., Shah, A., Xie, B., & Lo, B. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0048 · Cited by 300
(2012) · Cited by 253
A decision rule for identifying children at low risk for brain injuries after blunt head trauma
Palchak, M. J., Holmes, J. F., Vance, C. W., Gelber, R. E., Schauer, B. A., Harrison, M. J., Willis-Shore, J., Wootton-Gorges, S. L., Derlet, R. W., & Kuppermann, N. (2003) · Annals of Emergency Medicine · doi:10.1067/s0196-0644(03)00425-6 · Cited by 249
Adams, S. T., & Leveson, S. H. (2012) · BMJ · doi:10.1136/bmj.d8312 · Cited by 247
Chen, L., Zeng, W.-M., Cai, Y.-D., Feng, K.-Y., & Chou, K.-C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0035254 · Cited by 224
Implementing Electronic Health Care Predictive Analytics: Considerations And Challenges
Amarasingham, R., Patzer, R. E., Huesch, M., Nguyen, N. Q., & Xie, B. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0352 · Cited by 180
Laboratory Critical Values Policies and Procedures
Howanitz, P. J., Steindel, S. J., & Heard, N. V. (2002) · Archives of Pathology & Laboratory Medicine · doi:10.5858/2002-126-0663-lcvpap · Cited by 173
Implementation of Clinical Decision Rules in the Emergency Department
Stiell, I. G., & Bennett, C. L. (2007) · Academic Emergency Medicine · doi:10.1197/j.aem.2007.06.039 · Cited by 149
Exploiting missing clinical data in Bayesian network modeling for predicting medical problems
Lin, J.-H., & Haug, P. J. (2007) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2007.06.001 · Cited by 136
Finlay, G. D., Rothman, M. J., & Smith, R. A. (2013) · Journal of Hospital Medicine · doi:10.1002/jhm.2132 · Cited by 121
Performance of Severity of Illness Scoring Systems in Emergency Department Patients with Infection
Howell, M. D., Donnino, M. W., Talmor, D. S., Clardy, P. F., Ngo, L. H., & Shapiro, N. I. (2007) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2007.tb01866.x · Cited by 116
Exploring the Potential of Predictive Analytics and Big Data in Emergency Care
Janke, A. T., Overbeek, D. L., Kocher, K. E., & Levy, P. D. (2015) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2015.06.024 · Cited by 113
Ng, K., Ghoting, A., Steinhubl, S. R., Stewart, W. F., Malin, B., & Sun, J. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.12.012 · Cited by 111
Runyon, M. S., Richman, P. B., & Kline, J. A. (2006) · Academic Emergency Medicine · doi:10.1197/j.aem.2006.07.032 · Cited by 85
The Role of Physicians in the Era of Predictive Analytics
Sniderman, A. D., D’Agostino, R. B., & Pencina, M. J. (2015) · JAMA · doi:10.1001/jama.2015.6177 · Cited by 82
Improving classification performance with discretization on biomedical datasets.
Lustgarten, J. L., Gopalakrishnan, V., Grover, H., & Visweswaran, S. (2008) · PubMed Central · Cited by 82
Methodologic Standards for Interpreting Clinical Decision Rules in Emergency Medicine: 2014 Update
GREEN, S. M., Schriger, D. L., & Yealy, D. M. (2014) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2014.01.016 · Cited by 79
Olsson, T., Terént, A., & Lind, L. (2004) · Academic Emergency Medicine · doi:10.1197/j.aem.2004.05.027 · Cited by 78
Newgard, C. D., Lewis, R. J., & Jolly, B. T. (2002) · Annals of Emergency Medicine · doi:10.1067/mem.2002.123549 · Cited by 52
Identifying Confounding by Indication through Blinded Prospective Review
Johnston, S. C. (2001) · American Journal of Epidemiology · doi:10.1093/aje/154.3.276 · Cited by 37
Kulstad, C., Crowe, C., Kulstad, E. B., & Mistry, C. (2010) · Journal of Emergencies Trauma and Shock · doi:10.4103/0974-2700.70761 · Cited by 29
Predictive analytics on Electronic Health Records (EHRs) using Hadoop and Hive
Chennamsetty, H., Chalasani, S., & Riley, D. (2015) · International Conference on Electrical, Computer and Communication Technologies (ICECCT) · doi:10.1109/icecct.2015.7226129 · Cited by 24
Khourdaji, I. S., Parke, J., Chennamsetty, A., & Burks, F. N. (2015) · Advances in Urology · doi:10.1155/2015/476390 · Cited by 20
Random Forests Classification Analysis for the Assessment of Diagnostic Skill
Katz, J. D., Mamyrova, G., Guzhva, O., & Furmark, L. (2010) · American Journal of Medical Quality · doi:10.1177/1062860609354639 · Cited by 9
Liu, V. X., Soule, J., Escobar, G. J., Morehouse, J. W., Adams, C. L., Scruth, E. A., & Whippy, A. (2013) · Critical Care Medicine · doi:10.1097/01.ccm.0000440271.62632.a8 · Cited by 2
Cited by
Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119
Logistic regression was as good as machine learning for predicting major chronic diseases
Nusinovici, S., Tham, Y. C., Chak Yan, M. Y., Wei Ting, D. S., Li, J., Sabanayagam, C., Wong, T. Y., & Cheng, C.-Y. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.002 · Cited by 541
Emergency department triage prediction of clinical outcomes using machine learning models
Raita, Y., Goto, T., Faridi, M. K., Brown, D. F. M., Camargo, C. A., & Hasegawa, K. (2019) · Critical Care · doi:10.1186/s13054-019-2351-7 · Cited by 519
Remote patient monitoring using artificial intelligence: Current state, applications, and challenges
Shaik, T., Tao, X., Higgins, N., Li, L., Gururajan, R., Zhou, X., & Acharya, U. R. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1485 · Cited by 515
Data mining in clinical big data: the frequently used databases, steps, and methodological models
Wu, W.-T., Li, Y.-J., Feng, A.-Z., Li, L., Huang, T., Xu, A.-D., & Lyu, J. (2021) · Military Medical Research · doi:10.1186/s40779-021-00338-z · Cited by 505
Metadata
- Authors
- Taylor, R. A., Pare, J. R., Venkatesh, A. K., Mowafi, H., Melnick, E. R., Fleischman, W., & Hall, M. K.
- Venue
- Academic Emergency Medicine
- Publisher
- Wiley
- Published
- 2016-02-13
- Volume
- 23
- Issue
- 3
- Pages
- 269-278
- ISSN
- 1069-6563 (ISSN-L)
- 1553-2712
- OpenAlex
- W2200122354
- Cited by
- 491
- References
- 55
- FWCI
- 11.24
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.