Multicenter Development and Validation of a Risk Stratification Tool for Ward Patients
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
RATIONALE: Most ward risk scores were created using subjective opinion in individual hospitals and only use vital signs. OBJECTIVES: To develop and validate a risk score using commonly collected electronic health record data. METHODS: All patients hospitalized on the wards in five hospitals were included in this observational cohort study. Discrete-time survival analysis was used to predict the combined outcome of cardiac arrest (CA), intensive care unit (ICU) transfer, or death on the wards. Laboratory results, vital signs, and demographics were used as predictor variables. The model was developed in the first 60% of the data at each hospital and then validated in the remaining 40%. The final model was compared with the Modified Early Warning Score (MEWS) using the area under the receiver operating characteristic curve and the net reclassification index (NRI). MEASUREMENTS AND MAIN RESULTS: A total of 269,999 patient admissions were included, with 424 CAs, 13,188 ICU transfers, and 2,840 deaths occurring during the study period. The derived model was more accurate than the MEWS in the validation dataset for all outcomes (area under the receiver operating characteristic curve, 0.83 vs. 0.71 for CA; 0.75 vs. 0.68 for ICU transfer; 0.93 vs. 0.88 for death; and 0.77 vs. 0.70 for the combined outcome; P value < 0.01 for all comparisons). This accuracy improvement was seen across all hospitals. The NRI for the electronic Cardiac Arrest Risk Triage compared with the MEWS was 0.28 (0.18-0.38), with a positive NRI of 0.19 (0.09-0.29) and a negative NRI of 0.09 (0.09-0.09). CONCLUSIONS: We developed an accurate ward risk stratification tool using commonly collected electronic health record variables in a large multicenter dataset. Further study is needed to determine whether implementation in real-time would improve patient outcomes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
APACHE II: A severity of disease classification system
KNAUS, W. A., DRAPER, E. A., WAGNER, D. P., & ZIMMERMAN, J. E. (1985) · Critical Care Medicine · doi:10.1097/00003246-198510000-00009 · Cited by 13,664
APACHE II-A Severity of Disease Classification System: Reply
Knaus, W. A., Draper, E. A., Wagner, D. P., & Zimmerman, J. E. (1986) · Critical Care Medicine · doi:10.1097/00003246-198608000-00028 · Cited by 13,376
Early goal-directed therapy in the treatment of severe sepsis and septic shock
Rivers, E., Nguyen, B., Havstad, S., Ressler, J., Muzzin, A., Knoblich, B., Peterson, E., & Tomlanovich, M. (2001) · New England Journal of Medicine · doi:10.1056/nejmoa010307 · Cited by 10,743
Kuhn, M., & Johnson, K. (2013) · doi:10.1007/978-1-4614-6849-3 · Cited by 6,788 · cited 1×
“In addition, because cardiac arrest comprises a small proportion of all patients in the combined outcome, the importance of these patients in model estimation was then increased by using frequency weights by factors of 10, 25, 50, 75, and 100 to determine if this further improved model accuracy for cardiac arrest with…”
MethodPencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
APACHE II: a severity of disease classification system.
Knaus, W. A., Draper, E. S., Wagner, D. P., & Zimmerman, J. E. (1985) · PubMed · Cited by 6,034
The APACHE III Prognostic System
Knaus, W. A., Wagner, D. P., Draper, E. A., Zimmerman, J. E., Bergner, M., Bastos, P. G., Sirio, C. A., Murphy, D. J., Lotring, T., Damiano, A., et al. (1991) · Chest · doi:10.1378/chest.100.6.1619 · Cited by 3,439
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
The APACHE III Prognostic System
Teres, D., & Lemeshow, S. (1992) · Chest · doi:10.1378/chest.102.6.1919b · Cited by 2,880
Greenland, P. (2008) · Statistics in Medicine · doi:10.1002/sim.2976 · Cited by 1,300
Early thrombolytic treatment in acute myocardial infarction: reappraisal of the golden hour
Boersma, E., Maas, A. C., Deckers, J. W., & Simoons, M. L. (1996) · The Lancet · doi:10.1016/s0140-6736(96)02514-7 · Cited by 1,299
Plant, P., Owen, J., & Elliott, M. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02323-0 · Cited by 1,242
Logistic regression, survival analysis, and the Kaplan-Meier curve.
Efron, B. (1988) · Journal of the American Statistical Association · doi:10.1080/01621459.1988.10478612 · Cited by 1,065
Clinical Antecedents to In-Hospital Cardiopulmonary Arrest
Schein, R. M. H., Hazday, N., Pena, M., Ruben, B. H., & Sprung, C. L. (1990) · Chest · doi:10.1378/chest.98.6.1388 · Cited by 905
Trends in survival after in-hospital cardiac arrest
Girotra, S., Nallamothu, B. K., Spertus, J. A., Li, Y., Krumholz, H. M., & Chan, P. S. (2012) · New England Journal of Medicine · doi:10.1056/nejmoa1109148 · Cited by 895
It's about Time: Using Discrete-Time Survival Analysis to Study Duration and the Timing of Events
Singer, J. D., & Willett, J. B. (1993) · Journal of Educational Statistics · doi:10.2307/1165085 · Cited by 878
Royston, P., & Sauerbrei, W. (2008) · Wiley Series in Probability and Statistics · doi:10.1002/9780470770771 · Cited by 712
Royston, P. J., & Sauerbrei, W. (2008) · Cited by 601
Leening, M. J. G., Vedder, M. M., Witteman, J. C. M., Pencina, M. J., & Steyerberg, E. W. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1522 · Cited by 594
ViEWS—Towards a national early warning score for detecting adult inpatient deterioration
Prytherch, D. R., Smith, G. B., Schmidt, P. E., & Featherstone, P. I. (2010) · Resuscitation · doi:10.1016/j.resuscitation.2010.04.014 · Cited by 535
Gao, H., McDonnell, A., Harrison, D. A., Moore, T., Adam, S., Daly, K., Esmonde, L., Goldhill, D. R., Parry, G. J., Rashidian, A., et al. (2007) · Intensive Care Medicine · doi:10.1007/s00134-007-0532-3 · Cited by 466
Boogaard, M. V. D., Pickkers, P., Slooter, A. J. C., Kuiper, M. A., Spronk, P. E., Voort, P. H. J. V. D., Hoeven, J. G. V. D., Donders, R., Achterberg, T. V., & Schoonhoven, L. (2012) · BMJ · doi:10.1136/bmj.e420 · Cited by 447
Subbe, C. P., Davies, R. G., Williams, E., Rutherford, P., & Gemmell, L. (2003) · Anaesthesia · doi:10.1046/j.1365-2044.2003.03258.x · Cited by 446
Incidence of treated cardiac arrest in hospitalized patients in the United States*
Merchant, R. M., Yang, L., Becker, L. B., Berg, R. A., Nadkarni, V., Nichol, G., Carr, B. G., Mitra, N., Bradley, S. M., Abella, B. S., et al. (2011) · Critical Care Medicine · doi:10.1097/ccm.0b013e3182257459 · Cited by 431
Hodgetts, T. J., Kenward, G., Vlackonikolis, I., Payne, S., Castle, N., Crouch, R., Ineson, N., & Shaikh, L. (2002) · Resuscitation · doi:10.1016/s0300-9572(02)00098-9 · Cited by 417
Review and performance evaluation of aggregate weighted ‘track and trigger’ systems
Smith, G. B., Prytherch, D. R., Schmidt, P. E., & Featherstone, P. I. (2008) · Resuscitation · doi:10.1016/j.resuscitation.2007.12.004 · Cited by 337
Young, M. P., Gooder, V. J., McBride, K., James, B., & Fisher, E. S. (2003) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2003.20441.x · Cited by 223
Derivation of a cardiac arrest prediction model using ward vital signs*
Churpek, M. M., Yuen, T. C., Park, S. Y., Meltzer, D. O., Hall, J. B., & Edelson, D. P. (2012) · Critical Care Medicine · doi:10.1097/ccm.0b013e318250aa5a · Cited by 199
Self-citationEscobar, G. J., LaGuardia, J. C., Turk, B. J., Ragins, A., Kipnis, P., & Draper, D. (2012) · Journal of Hospital Medicine · doi:10.1002/jhm.1929 · Cited by 185
Predicting Cardiac Arrest on the Wards
Churpek, M. M., Yuen, T. C., Huber, M. T., Park, S. Y., Hall, J. B., & Edelson, D. P. (2012) · Chest · doi:10.1378/chest.11-1301 · Cited by 185
Self-citationA review, and performance evaluation, of single-parameter “track and trigger” systems
Smith, G. B., Prytherch, D. R., Schmidt, P. E., Featherstone, P. I., & Higgins, B. (2008) · Resuscitation · doi:10.1016/j.resuscitation.2008.05.004 · Cited by 184
Longitudinal analysis of one million vital signs in patients in an academic medical center
Bleyer, A. J., Vidya, S., Russell, G. B., Jones, C. M., Sujata, L., Daeihagh, P., & Hire, D. (2011) · Resuscitation · doi:10.1016/j.resuscitation.2011.06.033 · Cited by 178
Risk Stratification of Hospitalized Patients on the Wards
Churpek, M. M., Yuen, T. C., & Edelson, D. P. (2013) · CHEST Journal · doi:10.1378/chest.12-1605 · Cited by 166
Self-citationCuthbertson, B. H., Boroujerdi, M., McKie, L., Aucott, L., & Prescott, G. (2007) · Critical Care Medicine · doi:10.1097/01.ccm.0000254826.10520.87 · Cited by 165
Churpek, M. M., Yuen, T. C., Park, S. Y., Gibbons, R., & Edelson, D. P. (2014) · Critical Care Medicine · doi:10.1097/ccm.0000000000000038 · Cited by 138
Self-citationBailey, T. C., Chen, Y., Mao, Y., Lu, C., Hackmann, G., Micek, S. T., Heard, K. M., Faulkner, K. M., & Kollef, M. H. (2013) · Journal of Hospital Medicine · doi:10.1002/jhm.2009 · Cited by 120
Kellett, J. G., & Kim, A. (2011) · Resuscitation · doi:10.1016/j.resuscitation.2011.08.022 · Cited by 103
Anticipating events of in-hospital cardiac arrest
Berlot, G., Pangher, A., Petrucci, L., Bussani, R., & Lucangelo, U. (2004) · European Journal of Emergency Medicine · doi:10.1097/00063110-200402000-00005 · Cited by 70
Jarvis, S. W., Kovacs, C., Badriyah, T., Briggs, J., Mohammed, M. A., Meredith, P., Schmidt, P. E., Featherstone, P. I., Prytherch, D. R., & Smith, G. B. (2013) · Resuscitation · doi:10.1016/j.resuscitation.2013.05.018 · Cited by 52
Common laboratory tests predict imminent death in ward patients
Loekito, E., Bailey, J., Bellomo, R., Hart, G. K., Hegarty, C., Davey, P. G., Bain, C., Pilcher, D. V., & Schneider, H. G. (2012) · Resuscitation · doi:10.1016/j.resuscitation.2012.07.025 · Cited by 43
Kho, A., Rotz, D., Alrahi, K., Cárdenas, W., Ramsey, K., Liebovitz, D., Noskin, G. A., & Watts, C. (2007) · PubMed · Cited by 38
Waiting for organ transplantation: results of an analysis by an Institute of Medicine Committee
Gibbons, R. D., Duan, N., Meltzer, D., Pope, A., Penhoet, E. D., Dubler, N. N., Francis, C., Gill, B., Guinan, E., Henderson, M., et al. (2003) · Biostatistics · doi:10.1093/biostatistics/4.2.207 · Cited by 36
Self-citationToward a Two-Tier Clinical Warning System for Hospitalized Patients
Hackmann, G. (2010) · PubMed · doi:10.7936/k70v8b1g · Cited by 34
Cuthbertson, B. H., Boroujerdi, M. A., & Prescott, G. J. (2010) · The Journal of the Royal College of Physicians of Edinburgh · doi:10.4997/jrcpe.2010.105 · Cited by 26
Cited by
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817
Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936
Churpek, M. M., Snyder, A., Han, X., Sokol, S., Pettit, N., Howell, M. D., & Edelson, D. P. (2017) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201604-0854oc · Cited by 689
Self-citationChurpek, M. M., Yuen, T. C., Winslow, C., Meltzer, D. O., Kattan, M. W., & Edelson, D. P. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000001571 · Cited by 638
Self-citationGerry, S., Bonnici, T., Birks, J., Kirtley, S., Virdee, P. S., Watkinson, P. J., & Collins, G. S. (2020) · BMJ · doi:10.1136/bmj.m1501 · Cited by 383
Metadata
- Authors
- Churpek, M. M., Yuen, T. C., Winslow, C., Robicsek, A. A., Meltzer, D. O., Gibbons, R. D., & Edelson, D. P.
- Venue
- American Journal of Respiratory and Critical Care Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-09-15
- Volume
- 190
- Issue
- 6
- Pages
- 649-655
- ISSN
- 1073-449X (ISSN-L)
- 1535-4970
- OpenAlex
- W2096978103
- Cited by
- 259
- References
- 44
- FWCI
- 17.25
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.