CoChat
Sign Up
Cited by 162Healthcare Technology and Patient MonitoringElectronic Health Records SystemsPatient Safety and Medication Errors

Minimal Impact of Implemented Early Warning Score and Best Practice Alert for Patient Deterioration*

Open access
Bedoya, Armando D., Clement, Meredith E., Phelan, Matthew, Steorts, Rebecca C., O’Brien, Cara, Goldstein, Benjamin A.·2019·Critical Care Medicine
DOI 10.1097/ccm.0000000000003439PMC PMC6298839
Open this paper in CoChat — free account

Abstract

Objectives: Previous studies have looked at National Early Warning Score performance in predicting in-hospital deterioration and death, but data are lacking with respect to patient outcomes following implementation of National Early Warning Score. We sought to determine the effectiveness of National Early Warning Score implementation on predicting and preventing patient deterioration in a clinical setting. Design: Retrospective cohort study. Setting: Tertiary care academic facility and a community hospital. Patients: Patients 18 years old or older hospitalized from March 1, 2014, to February 28, 2015, during preimplementation of National Early Warning Score to August 1, 2015, to July 31, 2016, after National Early Warning Score was implemented. Interventions: Implementation of National Early Warning Score within the electronic health record and associated best practice alert. Measurements and Main Results: In this study of 85,322 patients (42,402 patients pre-National Early Warning Score and 42,920 patients post-National Early Warning Score implementation), the primary outcome of rate of ICU transfer or death did not change after National Early Warning Score implementation, with adjusted hazard ratio of 0.94 (0.84–1.05) and 0.90 (0.77–1.05) at our academic and community hospital, respectively. In total, 175,357 best practice advisories fired during the study period, with the best practice advisory performing better at the community hospital than the academic at predicting an event within 12 hours 7.4% versus 2.2% of the time, respectively. Retraining National Early Warning Score with newly generated hospital-specific coefficients improved model performance. Conclusions: At both our academic and community hospital, National Early Warning Score had poor performance characteristics and was generally ignored by frontline nursing staff. As a result, National Early Warning Score implementation had no appreciable impact on defined clinical outcomes. Refitting of the model using site-specific data improved performance and supports validating predictive models on local data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1097/ccm.0000000000003439

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • False positive rates can contribute to unnecessary diagnostics and antimicrobial use, potentially avoidable ICU transfers, and alert fatigue.

    “False positive rates, on the other hand, can contribute to unnecessary diagnostics and antimicrobial use, potentially avoidable intensive care unit (ICU) transfers, and alert fatigue.6,7”

    From A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools (2026) · cites this paper as [6]

    The Discussion reports that alerts requiring no clinical intervention contributed to alert fatigue. It does not report that false positives caused unnecessary diagnostics, antimicrobial use, or avoidable ICU transfers; the Results report no change in ICU transfer rates after implementation. Page numbers are not available in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Many sepsis CDS systems fail to improve outcomes because of poor diagnostic accuracy and program implementation.

    “Unfortunately, many sepsis CDS systems fail to improve outcomes because of poor diagnostic accuracy and program implementation [23-25].”

    From Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study (2020) · cites this paper as [23]

    The Abstract conclusions and Discussion report that the implemented NEWS alert had no appreciable impact on clinical outcomes, performed poorly, and was generally ignored by nursing staff. The paper concerns one early-warning system for patient deterioration, not many sepsis-specific CDS systems, so it does not support that scope of the claim. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Previous attempts at sending automated CDS alerts directly to the treating provider have been associated with high levels of alert fatigue.

    “Previous attempts at sending automated CDS alerts directly to the treating provider have been associated with high levels of alert fatigue [23].”

    From Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study (2020) · cites this paper as [23]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. [1]van Galen LS, Struik PW, Driesen BEJM,: Delayed Recognition of Deterioration of Patients in General Wards Is Mostly Caused by Human Related Monitoring Failures: A Root Cause Analysis of Unplanned ICU Admissions [Internet]. PLoS One 2016; 11: e0161393Available from: 10.1371/journal.pone.0161393 27537689 PMC4990328 · DOI
  2. [2]Chalfin DB, Trzeciak S, Likourezos A,: Impact of delayed transfer of critically ill patients from the emergency department to the intensive care unit. Crit Care Med 2007; 35: 1477– 1483 17440421
  3. [3]Liu V, Kipnis P, Rizk NW,: Adverse outcomes associated with delayed intensive care unit transfers in an integrated healthcare system. J Hosp Med 2012; 7: 224– 230 22038879
  4. [4]Jaderling G, Bell M, Martling CR,: ICU admittance by a rapid response team versus conventional admittance, characteristics, and outcome. Crit Care Med 2013; 41: 725– 731 23318488
  5. [5]Renaud B, Brun-Buisson C, Santin A,: Outcomes of early, late, and no admission to the intensive care unit for patients hospitalized with community-acquired pneumonia. Acad Emerg Med 2012; 19: 294– 303 22435862
  6. [7]Girotra S, Nallamothu BK, Spertus JA,: Trends in Survival after In-Hospital Cardiac Arrest [Internet]. N Engl J Med 2012; 367: 1912– 1920Available from: http://www.nejm.org/doi/full/10.1056/NEJMoa1109148 23150959 PMC3517894 · DOI · Paper page
  7. [8]Schein RM, Hazday N, Pena M,: Clinical antecedents to in-hospital cardiopulmonary arrest. Chest 1990; 98: 1388– 1392 2245680
  8. [9]Hillman KM, Bristow PJ, Chey T,: Antecedents to hospital deaths [Internet]. Intern Med J 2001; 31: 343– 348Available from: http://onlinelibrary.wiley.com/store/10.1046/j.1445-5994.2001.00077.x/asset/j.1445-5994.2001.00077.x.pdf?v=1&t=j9von1g4&s=4dd4122742851abc832c29d648066b4569ebcab8 11529588 · DOI
  9. [10]Berlot G, Pangher A, Petrucci L,: Anticipating events of in-hospital cardiac arrest. Eur J Emerg Med 2004; 11: 24– 28 15167189
  10. [11]Hodgetts TJ, Kenward G, Vlachonikolis IG,: The identification of risk factors for cardiac arrest and formulation of activation criteria to alert a medical emergency team. Resuscitation 2002; 54: 125– 131 12161291
  11. [12]Hodgetts TJ, Kenward G, Vlackonikolis I,: Incidence, location and reasons for avoidable in-hospital cardiac arrest in a district general hospital. Resuscitation 2002; 54: 115– 123 12161290
  12. [13]Goldstein BA, Navar AM, Pencina MJ,: Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review [Internet]. J Am Med Inf Assoc 2017; 24: 198– 208Available from: https://www.ncbi.nlm.nih.gov/pubmed/27189013 PMC5201180 27189013
  13. [14]McGaughey J, Alderdice F, Fowler R,: Outreach and Early Warning Systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev 2007; Cd005529 17636805
  14. [15]Smith GB, Prytherch DR, Meredith P,: The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation 2013; 84: 465– 470 23295778
  15. [17]Sbiti-Rohr D, Kutz A, Christ-Crain M,: The National Early Warning Score (NEWS) for outcome prediction in emergency department patients with community-acquired pneumonia: results from a 6-year prospective cohort study. BMJ Open 2016; 6: e011021 PMC5051330 27683509
  16. [18]Abbott TE, Vaid N, Ip D,: A single-centre observational cohort study of admission National Early Warning Score (NEWS). Resuscitation 2015; 92: 89– 93 25943011
  17. [19]Kovacs C, Jarvis SW, Prytherch DR,: Comparison of the National Early Warning Score in non-elective medical and surgical patients. Br J Surg 2016; 103: 1385– 1393 27487317
  18. [20]Spangfors M, Arvidsson L, Karlsson V,: The National Early Warning Score: Translation, testing and prediction in a Swedish setting. Intensive Crit Care Nurs 2016; 37: 62– 67 27386753
  19. [21]Subbe CP, Davies RG, Williams E,: Effect of introducing the Modified Early Warning score on clinical outcomes, cardio-pulmonary arrests and intensive care utilisation in acute medical admissions. [Internet]. Anaesthesia 2003; 58: 797– 802[cited 2018 Mar 4] Available from: http://www.ncbi.nlm.nih.gov/pubmed/12859475 12859475
  20. [24]Alam N, Vegting IL, Houben E,: Exploring the performance of the National Early Warning Score (NEWS) in a European emergency department. Resuscitation 2015; 90: 111– 115 25748878
  21. [25]Spagnolli W, Rigoni M, Torri E,: Application of the National Early Warning Score (NEWS) as a stratification tool on admission in an Italian acute medical ward: A perspective study. Int J Clin Pr 2017; 71 28276182
  22. [26]Silcock DJ, Corfield AR, Gowens PA,: Validation of the National Early Warning Score in the prehospital setting. Resuscitation 2015; 89: 31– 35 25583148
  23. [27]Prytherch DR, Smith GB, Schmidt PE,: ViEWS--Towards a national early warning score for detecting adult inpatient deterioration. Resuscitation 2010; 81: 932– 937 20637974
  24. [28]Churpek MM, Yuen TC, Huber MT,: Predicting cardiac arrest on the wards: a nested case-control study. Chest 2012; 141: 1170– 1176 22052772 PMC3342781
  25. [29]Cvach M: Monitor alarm fatigue: an integrative review. Biomed Instrum Technol 2012; 46: 268– 277 22839984
  26. [30]Isaac T, Weissman JS, Davis RB,: Overrides of medication alerts in ambulatory care. Arch Intern Med 2009; 169: 305– 311 19204222
  27. [31]van der Sijs H, Aarts J, Vulto A,: Overriding of drug safety alerts in computerized physician order entry. J Am Med Inf Assoc 2006; 13: 138– 147 PMC1447540 16357358
  28. [32]Embi PJ, Leonard AC: Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study. J Am Med Inf Assoc 2012; 19: e145– 8 PMC3392862 22534081
  29. [33]Goldstein BA, Pencina MJ, Montez-Rath ME,: Predicting mortality over different time horizons: which data elements are needed? [Internet]. J Am Med Inf Assoc 2017; 24: 176– 181Available from: https://www.ncbi.nlm.nih.gov/pubmed/27357832 PMC5201182 27357832
  30. [35]Rothman MJ, Rothman SI, Beals J: Development and validation of a continuous measure of patient condition using the Electronic Medical Record. J Biomed Inform 2013; 46: 837– 848 23831554
  31. [36]Siontis GC, Tzoulaki I, Castaldi PJ,: External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 2015; 68: 25– 34 25441703
  32. [37]Goldstein BA, Navar AM, Pencina MJ: Risk Prediction With Electronic Health Records: The Importance of Model Validation and Clinical Context [Internet]. JAMA Cardiol 2016; 1: 976– 977Available from: https://www.ncbi.nlm.nih.gov/pubmed/27732706 27732706 PMC5510620

Cited by

Sorted by citations

Authors
Bedoya, A. D., Clement, M. E., Phelan, M., Steorts, R. C., O’Brien, C., & Goldstein, B. A.
Affiliations
Duke University (US)
ORCID
Venue
Critical Care Medicine
Publisher
Ovid Technologies (Wolters Kluwer Health)
Published
2019-01
Volume
47
Issue
1
Pages
49-55
ISSN
0090-3493 (ISSN-L)
DOI
10.1097/ccm.0000000000003439
PMC
PMC6298839
OpenAlex
W2895072154
Cited by
162
References
3
Citations by year
2026: 13 · 2025: 21 · 2024: 26 · 2023: 20 · 2022: 25 · 2021: 26 · 2020: 20 · 2019: 9 · 2018: 1
FWCI
14.55
Type
journal article
Language
en
Open access
Locations
  • Critical Care Medicine (publishedVersion)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • National Institute of Allergy and Infectious Diseases (T32AI007392)
  • National Institute of Diabetes and Digestive and Kidney Diseases (K25DK097279)
Topics
  • Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Health Professions › Emergency Medical Services › Patient Safety and Medication Errors
Keywords
National Early Warning Score, clinical deterioration, early warning scores, best practice alerts, electronic health record alerts, in-hospital mortality, ICU transfer, prediction model validation, community hospitals, academic medical centers
MeSH terms
Academic Medical Centers, Clinical Deterioration, Adult, Aged, Attitude of Health Personnel, Female, Hospitals, Community, Humans, Intensive Care Units, Male, Middle Aged, North Carolina, Nursing Staff, Hospital, Patient Transfer, Retrospective Studies, Cohort Studies, Hospital Mortality, Early Diagnosis, Clinical Alarms, Patient Acuity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review