CoChat
Sign Up
Cited by 102Healthcare Technology and Patient MonitoringSepsis Diagnosis and TreatmentClinical Reasoning and Diagnostic Skills

Development and validation of early warning score system: A systematic literature review

Open access
Fu, Li-Heng, Schwartz, Jessica, Moy, Amanda, Knaplund, Chris, Kang, Min-Jeoung, Schnock, Kumiko O., Garcia, Jose P., Jia, Haomiao +4 more·2020·Journal of Biomedical Informatics
DOI 10.1016/j.jbi.2020.103410PMC PMC7295317
Open this paper in CoChat — free accountOpen PDF

Abstract

Objectives This review aims to: 1) evaluate the quality of model reporting, 2) provide an overview of methodology for developing and validating Early Warning Score Systems (EWSs) for adult patients in acute care settings, and 3) highlight the strengths and limitations of the methodologies, as well as identify future directions for EWS derivation and validation studies. Methodology A systematic search was conducted in PubMed, Cochrane Library, and CINAHL. Only peer reviewed articles and clinical guidelines regarding developing and validating EWSs for adult patients in acute care settings were included. 615 articles were extracted and reviewed by five of the authors. Selected studies were evaluated based on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist. The studies were analyzed according to their study design, predictor selection, outcome measurement, methodology of modeling, and validation strategy. Results A total of 29 articles were included in the final analysis. Twenty-six articles reported on the development and validation of a new EWS, while three reported on validation and model modification. Only eight studies met more than 75% of the items in the TRIPOD checklist. Three major techniques were utilized among the studies to inform their predictive algorithms: 1) clinical-consensus models (n = 6), 2) regression models (n = 15), and 3) tree models (n = 5). The number of predictors included in the EWSs varied from 3 to 72 with a median of seven. Twenty-eight models included vital signs, while 11 included lab data. Pulse oximetry, mental status, and other variables extracted from electronic health records (EHRs) were among other frequently used predictors. In-hospital mortality, unplanned transfer to the intensive care unit (ICU), and cardiac arrest were commonly used clinical outcomes. Twenty-eight studies conducted a form of model validation either within the study or against other widely-used EWSs. Only three studies validated their model using an external database separate from the derived database. Conclusion This literature review demonstrates that the characteristics of the cohort, predictors, and outcome selection, as well as the metrics for model validation, vary greatly across EWS studies. There is no consensus on the optimal strategy for developing such algorithms since data-driven models with acceptable predictive accuracy are often site-specific. A standardized checklist for clinical prediction model reporting exists, but few studies have included reporting aligned with it in their publications. Data-driven models are subjected to biases in the use of EHR data, thus it is particularly important to provide detailed study protocols and acknowledge, leverage, or reduce potential biases of the data used for EWS development to improve transparency and generalizability.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2020.103410

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • CONCERN EWS overcomes the limitations of physiological measurements by leveraging nursing surveillance patterns and an existing expert knowledge base.

    “By leveraging nursing surveillance patterns rather than physiological measures10,44, CONCERN EWS overcomes the limitations of physiological measurements8 and leverages an existing expert knowledge base: nurses’ autonomous decisions to take greater action and document more observations or notes than mandated by policy in direct response to concerning changes in a patient’s clinical state10,13.”

    From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [8]

    The Discussion sections on data-driven EWS derivation and potential bias in EHR data describe limitations of individual physiological observations and say nursing documentation can help predict patient changes. They do not establish that CONCERN EWS overcomes those limitations or that it uses nurses’ autonomous decisions as an expert knowledge base. No page numbers are available in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The comprehensive implementation science approach resulted in 42-hour greater lead time than other EWSs.

    “This comprehensive implementation science approach resulted in 42-hour greater lead time than other EWSs8, a time frame that enables identification of patient deterioration several clinical shifts before an adverse event and provides the additional time and opportunity needed to intervene in clinically meaningful ways.”

    From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [8]

    The Results section describes EWS development and notes that 24- and 48-hour windows were commonly used to evaluate outcomes, but it does not report a 42-hour lead-time advantage. Searches for “42-hour greater lead time” and “implementation science approach” found no passage supporting the attributed comparison. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 92 · Cited by 102

Research works printed but not linked: 77

  1. [1]Merchant RM, Incidence of treated cardiac arrest in hospitalized patients in the United States. Crit Care Med, 2011 39(11): p. 2401– 6. 21705896 PMC3196742
  2. [2]Bapoje SR, Unplanned transfers to a medical intensive care unit: causes and relationship to preventable errors in care. J Hosp Med, 2011 6(2): p. 68– 72. 21290577
  3. [3]Hillman KM, Duration of life-threatening antecedents prior to intensive care admission. Intensive Care Med, 2002 28(11): p. 1629– 34. 12415452
  4. [4]Kause J, A comparison of antecedents to cardiac arrests, deaths and emergency intensive care admissions in Australia and New Zealand, and the United Kingdom--the ACADEMIA study. Resuscitation, 2004 62(3): p. 275– 82. 15325446
  5. [6]Subbe CP, Validation of a modified Early Warning Score in medical admissions. QJM, 2001 94(10): p. 521– 6. 11588210
  6. [7]Smith ME, Early warning system scores for clinical deterioration in hospitalized patients: a systematic review. Ann Am Thorac Soc, 2014 11(9): p. 1454– 65. 25296111
  7. [8]Smith MEB, in Early Warning System Scores: A Systematic Review. 2014: Washington (DC). 25506953
  8. [9]Churpek MM, Yuen TC, and Edelson DP, Risk stratification of hospitalized patients on the wards. Chest, 2013 143(6): p. 1758– 1765. 23732586 PMC3673668
  9. [10]Kyriacos U, Jelsma J, and Jordan S, Monitoring vital signs using early warning scoring systems: a review of the literature. J Nurs Manag, 2011 19(3): p. 311– 30. 21507102
  10. [11]Gao H, Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med, 2007 33(4): p. 667– 79. 17318499
  11. [12]Smith GB, Review and performance evaluation of aggregate weighted ‘track and trigger’systems. Resuscitation, 2008 77(2): p. 170– 9. 18249483
  12. [13]Moons KG, Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med, 2015 162(1): p. W1– 73. 25560730
  13. [14]Collins GS, Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med, 2015 162(1): p. 55– 63. 25560714
  14. [15]Heus P, Poor reporting of multivariable prediction model studies: towards a targeted implementation strategy of the TRIPOD statement. BMC Med, 2018 16(1): p. 120. 30021577 PMC6052616
  15. [16]Moher D, Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol, 2009 62(10): p. 1006– 12. 19631508
  16. [17]Cretikos M, The objective medical emergency team activation criteria: a case-control study. Resuscitation, 2007 73(1): p. 62– 72. 17241732
  17. [18]Prytherch DR, ViEWS--Towards a national early warning score for detecting adult inpatient deterioration. Resuscitation, 2010 81(8): p. 932– 7. 20637974
  18. [19]Kellett J and Kim A, Validation of an abbreviated Vitalpac Early Warning Score (ViEWS) in 75,419 consecutive admissions to a Canadian regional hospital. Resuscitation, 2012 83(3): p. 297– 302. 21907689
  19. [21]Luis L and Nunes C, Short National Early Warning Score - Developing a Modified Early Warning Score. Aust Crit Care, 2018 31(6): p. 376– 381. 29242109
  20. [22]Duckitt RW, Worthing physiological scoring system: derivation and validation of a physiological early-warning system for medical admissions. An observational, population-based single-centre study. Br J Anaesth, 2007 98(6): p. 769– 74. 17470844
  21. [23]Cuthbertson BH, Can physiological variables and early warning scoring systems allow early recognition of the deteriorating surgical patient? Crit Care Med, 2007 35(2): p. 402– 9. 17205002
  22. [24]Cuthbertson BH, Boroujerdi M, and Prescott G, The use of combined physiological parameters in the early recognition of the deteriorating acute medical patient. J R Coll Physicians Edinb, 2010 40(1): p. 19– 25. 21125034
  23. [25]Bleyer AJ, Longitudinal analysis of one million vital signs in patients in an academic medical center. Resuscitation, 2011 82(11): p. 1387– 92. 21756971
  24. [26]Tarassenko L, Centile-based early warning scores derived from statistical distributions of vital signs. Resuscitation, 2011 82(8): p. 1013– 8. 21482011
  25. [27]Churpek MM, Derivation of a cardiac arrest prediction model using ward vital signs*. Crit Care Med, 2012 40(7): p. 2102– 8. 22584764 PMC3378796
  26. [28]Escobar GJ, Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med, 2012 7(5): p. 388– 95. 22447632
  27. [29]Alvarez CA, Predicting out of intensive care unit cardiopulmonary arrest or death using electronic medical record data. BMC Med Inform Decis Mak, 2013 13: p. 28. 23442316 PMC3599266
  28. [30]Jarvis SW, Development and validation of a decision tree early warning score based on routine laboratory test results for the discrimination of hospital mortality in emergency medical admissions. Resuscitation, 2013 84(11): p. 1494– 9. 23732049
  29. [31]Kirkland LL, A clinical deterioration prediction tool for internal medicine patients. Am J Med Qual, 2013 28(2): p. 135– 42. 22822159
  30. [32]Mohammed MA, Index blood tests and national early warning scores within 24 hours of emergency admission can predict the risk of in-hospital mortality: a model development and validation study. PLoS One, 2013 8(5): p. e64340. 23734195 PMC3667137
  31. [33]Badriyah T, Decision-tree early warning score (DTEWS) validates the design of the National Early Warning Score (NEWS). Resuscitation, 2014 85(3): p. 418– 23. 24361673
  32. [34]Churpek MM, Using electronic health record data to develop and validate a prediction model for adverse outcomes in the wards*. Crit Care Med, 2014 42(4): p. 841– 8. 24247472 PMC3959228
  33. [35]Churpek MM, Multicenter development and validation of a risk stratification tool for ward patients. Am J Respir Crit Care Med, 2014 190(6): p. 649– 55. 25089847 PMC4214112
  34. [36]Churpek MM, Adhikari R, and Edelson DP, The value of vital sign trends for detecting clinical deterioration on the wards. Resuscitation, 2016 102: p. 1– 5. 26898412 PMC4834231
  35. [37]Churpek MM, Multicenter Comparison of Machine Learning Methods and Conventional Regression for Predicting Clinical Deterioration on the Wards. Crit Care Med, 2016 44(2): p. 368– 74. 26771782 PMC4736499
  36. [38]Kipnis P, Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU. J Biomed Inform, 2016 64: p. 10– 19. 27658885 PMC5510648
  37. [39]Moore CC, Derivation and validation of a universal vital assessment (UVA) score: a tool for predicting mortality in adult hospitalised patients in sub-Saharan Africa. BMJ Glob Health, 2017 2(2): p. e000344. PMC5656117 29082001
  38. [40]Dziadzko MA, Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital. Crit Care, 2018 22(1): p. 286. 30373653 PMC6206729
  39. [41]Faisal M, Development and validation of a novel computer-aided score to predict the risk of in-hospital mortality for acutely ill medical admissions in two acute hospitals using their first electronically recorded blood test results and vital signs: a cross-sectional study. BMJ Open, 2018 8(12): p. e022939. PMC6286481 30530474
  40. [42]Ghosh E, Early Deterioration Indicator: Data-driven approach to detecting deterioration in general ward. Resuscitation, 2018 122: p. 99– 105. 29122648
  41. [43]Redfern OC, Predicting in-hospital mortality and unanticipated admissions to the intensive care unit using routinely collected blood tests and vital signs: Development and validation of a multivariable model. Resuscitation, 2018 133: p. 75– 81. 30253229 PMC6562198
  42. [44]Watkinson PJ, Manual centile-based early warning scores derived from statistical distributions of observational vital-sign data. Resuscitation, 2018 129: p. 55– 60. 29879432 PMC6062656
  43. [45]Schein RM, Clinical antecedents to in-hospital cardiopulmonary arrest. Chest, 1990 98(6): p. 1388– 92. 2245680
  44. [46]Berlot G, Anticipating events of in-hospital cardiac arrest. Eur J Emerg Med, 2004 11(1): p. 24– 8. 15167189
  45. [47]in Acutely Ill Patients in Hospital: Recognition of and Response to Acute Illness in Adults in Hospital. 2007: London. 21204323
  46. [48]Hodgson LE, A validation of the National Early Warning Score to predict outcome in patients with COPD exacerbation. Thorax, 2017 72(1): p. 23– 30. 27553223
  47. [49]Opio MO, Nansubuga G, and Kellett J, Validation of the VitalPAC Early Warning Score (ViEWS) in acutely ill medical patients attending a resource-poor hospital in sub-Saharan Africa. Resuscitation, 2013 84(6): p. 743– 6. 23438452
  48. [50]Lammers EJ and McLaughlin CG, Meaningful Use of Electronic Health Records and Medicare Expenditures: Evidence from a Panel Data Analysis of U.S. Health Care Markets, 2010-2013. Health services research, 2017 52(4): p. 1364– 1386. 27546309 PMC5517685
  49. [53]Wunsch H, Comparison of medical admissions to intensive care units in the United States and United Kingdom. Am J Respir Crit Care Med, 2011 183(12): p. 1666– 73. 21471089
  50. [55]Poolsawad N, Kambhampati C, and Cleland J. Balancing class for performance of classification with a clinical dataset. in Proceedings of the World Congress on engineering 2014.
  51. [56]Gephart SM, McGrath JM, and Effken JA, Failure to rescue in neonatal care. J Perinat Neonatal Nurs, 2011 25(3): p. 275– 82. 21825918
  52. [57]Manojlovich M and Talsma A, Identifying nursing processes to reduce failure to rescue. J Nurs Adm, 2007 37(11): p. 504– 9. 17975467
  53. [58]Sheetz KH, Dimick JB, and Ghaferi AA, Impact of Hospital Characteristics on Failure to Rescue Following Major Surgery. Ann Surg, 2016 263(4): p. 692– 7. 26501706 PMC4777662
  54. [59]Green M, Comparison of the Between the Flags calling criteria to the MEWS, NEWS and the electronic Cardiac Arrest Risk Triage (eCART) score for the identification of deteriorating ward patients. Resuscitation, 2018 123: p. 86– 91. 29169912 PMC6556215
  55. [60]Smith GB, A Comparison of the Ability of the Physiologic Components of Medical Emergency Team Criteria and the U.K. National Early Warning Score to Discriminate Patients at Risk of a Range of Adverse Clinical Outcomes. Crit Care Med, 2016 44(12): p. 2171– 2181. 27513547
  56. [61]Lyons PG, Edelson DP, and Churpek MM, Rapid response systems. Resuscitation, 2018 128: p. 191– 197. 29777740 PMC6147149
  57. [62]Gonzalez Del Castillo J, Prognostic accuracy of SIRS criteria, qSOFA score and GYM score for 30-day-mortality in older non-severely dependent infected patients attended in the emergency department. Eur J Clin Microbiol Infect Dis, 2017 36(12): p. 2361– 2369. 28755060
  58. [63]Wolpert DH and Macready WG, No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1997 1(1): p. 67– 82.
  59. [64]Watson DS, Clinical applications of machine learning algorithms: beyond the black box. BMJ, 2019 364: p. l886. 30862612
  60. [66]Parikh RB, Obermeyer Z, and Navathe AS, Regulation of predictive analytics in medicine. Science, 2019 363(6429): p. 810. 30792287 PMC6557272
  61. [68]Kawazoe Y, Prediction-based threshold for medication alert. Stud Health Technol Inform, 2013 192: p. 229– 33. 23920550
  62. [70]Verweij LM, Data quality issues impede comparability of hospital treatment delay performance indicators. Neth Heart J, 2015 23(9): p. 420– 7. 26021617 PMC4547943
  63. [71]Weiskopf NG, Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform, 2013 46(5): p. 830– 6. 23820016 PMC3810243
  64. [72]Verheij RA, Possible Sources of Bias in Primary Care Electronic Health Record Data Use and Reuse. J Med Internet Res, 2018 20(5): p. e185. 29844010 PMC5997930
  65. [73]Hripcsak G and Albers DJ, Next-generation phenotyping of electronic health records. J Am Med Inform Assoc, 2013 20(1): p. 117– 21. 22955496 PMC3555337
  66. [74]Hripcsak G and Albers DJ, Correlating electronic health record concepts with healthcare process events. J Am Med Inform Assoc, 2013 20(e2): p. e311– 8. 23975625 PMC3861922
  67. [75]Hripcsak G and Albers DJ, High-fidelity phenotyping: richness and freedom from bias. J Am Med Inform Assoc, 2018 25(3): p. 289– 294. 29040596 PMC7282504
  68. [76]Hripcsak G, Albers DJ, and Perotte A, Exploiting time in electronic health record correlations. J Am Med Inform Assoc, 2011 18 Suppl 1: p. i109– 15. 22116643 PMC3241180
  69. [77]Albers DJ and Hripcsak G, A statistical dynamics approach to the study of human health data: resolving population scale diurnal variation in laboratory data. Phys Lett A, 2010 374(9): p. 1159– 1164. 20544004 PMC2882798
  70. [78]Agniel D, Kohane IS, and Weber GM, Biases in electronic health record data due to processes within the healthcare system: retrospective observational study. BMJ, 2018 361: p. k1479. 29712648 PMC5925441
  71. [79]Collins SA, Relationship between nursing documentation and patients’ mortality. American journal of critical care: an official publication, American Association of Critical-Care Nurses, 2013 22(4): p. 306– 313. PMC3771321 23817819
  72. [80]Rossetti SC, Leveraging Clinical Expertise as a Feature - not an Outcome - of Predictive Models: Evaluation of an Early Warning System Use Case., in Proceeding of the American Medical Informatics Association Annual Fall Symposium 2019: Washington, DC (in press). PMC7153052 32308825
  73. [82]Sherman RE, Real-World Evidence - What Is It and What Can It Tell Us? N Engl J Med, 2016 375(23): p. 2293– 2297. 27959688
  74. [83]Reeves D, Combining multiple indicators of clinical quality: an evaluation of different analytic approaches. Med Care, 2007 45(6): p. 489– 96. 17515775
  75. [84]Hagar Y, Survival analysis with electronic health record data: Experiments with chronic kidney disease. Statistical Analysis and Data Mining: The ASA Data Science Journal, 2014 7(5): p. 385– 403. PMC8112603 33981381
  76. [85]Albers DJ, Estimating summary statistics for electronic health record laboratory data for use in high-throughput phenotyping algorithms. Journal of biomedical informatics, 2018 78: p. 87– 101. 29369797 PMC5856130
  77. [86]Collins GS and Moons KGM, Reporting of artificial intelligence prediction models. Lancet, 2019 393(10181): p. 1577– 1579. 31007185

Authors
Fu, L.-H., Schwartz, J., Moy, A., Knaplund, C., Kang, M.-J., Schnock, K. O., Garcia, J. P., Jia, H., Dykes, P. C., Cato, K., Albers, D., & Rossetti, S. C.
Affiliations
  • Columbia University Irving Medical Center (US)
  • Columbia University (US)
  • Brigham and Women's Hospital (US)
  • University of Colorado Anschutz (US)
Corresponding author
Fu, Li-Heng
ORCID
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2020-05
Volume
105
Pages
103410
ISSN
1532-0464 (ISSN-L)
DOI
10.1016/j.jbi.2020.103410
PMC
PMC7295317
OpenAlex
W3015248561
Cited by
102
References
92
Citations by year
2026: 11 · 2025: 16 · 2024: 22 · 2023: 22 · 2022: 19 · 2021: 10 · 2020: 2
FWCI
3.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Informatics (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
U.S. National Library of Medicine
Topics
  • Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
acute care, adult patients, prediction model development, prediction model validation, TRIPOD checklist, electronic health records, vital signs, in-hospital mortality, unplanned ICU transfer, cardiac arrest, external validation, regression models, tree-based models, reporting quality, risk stratification, early detection of deterioration
MeSH terms
Early Warning Score, Adult, Humans, Intensive Care Units, Prognosis, Models, Statistical, Vital Signs

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review