The DOI matches its Crossref record; 133 of the 133 references it cites were found in a registry or an index.
Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology
Open accessAbstract
Objective: To provide an overview and critical appraisal of early warning scores for adult hospital patients. Design: Systematic review. Data sources: Medline, CINAHL, PsycInfo, and Embase until June 2019. Eligibility criteria for study selection: Studies describing the development or external validation of an early warning score for adult hospital inpatients. Results: 13 171 references were screened and 95 articles were included in the review. 11 studies were development only, 23 were development and external validation, and 61 were external validation only. Most early warning scores were developed for use in the United States (n=13/34, 38%) and the United Kingdom (n=10/34, 29%). Death was the most frequent prediction outcome for development studies (n=10/23, 44%) and validation studies (n=66/84, 79%), with different time horizons (the most frequent was 24 hours). The most common predictors were respiratory rate (n=30/34, 88%), heart rate (n=28/34, 83%), oxygen saturation, temperature, and systolic blood pressure (all n=24/34, 71%). Age (n=13/34, 38%) and sex (n=3/34, 9%) were less frequently included. Key details of the analysis populations were often not reported in development studies (n=12/29, 41%) or validation studies (n=33/84, 39%). Small sample sizes and insufficient numbers of event patients were common in model development and external validation studies. Missing data were often discarded, with just one study using multiple imputation. Only nine of the early warning scores that were developed were presented in sufficient detail to allow individualised risk prediction. Internal validation was carried out in 19 studies, but recommended approaches such as bootstrapping or cross validation were rarely used (n=4/19, 22%). Model performance was frequently assessed using discrimination (development n=18/22, 82%; validation n=69/84, 82%), while calibration was seldom assessed (validation n=13/84, 15%). All included studies were rated at high risk of bias. Conclusions: Early warning scores are widely used prediction models that are often mandated in daily clinical practice to identify early clinical deterioration in hospital patients. However, many early warning scores in clinical use were found to have methodological weaknesses. Early warning scores might not perform as well as expected and therefore they could have a detrimental effect on patient care. Future work should focus on following recommended approaches for developing and evaluating early warning scores, and investigating the impact and safety of using these scores in clinical practice. Systematic review registration: PROSPERO CRD42017053324.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.08.010 · Cited by 51,384
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,265
To Err Is Human: Building a Safer Health System
(2000) · National Academies Press eBooks · doi:10.17226/9728 · Cited by 14,044
Multiple imputation using chained equations: Issues and guidance for practice
White, I. R., Royston, P., & Wood, A. M. (2011) · Statistics in Medicine · doi:10.1002/sim.4067 · Cited by 9,792
Statistical Analysis with Missing Data
Little, R. J. A., & Rubin, D. B. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9781119013563 · Cited by 8,473
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Brennan, T. A., Leape, L. L., Laird, N. M., Hebert, L., Localio, A. R., Lawthers, A. G., Newhouse, J. P., Weiler, P. C., & Hiatt, H. H. (1991) · New England Journal of Medicine · doi:10.1056/nejm199102073240604 · Cited by 4,627
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Self-citationPROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984
Internal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Introduction to the Analysis of Survival Data in the Presence of Competing Risks
Austin, P. C., Lee, D. S., & Fine, J. P. (2016) · Circulation · doi:10.1161/circulationaha.115.017719 · Cited by 2,653
Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119
Adverse events in British hospitals: preliminary retrospective record review
Vincent, C., Neale, G., & Woloshynowych, M. (2001) · BMJ · doi:10.1136/bmj.322.7285.517 · Cited by 2,065
Moons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018
Blanche, P., Dartigues, J., & Jacqmin‐Gadda, H. (2013) · Statistics in Medicine · doi:10.1002/sim.5958 · Cited by 1,852
Validation of a modified Early Warning Score in medical admissions
Subbe, C. P. (2001) · QJM · doi:10.1093/qjmed/94.10.521 · Cited by 1,762
Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409
Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators
Van Calster, B., Wynants, L., Verbeek, J. F. M., Verbakel, J. Y., Christodoulou, E., Vickers, A. J., Roobol, M. J., & Steyerberg, E. W. (2018) · European Urology · doi:10.1016/j.eururo.2018.08.038 · Cited by 1,271
Smith, G. B., Prytherch, D. R., Meredith, P., Schmidt, P. E., & Featherstone, P. I. (2013) · Resuscitation · doi:10.1016/j.resuscitation.2012.12.016 · Cited by 1,077
Regression with Missing X 's: A Review
Little, R. J. A. (1992) · Journal of the American Statistical Association · doi:10.1080/01621459.1992.10476282 · Cited by 968
Brennan, T. A. (2004) · Quality and Safety in Health Care · doi:10.1136/qshc.2002.003822 · Cited by 942
Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936
Prediction models for cardiovascular disease risk in the general population: systematic review
Damen, J. A. A. G., Hooft, L., Schuit, E., Debray, T. P. A., Collins, G. S., Tzoulaki, I., Lassale, C. M., Siontis, G. C. M., Chiocchia, V., Roberts, C., et al. (2016) · BMJ · doi:10.1136/bmj.i2416 · Cited by 935
Self-citationA calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910
Sample size for binary logistic prediction models: Beyond events per variable criteria
van Smeden, M., Moons, K. G., de Groot, J. A., Collins, G. S., Altman, D. G., Eijkemans, M. J., & Reitsma, J. B. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218784726 · Cited by 791
Self-citationChurpek, M. M., Snyder, A., Han, X., Sokol, S., Pettit, N., Howell, M. D., & Edelson, D. P. (2017) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201604-0854oc · Cited by 689
Collins, G. S., Ogundimu, E. O., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6787 · Cited by 665
Prognostic Models With Competing Risks: Methods and Application to Coronary Risk Prediction
Wolbers, M., Koller, M. T., Witteman, J. C. M., & Steyerberg, E. W. (2009) · Epidemiology · doi:10.1097/ede.0b013e3181a39056 · Cited by 625
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600
Missing covariate data in medical research: To impute is better than to ignore
Janssen, K. J. M., Donders, A. R. T., Harrell, F. E., Vergouwe, Y., Chen, Q., Grobbee, D. E., & Moons, K. G. M. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.12.008 · Cited by 592
Steyerberg, E. W., Borsboom, G. J. J. M., van Houwelingen, H. C., Eijkemans, M. J. C., & Habbema, J. D. F. (2004) · Statistics in Medicine · doi:10.1002/sim.1844 · Cited by 552
ViEWS—Towards a national early warning score for detecting adult inpatient deterioration
Prytherch, D. R., Smith, G. B., Schmidt, P. E., & Featherstone, P. I. (2010) · Resuscitation · doi:10.1016/j.resuscitation.2010.04.014 · Cited by 535
Collins, G. S., Mallett, S., Omar, O., & Yu, L.-M. (2011) · BMC Medicine · doi:10.1186/1741-7015-9-103 · Cited by 484
Gao, H., McDonnell, A., Harrison, D. A., Moore, T., Adam, S., Daly, K., Esmonde, L., Goldhill, D. R., Parry, G. J., Rashidian, A., et al. (2007) · Intensive Care Medicine · doi:10.1007/s00134-007-0532-3 · Cited by 466
The impact of the use of the Early Warning Score (EWS) on patient outcomes: A systematic review
Alam, N., Hobbelink, E. L., van Tienhoven, A. J., van de Ven, P. M., Jansma, E. P., & Nanayakkara, P. W. B. (2014) · Resuscitation · doi:10.1016/j.resuscitation.2014.01.013 · Cited by 432
Early Warning System Scores for Clinical Deterioration in Hospitalized Patients: A Systematic Review
Smith, M. E. B., Chiovaro, J. C., O’Neil, M., Kansagara, D., Quiñones, A. R., Freeman, M., Motu’apuaka, M. L., & Slatore, C. G. (2014) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201403-102oc · Cited by 387
Gardner-Thorpe, J., Love, N., Wrightson, J., Walsh, S., & Keeling, N. (2006) · The Annals of The Royal College of Surgeons of England · doi:10.1308/003588406x130615 · Cited by 375
Austin, P. C., & Steyerberg, E. W. (2014) · Statistics in Medicine · doi:10.1002/sim.5941 · Cited by 362
McGaughey, J., Alderdice, F., Fowler, R., Kapila, A., Mayhew, A., & Moutray, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005529.pub2 · Cited by 348
Duration of life-threatening antecedents prior to intensive care admission
Hillman, K., Bristow, P., Chey, T., Daffurn, K., Jacques, T., Norman, S., Bishop, G., & Simmons, G. (2002) · Intensive Care Medicine · doi:10.1007/s00134-002-1496-y · Cited by 347
Systematic review of prognostic models in traumatic brain injury
Perel, P., Edwards, P., Wentz, R., & Roberts, I. (2006) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-6-38 · Cited by 344
Review and performance evaluation of aggregate weighted ‘track and trigger’ systems
Smith, G. B., Prytherch, D. R., Schmidt, P. E., & Featherstone, P. I. (2008) · Resuscitation · doi:10.1016/j.resuscitation.2007.12.004 · Cited by 337
Competing risks analyses: objectives and approaches
Wolbers, M., Koller, M. T., Stel, V. S., Schaer, B., Jager, K. J., Leffondré, K., & Heinze, G. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu131 · Cited by 330
Physiological abnormalities in early warning scores are related to mortality in adult inpatients †
Goldhill, D. R., & McNarry, A. F. (2004) · British Journal of Anaesthesia · doi:10.1093/bja/aeh113 · Cited by 325
Fairclough, E., Cairns, E., Hamilton, J., & Kelly, C. (2009) · Clinical Medicine · doi:10.7861/clinmedicine.9-1-30 · Cited by 308
An algorithm based on deep learning for predicting in-hospital cardiac arrest
Kwon, J., Lee, Y., Lee, Y., Lee, S., & Park, J. (2018) · Journal of the American Heart Association · doi:10.1161/jaha.118.008678 · Cited by 299
Development and validation of a prediction model with missing predictor data: a practical approach
Vergouwe, Y., Royston, P., Moons, K. G. M., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.03.017 · Cited by 287
Multicenter Development and Validation of a Risk Stratification Tool for Ward Patients
Churpek, M. M., Yuen, T. C., Winslow, C., Robicsek, A. A., Meltzer, D. O., Gibbons, R. D., & Edelson, D. P. (2014) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201406-1022oc · Cited by 259
Ambler, G., Omar, R. Z., & Royston, P. (2007) · Statistical Methods in Medical Research · doi:10.1177/0962280206074466 · Cited by 233
The value of vital sign trends for detecting clinical deterioration on the wards
Churpek, M. M., Adhikari, R., & Edelson, D. P. (2016) · Resuscitation · doi:10.1016/j.resuscitation.2016.02.005 · Cited by 226
Derivation of a cardiac arrest prediction model using ward vital signs*
Churpek, M. M., Yuen, T. C., Park, S. Y., Meltzer, D. O., Hall, J. B., & Edelson, D. P. (2012) · Critical Care Medicine · doi:10.1097/ccm.0b013e318250aa5a · Cited by 199
Collins, G. S., Omar, O., Shanyinde, M., & Yu, L.-M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.06.020 · Cited by 192
Validation of the National Early Warning Score in the prehospital setting
Silcock, D. J., Corfield, A. R., Gowens, P. A., & Rooney, K. D. (2015) · Resuscitation · doi:10.1016/j.resuscitation.2014.12.029 · Cited by 187
Escobar, G. J., LaGuardia, J. C., Turk, B. J., Ragins, A., Kipnis, P., & Draper, D. (2012) · Journal of Hospital Medicine · doi:10.1002/jhm.1929 · Cited by 185
Predicting Cardiac Arrest on the Wards
Churpek, M. M., Yuen, T. C., Huber, M. T., Park, S. Y., Hall, J. B., & Edelson, D. P. (2012) · Chest · doi:10.1378/chest.11-1301 · Cited by 185
Longitudinal analysis of one million vital signs in patients in an academic medical center
Bleyer, A. J., Vidya, S., Russell, G. B., Jones, C. M., Sujata, L., Daeihagh, P., & Hire, D. (2011) · Resuscitation · doi:10.1016/j.resuscitation.2011.06.033 · Cited by 178
Meads, C., Ahmed, I., & Riley, R. D. (2012) · Breast Cancer Research and Treatment · doi:10.1007/s10549-011-1818-2 · Cited by 177
Duckitt, R. W., Buxton-Thomas, R., Walker, J., Cheek, E., Bewick, V., Venn, R., & Forni, L. G. (2007) · British Journal of Anaesthesia · doi:10.1093/bja/aem097 · Cited by 175
Kipnis, P., Turk, B. J., Wulf, D. A., LaGuardia, J. C., Liu, V., Churpek, M. M., Romero-Brufau, S., & Escobar, G. J. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.09.013 · Cited by 167
Pimentel, M. A. F., Redfern, O. C., Gerry, S., Collins, G. S., Malycha, J., Prytherch, D., Schmidt, P. E., Smith, G. B., & Watkinson, P. J. (2019) · Resuscitation · doi:10.1016/j.resuscitation.2018.09.026 · Cited by 161
Self-citationGreen, M., Lander, H., Snyder, A. M., Hudson, P., Churpek, M. M., & Edelson, D. P. (2017) · Resuscitation · doi:10.1016/j.resuscitation.2017.10.028 · Cited by 148
Risk Stratification of Hospitalized Patients on the Wards
Churpek, M. M., Yuen, T. C., & Edelson, D. P. (2013) · CHEST Journal · doi:10.1378/chest.12-1605 · Cited by 146
Utility of a single early warning score in patients with sepsis in the emergency department
Corfield, A. R., Lees, F., Zealley, I. A., Houston, J. G., Dickie, S., Ward, K., McGuffie, A. C., & on behalf of the Scottish Trauma Audit Group Sepsis Steering Group (2013) · Emergency Medicine Journal · doi:10.1136/emermed-2012-202186 · Cited by 146
The Simple Clinical Score predicts mortality for 30 days after admission to an acute medical unit
Kellett, J. G., & Deane, B. (2006) · QJM · doi:10.1093/qjmed/hcl112 · Cited by 144
Absolute risk regression for competing risks: interpretation, link functions, and prediction
Gerds, T. A., Scheike, T., & Andersen, P. K. (2012) · Statistics in Medicine · doi:10.1002/sim.5459 · Cited by 141
Churpek, M. M., Yuen, T. C., Park, S. Y., Gibbons, R. D., & Edelson, D. P. (2013) · Critical Care Medicine · doi:10.1097/ccm.0000000000000038 · Cited by 140
Centile-based early warning scores derived from statistical distributions of vital signs
Tarassenko, L., Clifton, D. A., Pinsky, M. R., Hravnak, M., Woods, J. R., & Watkinson, P. (2011) · Resuscitation · doi:10.1016/j.resuscitation.2011.03.006 · Cited by 133
Self-citationNEWSDIG: The National Early Warning Score Development and Implementation Group
Jones, M. (2012) · Clinical Medicine · doi:10.7861/clinmedicine.12-6-501 · Cited by 131
Finlay, G. D., Rothman, M. J., & Smith, R. A. (2013) · Journal of Hospital Medicine · doi:10.1002/jhm.2132 · Cited by 121
Calibration plots for risk prediction models in the presence of competing risks
Gerds, T. A., Andersen, P. K., & Kattan, M. W. (2014) · Statistics in Medicine · doi:10.1002/sim.6152 · Cited by 119
Silke, B., Kellett, J. G., Rooney, T., Bennett, K., & O’Riordan, D. M. R. (2009) · QJM · doi:10.1093/qjmed/hcp149 · Cited by 118
Moore, C. C., Hazard, R., Saulters, K. J., Ainsworth, J. E., Adakun, S. A., Abdallah, A., Andrews, B., Auma, M. A., Baker, T., Banura, P., et al. (2017) · BMJ Global Health · doi:10.1136/bmjgh-2017-000344 · Cited by 110
Cei, M., Bartolomei, C., & Mumoli, N. (2009) · International Journal of Clinical Practice · doi:10.1111/j.1742-1241.2008.01986.x · Cited by 104
Kruisselbrink, R. J., Kwizera, A., Crowther, M. A., Fox‐Robichaud, A., O’Shea, T. M., Nakibuuka, J., Ssinabulya, I., Nalyazi, J. N., Bonner, A., Devji, T., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0151408 · Cited by 104
Predicting clinical deterioration in the hospital: The impact of outcome selection
Churpek, M. M., Yuen, T. C., & Edelson, D. P. (2012) · Resuscitation · doi:10.1016/j.resuscitation.2012.09.024 · Cited by 99
Redfern, O., Smith, G. B., Prytherch, D. R., Meredith, P., Inada-Kim, M., & Schmidt, P. E. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000003359 · Cited by 97
Harrison, G. A., Jacques, T. C., McLaws, M., & Kilborn, G. (2006) · Resuscitation · doi:10.1016/j.resuscitation.2006.05.008 · Cited by 94
Kellett, J. G., & Kim, A. (2011) · Resuscitation · doi:10.1016/j.resuscitation.2011.08.022 · Cited by 92
Ghanem‐Zoubi, N., Vardi, M., Laor, A., Weber, R., & Bitterman, H. (2011) · Critical Care · doi:10.1186/cc10102 · Cited by 87
Kyriacos, U., Jelsma, J. M., James, M. F., & Jordan, S. E. (2014) · PLoS ONE · doi:10.1371/journal.pone.0087073 · Cited by 84
Douw, G., Waal, G. H., van Zanten, A. R. H., van der Hoeven, J. G., & Schoonhoven, L. (2016) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2016.04.006 · Cited by 83
Gerry, S., Birks, J., Bonnici, T., Watkinson, P., Kirtley, S., & Collins, P. G. S. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-019268 · Cited by 82
Self-citationBartkowiak, B., Snyder, A. M., Benjamin, A., Schneider, A., Twu, N. M., Churpek, M. M., Roggin, K. K., & Edelson, D. P. (2018) · Annals of Surgery · doi:10.1097/sla.0000000000002665 · Cited by 81
Smith, G., Prytherch, D., Jarvis, S., Kovacs, C., Meredith, P., Schmidt, P., & Briggs, J. S. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000002000 · Cited by 77
Churpek, M. M., Snyder, A. M., Sokol, S. S., Pettit, N. N., & Edelson, D. P. (2017) · Critical Care Medicine · doi:10.1097/ccm.0000000000002648 · Cited by 75
Comparison of the National Early Warning Score in non‐elective medical and surgical patients
Kovacs, C., Jarvis, S. W., Prytherch, D., Meredith, P., Schmidt, P., Briggs, J., & Smith, G. B. (2016) · British Journal of Surgery · doi:10.1002/bjs.10267 · Cited by 72
Widely used track and trigger scores: Are they ready for automation in practice?
Brufau, S. R., Huddleston, J. M., Naessens, J. M., Johnson, M. G., Hickman, J. A., Morlan, B. W., Jensen, J. B., Caples, S. M., Elmer, J. L., Schmidt, J. A., et al. (2014) · Resuscitation · doi:10.1016/j.resuscitation.2013.12.017 · Cited by 67
Hodgson, L. E., Dimitrov, B. D., Congleton, J., Venn, R. M., Forni, L. G., & Roderick, P. J. (2016) · Thorax · doi:10.1136/thoraxjnl-2016-208436 · Cited by 67
Douw, G., Waal, G. H., van Zanten, A. R. H., van der Hoeven, J. G., & Schoonhoven, L. (2016) · Journal of Clinical Nursing · doi:10.1111/jocn.13648 · Cited by 67
Álvarez, C., Clark, C. A., Zhang, S., Halm, E. A., Shannon, J. J., Girod, C. E., Cooper, L., & Amarasingham, R. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-28 · Cited by 66
Lee, Y. S., Choi, J. W., Park, Y., Chung, C., Park, D., Lee, J. E., Lee, H. S., & Moon, J. Y. (2018) · Journal of Critical Care · doi:10.1016/j.jcrc.2018.07.011 · Cited by 62
Badriyah, T., Briggs, J. S., Meredith, P., Jarvis, S. W., Schmidt, P. E., Featherstone, P. I., Prytherch, D. R., & Smith, G. B. (2013) · Resuscitation · doi:10.1016/j.resuscitation.2013.12.011 · Cited by 61
Dziadzko, M. A., Novotny, P. J., Sloan, J. A., Gajic, O., Herasevich, V., Mirhaji, P., Wu, Y., & Gong, M. N. (2018) · Critical Care · doi:10.1186/s13054-018-2194-7 · Cited by 61
Jo, S., Jeong, T., Lee, J. B., Jin, Y., Yoon, J. C., & Park, B. (2015) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2015.12.067 · Cited by 54
A single-centre observational cohort study of admission National Early Warning Score (NEWS)
Abbott, T. E. F., Vaid, N., Ip, D., Cron, N., Wells, M., Torrance, H. D. T., & Emmanuel, J. J. (2015) · Resuscitation · doi:10.1016/j.resuscitation.2015.04.020 · Cited by 53
Linnen, D. T., Escobar, G. J., Hu, X., Scruth, E. A., Liu, V. X., & Stephens, C. E. (2019) · Journal of Hospital Medicine · doi:10.12788/jhm.3151 · Cited by 53
Spagnolli, W., Rigoni, M., Torri, E., Cozzio, S., Vettorato, E., & Nollo, G. (2017) · International Journal of Clinical Practice · doi:10.1111/ijcp.12934 · Cited by 52
Tirkkonen, J., Olkkola, K. T., Huhtala, H., Tenhunen, J. J., & Hoppu, S. (2014) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.12277 · Cited by 51
Kellett, J. G., Deane, B., & Gleeson, M. (2008) · Resuscitation · doi:10.1016/j.resuscitation.2008.02.011 · Cited by 51
Redfern, O., Pimentel, M. A. F., Prytherch, D. R., Meredith, P., Clifton, D. A., Tarassenko, L., Smith, G. B., & Watkinson, P. (2018) · Resuscitation · doi:10.1016/j.resuscitation.2018.09.021 · Cited by 49
Self-citationBrangan, E., Banks, J., Brant, H., Pullyblank, A. M., Roux, H. L., & Redwood, S. (2018) · BMJ Open · doi:10.1136/bmjopen-2018-022528 · Cited by 49
Opio, M. O., Nansubuga, G., & Kellett, J. G. (2013) · Resuscitation · doi:10.1016/j.resuscitation.2013.02.007 · Cited by 48
Wynants, L., Kent, D. M., Timmerman, D., Lundquist, C. M., & CALSTER, B. V. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0046-9 · Cited by 47
Early Deterioration Indicator: Data-driven approach to detecting deterioration in general ward
Ghosh, E., Eshelman, L. J., Yang, L., Carlson, E., & Lord, B. (2017) · Resuscitation · doi:10.1016/j.resuscitation.2017.10.026 · Cited by 44
SEND: a system for electronic notification and documentation of vital sign observations
Wong, D., Bonnici, T., Knight, J., Morgan, L. J., Coombes, P., & Watkinson, P. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-015-0186-y · Cited by 39
Self-citationWatkinson, P., Pimentel, M. A. F., Clifton, D. A., & Tarassenko, L. (2018) · Resuscitation · doi:10.1016/j.resuscitation.2018.06.003 · Cited by 39
Self-citationA Clinical Deterioration Prediction Tool for Internal Medicine Patients
Kirkland, L. L., Malinchoc, M., O’Byrne, M. M., Benson, J. T., Kashiwagi, D. T., Burton, M. C., Varkey, P., & Morgenthaler, T. I. (2012) · American Journal of Medical Quality · doi:10.1177/1062860612450459 · Cited by 37
Beane, A., Silva, A. P. D., Silva, N. D., Sujeewa, J. A., Rathnayake, R. M. D., Sigera, C., Athapattu, P. L., Mahipala, P. G., Rashan, A., Munasinghe, S. B., et al. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019387 · Cited by 37
Nickel, C. H., Kellett, J. G., Cooksley, T., Bingisser, R. M., Henriksen, D. P., & Brabrand, M. (2016) · Resuscitation · doi:10.1016/j.resuscitation.2016.06.012 · Cited by 35
ROBERTS, D. E., & Djärv, T. (2017) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2017.04.072 · Cited by 35
Abbott, T. E. F., Torrance, H. D. T., Cron, N., Vaid, N., & Emmanuel, J. J. (2016) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2016.06.014 · Cited by 32
Evaluating Automated Rules for Rapid Response System Alarm Triggers in Medical and Surgical Patients
Brufau, S. R., Morlan, B. W., Johnson, M. G., Hickman, J. A., Kirkland, L. L., Naessens, J. M., & Huddleston, J. M. (2017) · Journal of Hospital Medicine · doi:10.12788/jhm.2712 · Cited by 27
Early warning score predicts acute mortality in stroke patients
Liljehult, J., & Christensen, T. (2015) · Acta Neurologica Scandinavica · doi:10.1111/ane.12452 · Cited by 24
Cuthbertson, B. H., Boroujerdi, M. A., & Prescott, G. J. (2010) · The Journal of the Royal College of Physicians of Edinburgh · doi:10.4997/jrcpe.2010.105 · Cited by 24
Brabrand, M., Hallas, P., Hansen, S. N., Jensen, K. M., Madsen, J. L. B., & Posth, S. (2017) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2017.09.042 · Cited by 24
Faisal, M., Scally, A., Jackson, N. A., Richardson, D., Beatson, K., Howes, R., Speed, K., Menon, M., Daws, J., Dyson, J., et al. (2018) · BMJ Open · doi:10.1136/bmjopen-2018-022939 · Cited by 24
von Lilienfeld‐Toal, M., Midgley, K. J., Lieberbach, S., Barnard, L., Glasmacher, A. G., Gilleece, M. H., & Cook, G. (2007) · Transplantation and Cellular Therapy · doi:10.1016/j.bbmt.2006.12.455 · Cited by 23
Development of a Modified Early Warning Score Using the Electronic Medical Record
Albert, B. L., & Huesman, L. A. (2011) · Dimensions of Critical Care Nursing · doi:10.1097/dcc.0b013e318227761d · Cited by 23
Brabrand, M., & Henriksen, D. P. (2018) · Lung · doi:10.1007/s00408-018-0105-y · Cited by 22
Short National Early Warning Score — Developing a Modified Early Warning Score
Luís, L., & Nunes, C. (2017) · Australian Critical Care · doi:10.1016/j.aucc.2017.11.004 · Cited by 21
Kim, D., Jo, S., Lee, J. B., et al. (2018) · Clinical and Experimental Emergency Medicine · doi:10.15441/ceem.17.268 · Cited by 19
Hydes, T., Meredith, P., Schmidt, P. E., Smith, G. B., Prytherch, D. R., & Aspinall, R. (2017) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2017.12.035 · Cited by 18
Lagadec, D. L., Dwyer, T., & Browne, M. (2019) · Australian Critical Care · doi:10.1016/j.aucc.2019.03.001 · Cited by 18
Stræde, M., & Brabrand, M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105695 · Cited by 16
Modifications of the National Early Warning Score for patients with chronic respiratory disease
Pedersen, N. E., Rasmussen, L. S., Petersen, J. A., Gerds, T. A., Østergaard, D., & Lippert, A. M. (2017) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.13020 · Cited by 16
Stark, A. P., Maciel, R. C., Sheppard, W. L., Sacks, G. D., & Hines, O. J. (2015) · The American Surgeon · doi:10.1177/000313481508101001 · Cited by 14
Validation of the VitalPAC Early Warning Score at the Intermediate Care Unit
Plate, J. D. J., Peelen, L. M., Leenen, L. P. H., & Hietbrink, F. (2018) · World Journal of Critical Care Medicine · doi:10.5492/wjccm.v7.i3.39 · Cited by 14
Mulligan, A. (2010) · Intensive and Critical Care Nursing · doi:10.1016/j.iccn.2010.03.002 · Cited by 11
Kim, W., Lee, J., Lee, J., Jung, Y. K., Kim, H. J., Huh, J. W., Lim, C., Koh, Y., & Hong, S. (2017) · Journal of Critical Care · doi:10.1016/j.jcrc.2017.04.024 · Cited by 9
Smith, H. J., Pasko, D. N., Haygood, C. L. W., Boone, J. D., Harper, L. M., & Straughn, J. M. (2016) · Gynecologic Oncology · doi:10.1016/j.ygyno.2016.08.153 · Cited by 3
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4<361::aid-sim168>3.0.co;2-4
Cited by
The myth of generalisability in clinical research and machine learning in health care
Futoma, J., Simons, M., Panch, T., Doshi-Velez, F., & Celi, L. A. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30186-2 · Cited by 467
The Importance of Respiratory Rate Monitoring: From Healthcare to Sport and Exercise
Nicolò, A., Massaroni, C., Schena, E., & Sacchetti, M. (2020) · Sensors · doi:10.3390/s20216396 · Cited by 440
Machine learning in clinical decision making
Adlung, L., Cohen, Y., Mor, U., & Elinav, E. (2021) · Med · doi:10.1016/j.medj.2021.04.006 · Cited by 282
Pimentel, M. A. F., Redfern, O. C., Malycha, J., Meredith, P., Prytherch, D., Briggs, J., Young, J. D., Clifton, D. A., Tarassenko, L., & Watkinson, P. J. (2021) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.202007-2700oc · Cited by 84
Self-citationExternal validation of AI-based scoring systems in the ICU: a systematic review and meta-analysis
Rockenschaub, P., Akay, E. M., Carlisle, B. G., Hilbert, A., Wendland, J., Meyer-Eschenbach, F., Näher, A.-F., Frey, D., & Madai, V. I. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02830-7 · Cited by 58
Metadata
- Authors
- Gerry, S., Bonnici, T., Birks, J., Kirtley, S., Virdee, P. S., Watkinson, P. J., & Collins, G. S.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2020-05-20
- Volume
- 369
- Pages
- m1501
- ISSN
- 1756-1833 (ISSN-L)
- OpenAlex
- W3013577764
- Cited by
- 383
- References
- 133
- FWCI
- 13.87
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.