The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Automated Identification of Adults at Risk for In-Hospital Clinical Deterioration
Open accessAbstract
BACKGROUND: Hospitalized adults whose condition deteriorates while they are in wards (outside the intensive care unit [ICU]) have considerable morbidity and mortality. Early identification of patients at risk for clinical deterioration has relied on manually calculated scores. Outcomes after an automated detection of impending clinical deterioration have not been widely reported. METHODS: On the basis of a validated model that uses information from electronic medical records to identify hospitalized patients at high risk for clinical deterioration (which permits automated, real-time risk-score calculation), we developed an intervention program involving remote monitoring by nurses who reviewed records of patients who had been identified as being at high risk; results of this monitoring were then communicated to rapid-response teams at hospitals. We compared outcomes (including the primary outcome, mortality within 30 days after an alert) among hospitalized patients (excluding those in the ICU) whose condition reached the alert threshold at hospitals where the system was operational (intervention sites, where alerts led to a clinical response) with outcomes among patients at hospitals where the system had not yet been deployed (comparison sites, where a patient's condition would have triggered a clinical response after an alert had the system been operational). Multivariate analyses adjusted for demographic characteristics, severity of illness, and burden of coexisting conditions. RESULTS: The program was deployed in a staggered fashion at 19 hospitals between August 1, 2016, and February 28, 2019. We identified 548,838 non-ICU hospitalizations involving 326,816 patients. A total of 43,949 hospitalizations (involving 35,669 patients) involved a patient whose condition reached the alert threshold; 15,487 hospitalizations were included in the intervention cohort, and 28,462 hospitalizations in the comparison cohort. Mortality within 30 days after an alert was lower in the intervention cohort than in the comparison cohort (adjusted relative risk, 0.84, 95% confidence interval, 0.78 to 0.90; P<0.001). CONCLUSIONS: The use of an automated predictive model to identify high-risk patients for whom interventions by rapid-response teams could be implemented was associated with decreased mortality. (Funded by the Gordon and Betty Moore Foundation and others.).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Translation of these approaches in clinical intervention studies has improved outcomes [16, 34, 35].”
From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [34]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Pragmatic randomized trials and quasi-experimental designs offer advantages over pre-post designs or comparisons against historical controls [34, 52].”
From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [34]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Other EWSs have influenced in-hospital mortality, sepsis or LOS, but not across the breadth of outcomes we observed4,5,26,27.”
From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, ease of use and integration into the workflow have been noted as important factors influencing adoption 26,28,47,48 .”
From Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing (2022) · cites this paper as [26]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While there are other existing multifactorial machine learning clinical decision support implementations that share similarities to ours27–29, the CONCERN intervention is distinct in its modeling approach by using metadata of nursing documentation as early predictors of patient deterioration in acute and intensive care settings, being fully integrated as a real-time EWS into the EHR and undergoing”
From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases
DEYO, R. (1992) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(92)90133-8 · Cited by 10,452
Hosmer, D. W., Lemeshow, S. A., & May, S. J. (2008) · Wiley series in probability and statistics · doi:10.1002/9780470258019 · Cited by 3,851
Estimating the Relative Risk in Cohort Studies and Clinical Trials of Common Outcomes
McNutt, L. (2003) · American Journal of Epidemiology · doi:10.1093/aje/kwg074 · Cited by 2,098
Practical recommendations for reporting F ine‐ G ray model analyses for competing risk data
Austin, P. C., & Fine, J. P. (2017) · Statistics in Medicine · doi:10.1002/sim.7501 · Cited by 1,299
Design and analysis of stepped wedge cluster randomized trials.
Hussey, M. A., & Hughes, J. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.05.007 · Cited by 1,297
Smith, G. B., Prytherch, D. R., Meredith, P., Schmidt, P. E., & Featherstone, P. I. (2013) · Resuscitation · doi:10.1016/j.resuscitation.2012.12.016 · Cited by 1,077
Review of statistical methods for analysing healthcare resources and costs
Mihaylova, B., Briggs, A., O'Hagan, A., & Thompson, S. G. (2011) · Health Economics · doi:10.1002/hec.1653 · Cited by 767
Multilevel modelling of medical data
Goldstein, H., Browne, W. J., & Rasbash, J. (2002) · Statistics in Medicine · doi:10.1002/sim.1264 · Cited by 394
Priestley, G., Watson, W., Rashidian, A., Mozley, C. G., Russell, D., Wilson, J., Cope, J., Hart, D., Kay, D., Cowley, K., et al. (2004) · Intensive Care Medicine · doi:10.1007/s00134-004-2268-7 · Cited by 374
Escobar, G. J., Greene, J., Scheirer, P., Gardner, M. N., Draper, D., & Kipnis, P. (2008) · Medical Care · doi:10.1097/mlr.0b013e3181589bb6 · Cited by 315
Self-citationMulticenter Development and Validation of a Risk Stratification Tool for Ward Patients
Churpek, M. M., Yuen, T. C., Winslow, C. J., Robicsek, A. A., Meltzer, D. O., Gibbons, R. D., & Edelson, D. P. (2014) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201406-1022oc · Cited by 263
Creating a Learning Health System through Rapid-Cycle, Randomized Testing
Horwitz, L. I., Kuznetsova, M., & Jones, S. A. (2019) · New England Journal of Medicine · doi:10.1056/nejmsb1900856 · Cited by 250
Rothman, M., Rothman, S. I., & Beals, J. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.06.011 · Cited by 243
Escobar, G. J., Gardner, M. N., Greene, J., Draper, D., & Kipnis, P. (2013) · Medical Care · doi:10.1097/mlr.0b013e3182881c8e · Cited by 199
Self-citationEffect of an automated notification system for deteriorating ward patients on clinical outcomes
Subbe, C. P., Duller, B., & Bellomo, R. (2017) · Critical Care · doi:10.1186/s13054-017-1635-z · Cited by 190
Escobar, G. J., LaGuardia, J. C., Turk, B. J., Ragins, A. I., Kipnis, P., & Draper, D. (2012) · Journal of Hospital Medicine · doi:10.1002/jhm.1929 · Cited by 185
Self-citationComparing alternative models: log vs Cox proportional hazard?
Basu, A., Manning, W., & Mullahy, J. (2004) · Health Economics · doi:10.1002/hec.852 · Cited by 183
Kipnis, P., Turk, B. J., Wulf, D. A., LaGuardia, J. C., Liu, V. X., Churpek, M. M., Brufau, S. R., & Escobar, G. J. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.09.013 · Cited by 167
Self-citationMinimal Impact of Implemented Early Warning Score and Best Practice Alert for Patient Deterioration*
Bedoya, A. D., Clement, M. E., Phelan, M., Steorts, R. C., O’Brien, C., & Goldstein, B. A. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003439 · Cited by 162
Churpek, M. M., Wendlandt, B., Zadravecz, F. J., Adhikari, R., Winslow, C., & Edelson, D. P. (2016) · Journal of Hospital Medicine · doi:10.1002/jhm.2630 · Cited by 143
Liu, V. X., Kipnis, P., Rizk, N. W., & Escobar, G. J. (2011) · Journal of Hospital Medicine · doi:10.1002/jhm.964 · Cited by 137
Self-citationEscobar, G. J., Greene, J., Gardner, M. N., Marelich, G. P., Quick, B., & Kipnis, P. (2010) · Journal of Hospital Medicine · doi:10.1002/jhm.817 · Cited by 113
Self-citationAssessment of Machine Learning vs Standard Prediction Rules for Predicting Hospital Readmissions
Morgan, D. J., Bame, B., Zimand, P., Dooley, P., Thom, K. A., Harris, A. D., Bentzen, S. M., Ettinger, W. H., Garrett-Ray, S., Tracy, J. K., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0348 · Cited by 111
Escobar, G. J., Turk, B. J., Ragins, A. I., Ha, J., Hoberman, B., LeVine, S. M., Ballesca, M. A., Liu, V. X., & Kipnis, P. (2016) · Journal of Hospital Medicine · doi:10.1002/jhm.2652 · Cited by 98
Self-citationDavey, C., Hargreaves, J., Thompson, J. A., Copas, A. J., Beard, E., Lewis, J. J., & Fielding, K. (2015) · Trials · doi:10.1186/s13063-015-0838-3 · Cited by 98
Kollef, M. H., Chen, Y., Heard, K. M., LaRossa, G. N., Lu, C., Martin, N. R., Martin, N., Micek, S. T., & Bailey, T. C. (2014) · Journal of Hospital Medicine · doi:10.1002/jhm.2193 · Cited by 96
Delgado, M. K., Liu, V. X., Pines, J. M., Kipnis, P., Gardner, M. N., & Escobar, G. J. (2012) · Journal of Hospital Medicine · doi:10.1002/jhm.1979 · Cited by 94
Self-citationBapoje, S. R., Gaudiani, J. L., Narayanan, V., & Albert, R. K. (2010) · Journal of Hospital Medicine · doi:10.1002/jhm.812 · Cited by 89
Nonelective Rehospitalizations and Postdischarge Mortality
Escobar, G. J., Ragins, A. I., Scheirer, P., Liu, V. X., Robles, J. A. M., & Kipnis, P. (2015) · Medical Care · doi:10.1097/mlr.0000000000000435 · Cited by 74
Self-citationImpact of a standardized rapid response system on outcomes in a large healthcare jurisdiction
Chen, J., Ou, L., Flabouris, A., Hillman, K., Bellomo, R., & Parr, M. J. (2016) · Resuscitation · doi:10.1016/j.resuscitation.2016.07.240 · Cited by 68
Automated detection of physiologic deterioration in hospitalized patients
Evans, R. S., Kuttler, K. G., Simpson, K. J., HOWE, S. E., Crossno, P. F., Johnson, K. V., Schreiner, M. N., Lloyd, J. F., Tettelbach, W. H., Keddington, R. K., et al. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2014-002816 · Cited by 59
Linnen, D. T., Escobar, G. J., Hu, X., Scruth, E. A., Liu, V. X., & Stephens, C. E. (2019) · Journal of Hospital Medicine · doi:10.12788/jhm.3151 · Cited by 52
Self-citationAssociation Between In-Hospital Critical Illness Events and Outcomes in Patients on the Same Ward
Volchenboum, S. L., Mayampurath, A. M., Göksu-Gürsoy, G., Edelson, D. P., Howell, M. D., & Churpek, M. M. (2016) · JAMA · doi:10.1001/jama.2016.15505 · Cited by 43
Multiyear Rehospitalization Rates and Hospital Outcomes in an Integrated Health Care System
Escobar, G. J., Plimier, C., Greene, J., Liu, V. X., & Kipnis, P. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.16769 · Cited by 36
Self-citationIncorporating an Early Detection System Into Routine Clinical Practice in Two Community Hospitals
Dummett, B. A., Adams, C. L., Scruth, E. A., Liu, V. X., Guo, M., & Escobar, G. J. (2016) · Journal of Hospital Medicine · doi:10.1002/jhm.2661 · Cited by 33
Self-citationEarly detection, prevention, and mitigation of critical illness outside intensive care settings
Escobar, G. J., & Dellinger, R. P. (2016) · Journal of Hospital Medicine · doi:10.1002/jhm.2653 · Cited by 33
Self-citationWhat Do We Do After the Pilot Is Done? Implementation of a Hospital Early Warning System at Scale
Paulson, S. S., Dummett, B. A., Green, J., Scruth, E. A., Reyes, V., & Escobar, G. J. (2020) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/j.jcjq.2020.01.003 · Cited by 24
Self-citationGranich, R., Sutton, Z., Kim, Y. S., Anderson, M. L., Wood, H. A., Scharf, J., Ragins, A. I., & Escobar, G. (2016) · Journal of Hospital Medicine · doi:10.1002/jhm.2660 · Cited by 15
Self-citation
Research works printed but not linked: 5
- Escobar GJ, Liu V, Kim YS, et al. Early detection of impending deterioration outside the ICU: a difference-in-differences (DiD) study. Am J Respir Crit Care Med 2016;93:A7614-A7614. abstract.
- Harrell F Jr. Regression modeling strategies with applications to linear models logistic regression and survival analysis. New York: Springer-Verlag 2010.
- Kalbfleisch JD Prentice RL. The statistical analysis of failure time data. 2nd ed. New York: John Wiley 2011.
- Harrell FE Jr. Semiparametric modeling of health care cost and resource utilization. Presented at the 24th Annual Midwest Biopharmaceutical Statistics Workshop Muncie IN May 21–23 2001.
- Collett D. Modelling survival data in medical research. 3rd ed. Boca Raton FL: Chapman and Hall/CRC Press 2014.
Cited by
Sorted by citations
Adams, R., Henry, K. E., Sridharan, A., Soleimani, H., Zhan, A., Rawat, N., Johnson, L., Hager, D. N., Cosgrove, S. E., Markowski, A., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01894-0 · Cited by 365
Armoundas, A. A., Narayan, S. M., Arnett, D. K., Spector-Bagdady, K., Bennett, D. A., Celi, L. A., Friedman, P. A., Gollob, M. H., Hall, J. L., Kwitek, A. E., et al. (2024) · Circulation · doi:10.1161/cir.0000000000001201 · Cited by 347
European Resuscitation Council Guidelines 2025 Adult Advanced Life Support
Soar, J., Böttiger, B. W., Carli, P. A., Carmona, F., Cimpoesu, D. C., Cole, G., Couper, K., D’Arrigo, S., Deakin, C. D., Ek, J. E., et al. (2025) · Resuscitation · doi:10.1016/j.resuscitation.2025.110769 · Cited by 192
Artificial Intelligence in U.S. Health Care Delivery
Sahni, N. R., & Carrus, B. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2204673 · Cited by 175
Use of artificial intelligence in critical care: opportunities and obstacles
Pinsky, M. R., Bedoya, A., Bihorac, A., Celi, L., Churpek, M., Economou-Zavlanos, N. J., Elbers, P., Saria, S., Liu, V., Lyons, P. G., et al. (2024) · Critical Care · doi:10.1186/s13054-024-04860-z · Cited by 160
“Translation of these approaches in clinical intervention studies has improved outcomes [16, 34, 35].”
BackgroundSelf-citation
Metadata
- Authors
- Escobar, G. J., Liu, V. X., Schuler, A., Lawson, B., Greene, J. D., & Kipnis, P.
- Affiliations
- Kaiser Permanente (US)
- ORCID
- Venue
- New England Journal of Medicine
- Publisher
- Massachusetts Medical Society
- Published
- 2020-11-12
- Volume
- 383
- Issue
- 20
- Pages
- 1951-1960
- ISSN
- 0028-4793 (ISSN-L)
- 1533-4406
- PMC
- PMC7787261
- OpenAlex
- W3106297998
- Cited by
- 314
- References
- 39
- Citations by year
- 2026: 52 · 2025: 63 · 2024: 51 · 2023: 47 · 2022: 56 · 2021: 42 · 2020: 3
- FWCI
- 33.81
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- New England Journal of Medicine (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Sidney Garfield Memorial Fund
- Gordon and Betty Moore Foundation (2663, 2663.01)
- The Permanente Medical Group, Inc.
- Kaiser Foundation Hospitals, Inc.
- National Institutes of Health (K23GM112018, R35GM128672)
- Agency for Healthcare Research and Quality (1R01HS018480-01)
- Topics
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Sepsis Diagnosis and Treatment
- Keywords
- clinical deterioration, early warning systems, rapid response teams, remote patient monitoring, 30-day mortality, electronic health records, hospitalized adults
- MeSH terms
- Alert Fatigue, Health Personnel, Laboratory Critical Values, Clinical Deterioration, Adult, Aged, Automation, Female, Hospitalization, Humans, Length of Stay, Male, Middle Aged, Models, Theoretical, Nursing Staff, Hospital, Patient Readmission, Telemetry, Multivariate Analysis, Hospital Mortality, Risk Assessment, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
