Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial

Classified as probably not a substantial claim. May combine two claims.

Randomization was performed at the unit level and stratified by site before trial initiation.

“Randomization was performed at the unit level and stratified by site before trial initiation53.”

• cites [53] Shcherbatykh, I., Holbrook, A., Thabane, L., Dolovich, L. & COMPETE III investigators. Methodologic issues in health informatics trials: the complexities of complex interventions. J. Am. Med. Inform. Assoc. 15, 575–580 (2008). sourceDisputed

  In “Blinding of subjects” (page unavailable), the paper explicitly says the COMPETE III trialists randomized individual patients, not units. Its “Choice of randomization” discussion treats unit-level allocation as an option, but the passages do not report stratification by site before the trial. (LLM, conf=0.85)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

May combine two claims.

The CONCERN intervention is distinct from other multifactorial machine learning clinical decision support implementations in its modeling approach, integration, and evaluation.

“While there are other existing multifactorial machine learning clinical decision support implementations that share similarities to ours27–29, the CONCERN intervention is distinct in its modeling approach by using metadata of nursing documentation as early predictors of patient deterioration in acute and intensive care settings, being fully integrated as a real-time EWS into the EHR and undergoing”

• cites [27] Escobar, G. J. et al. Automated identification of adults at risk for in-hospital clinical deterioration. N. Engl. J. Med. 383, 1951–1960 (2020). sourceSupported
• cites [28] Levin, M. A. et al. Real-time machine learning alerts to prevent escalation of care: a nonrandomized clustered pragmatic clinical trial. Crit. Care Med. 52, 1007–1020 (2024). sourceNot checked — no full text of the source
• cites [29] Churpek, M. M. et al. Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit. Care Med. 44, 368–374 (2016). sourcePartly supported

  The Methods (Predictor variables) and Discussion describe a multifactorial machine-learning model for predicting ward deterioration, supporting the citation as an example of a similar modeling approach. But the Discussion says the models *can be implemented* with an EHR; it does not report an implemented clinical decision-support system. The claim about how CONCERN differs is the citing authors’ own comparison, not a finding attributed to this paper. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

May combine two claims.

Most published studies target a particular condition or hospital setting, and only a minority of EWSs have been evaluated in randomized controlled trials.

“Most published studies target a particular condition (for example, sepsis) or hospital setting (for example, ICU)4 and only a minority of EWSs have been evaluated in randomized controlled trials4,6.”

• cites [4] Wan, Y.-K. J. et al. Information displays for automated surveillance algorithms of in-hospital patient deterioration: a scoping review. J. Am. Med. Inform. Assoc. 31, 256–273 (2023). sourceSupported
• cites [6] Lee, T. C., Shah, N. U., Haack, A. & Baxter, S. L. Clinical implementation of predictive models embedded within electronic health record systems: a systematic review. Informatics (MDPI) 7, 25 (2020). sourcePartly supported

  The Results (Study Characteristics; page not provided) describe studies focused on clinical domains including sepsis and report that 28 of 44 centered on inpatient populations. The Results (Quality Assessment; page not provided) find that randomized controlled trials were a minority (5 of 44). However, that figure covers studies of EHR-embedded predictive models generally, not early warning systems (EWSs) specifically, and the copy does not establish that most studies targeted a particular hospital setting such as an ICU. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

The team has published on statistically significant signals detected in nursing data using this metadata modeling approach since 2012, demonstrating its resilience as EHR functionality and documentation policies change over time and across multiple sites.

“Our team has published on statistically significant signals detected in nursing data using this metadata modeling approach since 2012 (refs. 13,21), demonstrating its resilience, first, as EHR functionality and documentation policies change over time and across multiple sites.”

• cites [13] Collins, S. A. et al. Relationship between nursing documentation and mortality. Am. J. Crit. Care. 22, 306–313 (2013). sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [21] Collins, S. A. & Vawdrey, D. K. “Reading between the lines” of flowsheet data: nurses’ optional documentation associated with cardiac arrest outcomes. Appl. Nurs. Res. 25, 251–257 (2012). sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

The trial findings are reported according to CONSORT guidelines and extensions for AI, cluster-randomized trials, and pragmatic trials.

“We report our trial findings according to the CONSORT guidelines31 and extensions for AI32, cluster-randomized trials33 and pragmatic trials34 (Supplementary Table 1).”

• cites [31] Moher, D. et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. Brit. Med. J. 340, c869 (2010). sourceSupported
• cites [32] Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J. & Denniston, A. K. on behalf of SPIRIT-AI and CONSORT-AI Working Group. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Lancet Digit. Health 2, e537–e548 (2020). sourceSupported
• cites [33] Campbell, M. K., Piaggio, G., Elbourne, D. R. & Altman, D. G. CONSORT Group. Consort 2010 statement: extension to cluster randomised trials. Brit. Med. J. 345, e5661 (2012). sourceSupported
• cites [34] Zwarenstein, M. et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. Brit. Med. J. 337, a2390 (2008). sourceSupported

Classified as probably not a substantial claim.

Evidence for escalation of medical interventions based on nurses' concerns has long remained classified as level 5 evidence (expert opinion).

“Evidence for escalation of medical interventions based on nurses’ concerns has long remained classified as level 5 evidence (expert opinion)15–20.”

• cites [15] Tokareva, I. & Romano, P. Failure to rescue. PSNet https://psnet.ahrq.gov/primer/failure-rescue (2025).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [16] Odell, M., Victor, C. & Oliver, D. Nurses’ role in detecting deterioration in ward patients: systematic literature review. J. Adv. Nurs. 65, 1992–2006 (2009). sourceNot checked — no full text of the source
• cites [17] Burns, P. B., Rohrich, R. J. & Chung, K. C. The levels of evidence and their role in evidence-based medicine. Plast. Reconstr. Surg. 128, 305–310 (2011). sourcePartly supported

  The History of Levels of Evidence section (page unavailable) places expert opinion at the lowest level of evidence, and Clinical examples using levels of evidence (page unavailable) describes some low-level evidence as level 4 or 5. The paper does not discuss nurses’ concerns, escalation of medical interventions, or whether evidence for that practice has long remained at level 5. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [18] Agency for Healthcare Research and Quality. Rapid Response Systems https://psnet.ahrq.gov/primer/rapid-response-systems (2019).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [19] Cioffi, J. Recognition of patients who require emergency assistance: a descriptive study. Heart Lung 29, 262–268 (2000). sourceNot checked — no full text of the source
• cites [20] Jones, L., King, L. & Wilson, C. A literature review: factors that impact on nurses’ effective use of the Medical Emergency Team (MET). J. Clin. Nurs. 18, 3379–3390 (2009). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A 2022 systematic review identified only 41 randomized controlled trials for machine learning interventions in health care, and none adhered to all CONSORT-AI standards.

“Additionally, a 2022 systematic review identified only 41 randomized controlled trials for machine learning interventions in health care, and none adhered to all Consolidated Standards of Reporting Trials-Artificial Intelligence (CONSORT-AI) standards30.”

• cites [30] Plana, D. et al. Randomized clinical trials of machine learning interventions in health care: a systematic review. JAMA Netw. Open 5, e2233946 (2022). sourceSupported

Classified as probably not a substantial claim. May combine two claims.

Earlier predictions provide clinicians with greater lead time for action, ultimately enabling the care team to identify patients who may be entering a risky state with enough time to intervene.

“Earlier predictions provide clinicians with greater lead time for action24, ultimately enabling the care team to identify patients who may be entering a risky state with enough time to intervene10.”

• cites [24] Singh, K. et al. Evaluating a widely implemented proprietary deterioration index model among hospitalized patients with COVID-19. Ann. Am. Thorac. Soc. 18, 1129–1137 (2021). sourceSupported
• cites [10] Rossetti, S. et al. Healthcare process modeling to phenotype clinician behaviors for exploiting the signal gain of clinical expertise (HPM–ExpertSignals): development and evaluation of a conceptual framework. J. Am. Med. Inform. Assoc. 28, 1242–1251 (2021). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

A quality assessment framework was applied to all variables to address well-known issues when reusing EHR data for secondary analyses.

“Fifth, there are well-known issues when reusing EHR data for secondary analyses49. To address this, we applied a quality assessment framework to all variables50.”

• cites [49] Lewis, A. E. et al. Electronic health record data quality assessment and tools: a systematic review. J. Am. Med. Inform. Assoc. 30, 1730–1740 (2023). sourceSupported
• cites [50] Weiskopf, N. G., Bakken, S., Hripcsak, G. & Weng, C. A data quality assessment guideline for electronic health record data reuse. EGEMS (Wash DC) 5, 14 (2017). sourceSupported

May combine two claims.

CONCERN EWS overcomes the limitations of physiological measurements by leveraging nursing surveillance patterns and an existing expert knowledge base.

“By leveraging nursing surveillance patterns rather than physiological measures10,44, CONCERN EWS overcomes the limitations of physiological measurements8 and leverages an existing expert knowledge base: nurses’ autonomous decisions to take greater action and document more observations or notes than mandated by policy in direct response to concerning changes in a patient’s clinical state10,13.”

• cites [10] Rossetti, S. et al. Healthcare process modeling to phenotype clinician behaviors for exploiting the signal gain of clinical expertise (HPM–ExpertSignals): development and evaluation of a conceptual framework. J. Am. Med. Inform. Assoc. 28, 1242–1251 (2021). sourceSupported
• cites [44] Pivovarov, R., Albers, D. J., Sepulveda, J. L. & Elhadad, N. Identifying and mitigating biases in EHR laboratory tests. J. Biomed. Inform. 51, 24–34 (2014). sourcePartly supported

  The Introduction says laboratory-test values alone cannot distinguish some disease states, and the Results (section 3.1) find additional information in test-ordering patterns. This supports a limitation of laboratory values, but the citation is attached to the broader phrase “limitations of physiological measurements”; the paper does not establish that broader claim. The sentence’s statements about CONCERN EWS and nursing surveillance are not attributed to this source. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [8] Fu, L. et al. Development and validation of early warning score system: a systematic literature review. J. Biomed. Inform. 105, 103410 (2020). sourcePartly supported

  The Discussion sections on data-driven EWS derivation and potential bias in EHR data describe limitations of individual physiological observations and say nursing documentation can help predict patient changes. They do not establish that CONCERN EWS overcomes those limitations or that it uses nurses’ autonomous decisions as an expert knowledge base. No page numbers are available in our copy. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [13] Collins, S. A. et al. Relationship between nursing documentation and mortality. Am. J. Crit. Care. 22, 306–313 (2013). sourcePartly supported

  The Discussion (page not provided) says increased optional nursing documentation may signal nurses’ concern and could help predict deterioration or death. It also says further work is needed to determine whether documented concerns outperform existing risk scores. The paper does not evaluate CONCERN EWS or establish that it overcomes the limitations of physiological measurements. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

May combine two claims.

Early ICU transfer is a window of critical opportunity for timely clinical interventions that alter the trajectory of a patient's clinical progression, prevent adverse outcomes and improve survival.

“Early ICU transfer has been shown to be a ‘window of critical opportunity’ for timely clinical interventions that alter the trajectory of a patient’s clinical progression, prevent adverse outcomes and improve survival35–39.”

• cites [35] Kiekkas, P. et al. Delayed admission to the intensive care unit and mortality of critically ill adults: systematic review and meta-analysis. Biomed. Res. Int. 2022, 4083494 (2022). sourcePartly supported

  The Results (section 3.6; page unavailable) report that delayed ICU admission was associated with higher mortality, supporting the survival-related part of the claim. But the paper does not show that early transfer itself enables interventions that change a patient’s trajectory or prevent adverse outcomes. The Discussion (section 4.3; page unavailable) says timely critical care should not depend on ICU admission. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [36] Churpek, M. M. et al. Association between intensive care unit transfer delay and hospital mortality: A multicenter investigation. J. Hosp. Med. 11, 757–762 (2016). sourcePartly supported

  The Introduction says early ICU transfer for treatment may improve outcomes, and the Discussion reports that delayed transfer was associated with higher hospital mortality. But the Discussion also says this observational study cannot estimate the true impact of timely transfer. It does not establish that early transfer enables interventions that alter a patient's trajectory or prevent adverse outcomes. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [37] Solomon, R. S., Corwin, G. S., Barclay, D. C., Quddusi, S. F. & Dannenberg, M. D. Effectiveness of rapid response teams on rates of in-hospital cardiopulmonary arrest and mortality: A systematic review and meta-analysis. J. Hosp. Med. 11, 438–445 (2016). sourceNot checked — no full text of the source
• cites [38] Hu, W., Chan, C. W., Zubizarreta, J. R. & Escobar, G. J. An examination of early transfers to the ICU based on a physiologic risk score. Manuf. Serv. Oper. Manag. 20, 531–549 (2018). sourceNot checked — no full text of the source
• cites [39] Grieve, R. et al. Analysis of benefit of intensive care unit transfer for deteriorating ward patients: a patient-centered approach to clinical evaluation. JAMA Netw. Open 2, e187704 (2019). sourcePartly supported

  The Results report an estimated reduction in 28-day mortality after ICU transfer, with greater benefit for some patient groups, but the overall estimate has a wide confidence interval. The Introduction explicitly says this study assessed ICU transfer itself, unlike an earlier study focused on transfer within four hours. It does not establish that early transfer provides a critical window for interventions that alter clinical progression or prevent adverse outcomes. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors are not aware of any other EWS decision-support implementations that use a similar multifactorial, advanced computational and explainable modeling approach.

“We are not aware of any other EWS decision-support implementations that use a similar multifactorial, advanced computational and explainable modeling approach10.”

• cites [10] Rossetti, S. et al. Healthcare process modeling to phenotype clinician behaviors for exploiting the signal gain of clinical expertise (HPM–ExpertSignals): development and evaluation of a conceptual framework. J. Am. Med. Inform. Assoc. 28, 1242–1251 (2021). sourceSupported

Classified as probably not a substantial claim.

Secondary outcomes were cardiopulmonary arrest, sepsis, unanticipated ICU transfer, and 30-day hospital readmission.

“Secondary outcomes were cardiopulmonary arrest, sepsis51, unanticipated ICU transfer and 30-day hospital readmission52.”

• cites [51] Singer, M. et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 315, 801–810 (2016). sourceSupported
• cites [52] Yale New Haven Health Services Corporation – Center for Outcomes Research and Evaluation (YNHHSC/CORE). 2023 Hospital-Wide Readmission Measure Updates and Specifications Report – Version 12.0. (2023); https://qpp.cms.gov/resources/document/fab5e228-1667-4e8f-9b49-ea04fb3cfd81Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

CONCERN EWS's methodological choices, including robust modeling of temporal data patterns and use of nursing assessment and observational data, allowed it to have accurate and robust predictive power and sufficient lead time to alter a patient's clinical trajectory.

“Several methodological choices allowed CONCERN EWS to have accurate and robust predictive power and sufficient lead time to alter a patient’s clinical trajectory as we observed in this trial, including: (1) robust modeling of temporal data patterns40–42 (for example, time of day, day of week and patient hospital day) for health care processes (for example, nurses’ surveillance and medication”

• cites [40] Hripcsak, G. & Albers, D. J. Next-generation phenotyping of electronic health records. J. Am. Med. Inform. Assoc. 20, 117–121 (2013). sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [41] Albers, D. J. & Hripcsak, G. Using time-delayed mutual information to discover and interpret temporal correlation structure in complex populations. Chaos 22, 13111 (2012). sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [42] Albers, D. J. & Hripcsak, G. A statistical dynamics approach to the study of human health data: resolving population scale diurnal variation in laboratory data. Phys. Lett. A 374, 1159–1164 (2010). sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [10] Rossetti, S. et al. Healthcare process modeling to phenotype clinician behaviors for exploiting the signal gain of clinical expertise (HPM–ExpertSignals): development and evaluation of a conceptual framework. J. Am. Med. Inform. Assoc. 28, 1242–1251 (2021). sourceSupported
• cites [43] Pivovarov, R., Albers, D. J., Hripcsak, G., Sepulveda, J. L. & Elhadad, N. Temporal trends of hemoglobin A1c testing. J. Am. Med. Inform. Assoc. 21, 1038–1044 (2014). sourcePartly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [44] Pivovarov, R., Albers, D. J., Sepulveda, J. L. & Elhadad, N. Identifying and mitigating biases in EHR laboratory tests. J. Biomed. Inform. 51, 24–34 (2014). sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [13] Collins, S. A. et al. Relationship between nursing documentation and mortality. Am. J. Crit. Care. 22, 306–313 (2013). sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [21] Collins, S. A. & Vawdrey, D. K. “Reading between the lines” of flowsheet data: nurses’ optional documentation associated with cardiac arrest outcomes. Appl. Nurs. Res. 25, 251–257 (2012). sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

May combine two claims.

CONCERN EWS was developed by an interdisciplinary team using rigorous methods for translating a predictive model to a clinical setting, including input on health care process effects, integration with workflows, transparency and explainability, and robust evaluation for external validity and model fairness.

“CONCERN EWS was developed by an interdisciplinary team that employed rigorous methods for translating a predictive model to a clinical setting, including nurse and prescribing-provider input on health care process effects and real-time data availability, integration with existing clinical workflows, transparency and explainability to gain trust10,45, as well as robust evaluation for external”

• cites [10] Rossetti, S. et al. Healthcare process modeling to phenotype clinician behaviors for exploiting the signal gain of clinical expertise (HPM–ExpertSignals): development and evaluation of a conceptual framework. J. Am. Med. Inform. Assoc. 28, 1242–1251 (2021). sourcePartly supported

  The Materials and Methods describe iterative model refinement with clinical experts, simulation feedback from nurses and physicians, and a user-centered tool designed not to require new nursing workflows. They also report performance evaluation and corrections for demographic biases. The paper does not establish the claimed robust evaluation for external validity or model fairness, or specifically document transparency and explainability measures to gain trust; it describes broader applicability as future work (Discussion; page numbers unavailable). (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites [45] Schwartz, J. M. et al. Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: qualitative descriptive study. JMIR Hum Factors 9, e33960 (2022). sourcePartly supported

  Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [46] Wong, A. et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern. Med. 181, 1065–1070 (2021). sourceCould not verify

  The Introduction and Results describe an external validation of the Epic Sepsis Model, not the development or evaluation of CONCERN EWS. Searches for CONCERN, the claimed interdisciplinary development practices, and model fairness found no passage addressing those claims. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

May combine two claims.

The models achieved more than 97% average accuracy and 95% average area under the curve.

“with more than 97% average accuracy and 95% average area under the curve (average performance reported for our ensemble models22) (Extended Data Table 5 and Supplementary Methods 2)”

• cites [22] Opitz, D. & Maclin, R. Popular ensemble methods: an empirical study. J. Artif. Intell. Res. 11, 169–198 (1999). sourceCould not verify

  The Results (p. 9) report test-set error rates for ensemble methods, and Section 3.5 (p. 13) discusses average error across 23 data sets. Neither passage reports more than 97% average accuracy or 95% average area under the curve; searches for those figures and for AUC found no supporting passage in our copy. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: The first occurrence of cardiopulmonary arrest, rapid response, sepsis, transfer to ICU, and death are valid proxy measures for patient deterioration.

May combine two claims.

CONCERN EWS comprises a predictive model that uses an ensemble machine learning approach to process nursing surveillance patterns in the EHR, predicts clinical deterioration within the subsequent 24-hour period, and displays the risk score as a green, yellow or red icon on the EHR patient list.

“CONCERN EWS comprises a predictive model that uses an ensemble22 machine learning approach to process nursing surveillance patterns in the EHR, predicts clinical deterioration within the subsequent 24-hour period and displays the CONCERN EWS risk score as a green, yellow or red icon (non-interruptive alert) on the EHR patient list (main landing screen upon login) (Fig. 1).”

• cites [22] Opitz, D. & Maclin, R. Popular ensemble methods: an empirical study. J. Artif. Intell. Res. 11, 169–198 (1999). sourceSupported

May combine two claims.

CONCERN intervention was associated with a higher instantaneous risk of unanticipated ICU transfer (adjusted HR 1.25; 95% CI 1.09-1.43; P=0.0011) and a 24.9% increase in in-hospital instantaneous risk of unanticipated ICU transfers.

“For unanticipated ICU transfers, there were 505 hospital–patient encounters in the intervention group and 377 hospital–patient encounters in the usual-care group (adjusted HR, 1.25; 95% CI, 1.09–1.43; P = 0.0011).”

• Also stated in: § Discussion · plain

  “A hospital-wide EWS based on nursing surveillance patterns resulted in a 35.6% decrease in in-hospital instantaneous risk of mortality, 11.2% decrease in LOS, 7.5% decrease in in-hospital instantaneous risk of sepsis and 24.9% increase in in-hospital instantaneous risk of unanticipated ICU transfers, when integrated into the care team’s EHR workflow compared with usual care.”

May combine two claims.

The estimated unadjusted mean LOS was 157.9 hours (6.6 days) for hospital-patient encounters in intervention units and 172.7 hours (7.2 days) for hospital-patient encounters in usual-care units.

“The estimated unadjusted mean LOS (a known underestimate given that we cannot account for patients who died) was 157.9 hours (6.6 days) for hospital–patient encounters in intervention units and 172.7 hours (7.2 days) for hospital–patient encounters in usual-care units (Table 2).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Among patients without a COVID-19 diagnosis, the mortality rate was 158/31,777 (0.49%) in the intervention group.

“Mortality rate for patients without COVID-19 Diagnosis | 158/31,777 (0.49%) | 204/25,702 (0.79%)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The distribution of study units was: site A intervention-19 ACUs and 5 ICUs; site A usual care-17 ACUs and 7 ICUs; site B intervention-9 ACUs and 4 ICUs; site B usual care-8 ACUs and 5 ICUs.

“site A intervention—19 ACUs and 5 ICUs; site A usual care—17 ACUs and 7 ICUs; site B intervention—9 ACUs and 4 ICUs; site B usual care—8 ACUs and 5 ICUs (Extended Data Table 4).”

Probably not a substantial claim: may not pass S3 (load-bearing).

No adverse events were reported during the trial.

“No adverse events were reported during the trial.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The gradient-boosted decision-tree models were trained at site B on 70% of a retrospective dataset, with 30% used for tenfold cross-validation.

“The gradient-boosted decision-tree models were trained at site B on 70% of a retrospective dataset, with 30% used for tenfold cross-validation”

• AssumptionRests on an assumption: The first occurrence of cardiopulmonary arrest, rapid response, sepsis, transfer to ICU, and death are valid proxy measures for patient deterioration.

Probably not a substantial claim: may not pass S3 (load-bearing).

Among patients with a COVID-19 diagnosis, the mortality rate was 61/2,167 (2.81%) in the usual-care group.

“Mortality rate for patients with COVID-19 Diagnosis | 23/1,247 (1.84%) | 61/2,167 (2.81%)”

Classified as probably not a substantial claim.

The trial included 74 clinical units, with 37 intervention and 37 usual care.

“The trial included 74 clinical units (37 intervention, 37 usual care)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Among patients with a COVID-19 diagnosis, the mortality rate was 23/1,247 (1.84%) in the intervention group.

“Mortality rate for patients with COVID-19 Diagnosis | 23/1,247 (1.84%) | 61/2,167 (2.81%)”

May combine two claims.

Model development used 217,166 distinct inpatient-hospital encounters between 2015 and 2019 from two study sites, with a task of predicting a deterioration event in the next 24 hours based on the patient's preceding 24-hour data.

“Briefly, model development used 217,166 distinct inpatient–hospital encounters between 2015 and 2019 from two study sites, with a task of predicting a deterioration event in the next 24 hours based on the patient’s preceding 24-hour data.”

May combine two claims.

CONCERN EWS improved outcomes over usual care across both study sites despite poor staffing, increased patient acuity, and limited hospital resources during 2020-2022.

“Despite the well-documented poor staffing, increased patient acuity and limited hospital resources rampant across the globe during the time we conducted our study (2020 to 2022)1,48, CONCERN EWS still improved outcomes over usual care across both study sites.”

• cites [1] American Hospital Association. Report: Rising patient acuity driving up hospital costs as payments fall. AHA News https://www.aha.org/news/headline/2022-08-15-report-rising-patient-acuity-driving-hospital-costs-payments-fall (2022).Not checked yet

  Not checked in the free run.

• cites [48] Office of Surgeon General. Addressing Health Worker Burnout: The U.S. Surgeon General’s Advisory on Building a Thriving Health Workforce (US Department of Health and Human Services, 2022).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Among patients without a COVID-19 diagnosis, the mortality rate was 204/25,702 (0.79%) in the usual-care group.

“Mortality rate for patients without COVID-19 Diagnosis | 158/31,777 (0.49%) | 204/25,702 (0.79%)”

Classified as probably not a substantial claim.

Eligible adult hospital encounters were included in the trial.

“Eligible adult hospital encounters were included.”

May combine two claims.

The trial was a 1-year, multisite, pragmatic, cluster-randomized controlled trial with 74 clinical units (37 intervention; 37 usual care) across 2 health systems.

“We conducted a 1-year, multisite, pragmatic trial with cluster-randomization of 74 clinical units (37 intervention; 37 usual care) across 2 health systems.”

• Also stated in: § Abstract · plain

  “this 1-year, multisite, pragmatic, cluster-randomized controlled clinical trial at two large health systems (four hospitals)”

May combine two claims.

A total of 60,893 hospital encounters were included in trial analyses, with 33,024 in the intervention group and 27,869 in the usual-care group.

“Among 60,893 hospital encounters (33,024 intervention; 27,869 usual care)”

• Also stated in: § Trial participants and hospital encounters · plain

  “a total of 60,893 were included in trial analyses, with 33,024 in the CONCERN EWS intervention group and 27,869 in the usual-care control group for our survival analyses”

Probably not a substantial claim: may not pass S3 (load-bearing).

The trial was registered at ClinicalTrials.gov as NCT03911687.

“ClinicalTrials.gov registration: NCT03911687.”

May combine two claims.

The LOS analysis was conducted in two ways, both indicating a statistically significant decrease, demonstrating the reliability of the results.

“Given common challenges in interpreting LOS, we analyzed LOS in two ways and both indicated a statistically significant decrease, demonstrating the reliability of our results.”

Classified as probably not a substantial claim.

Nurse surveillance is a core component of nursing practice aimed at preventing adverse events.

“Nurse surveillance is a core component of nursing practice aimed at preventing adverse events11.”

• cites [11] Halverson, C. C. & Scott Tilley, D. Nursing surveillance: a concept analysis. Nurs. Forum 57, 454–460 (2022). sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

During the 42-hour period leading up to outcome events, the intervention group's average time on CONCERN-assigned units was 40.27 hours (s.d.=5.11) and the control group's average time on usual-care-assigned units was 40.19 hours (s.d.=5.47).

“During the 42-hour period leading up to hospital encounter outcome events, the intervention group’s average time spent on CONCERN-assigned units was 40.27 (s.d. = 5.11) hours, and the control group’s average time spent on usual-care-assigned units was 40.19 (s.d. = 5.47) hours.”

May combine two claims.

Ensemble learning allowed the authors to identify the best model for a patient who has been in an ICU for 5 hours on a Friday night and to make the EWS transparent and explainable.

“For example, ensemble learning allowed us to identify the best model for a patient who has been in an ICU for 5 hours on a Friday night and to make our EWS transparent and explainable.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Unmeasured confounding might impact differences in trial outcomes, but the analyses adjusted for factors commonly established as explanatory for variation in the outcomes.

“Fourth, unmeasured confounding might impact differences in trial outcomes. However, our analyses adjusted for factors commonly established as explanatory for variation in our outcomes.”

Classified as probably not a substantial claim.

Patient encounters that included hospice and palliative care, or DNR and/or DNI and comfort care orders activated before any trial outcome event were excluded.

“Patient encounters that included hospice and palliative care, or DNR and/or DNI and comfort care orders activated before any trial outcome event, were excluded.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The standard Centers for Medicare & Medicaid Services definition of 30-day readmission has known limitations, including loss of follow-up for patients readmitted within a different hospital system and patients who died and are not known to have died.

“Seventh, we acknowledge the known limitations to the standard Centers for Medicare & Medicaid Services definition of 30-day readmission, including loss of follow-up, for patients readmitted within a different hospital system and patients who died and are not known to have died.”

Classified as probably not a substantial claim.

GLM analyses for LOS and 30-day hospital readmission were adjusted for age, Charlson Comorbidity Index, sex, race, ethnicity, and study site.

“GLM analyses for LOS and 30-day hospital readmission were adjusted for age, Charlson Comorbidity Index, sex, race, ethnicity and study site.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study had 80% statistical power to detect a difference of less than 1% relative difference in mortality rates.

“• 80% statistical power to detect a difference of less than 1% relative difference in mortality rates”

Classified as probably not a substantial claim.

During model development, the first occurrence of cardiopulmonary arrest, rapid response, sepsis, transfer to ICU, and death were identified as proxy measures for patient deterioration.

“During model development, the first occurrence of cardiopulmonary arrest, rapid response, sepsis, transfer to ICU and death were identified as proxy measures for patient deterioration.”

Classified as probably not a substantial claim.

All existing hospital policies and procedures related to patient deterioration and escalation of care remained the same across intervention and usual-care groups.

“All existing hospital policies and procedures related to patient deterioration and escalation of care remained the same across intervention and usual-care groups.”

Classified as probably not a substantial claim.

The power analysis was based on an expected sample of 2,000 total admissions per month with 50,000 total inpatient days based on average occupancy and a mean mortality rate of 37.5 deaths per 10,000 inpatient days (ranging from 11.9 to 48.8 across study sites).

“This is based on an expected sample of 2,000 total admissions per month with 50,000 total inpatient days based on average occupancy and a mean mortality rate of 37.5 deaths per 10,000 inpatient days (ranging from 11.9 to 48.8 across our study sites) (Extended Data Table 7).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Study design remained the same as originally planned and specified in ClinicalTrial.gov.

“Study design remained the same as originally planned and specified in ClinicalTrial.gov.”

Classified as probably not a substantial claim. May combine two claims.

The power analysis was based on a cluster-randomized trial using intracluster correlation coefficient equal to 0.2 and generalized linear mixed models for the comparison of mortality rates between intervention and usual-care units.

“This power analysis was based on a cluster-randomized trial using intracluster correlation coefficient equal to 0.2 and generalized linear mixed models for the comparison of mortality rates between those randomized to CONCERN intervention and usual-care units.”

Probably not a substantial claim: may not pass S4 (based).

LOS can be used as a proxy for other outcomes (for example, shorter LOS associated with a quicker recovery).

“As discussed earlier, LOS can be used as a proxy for other outcomes (for example, shorter LOS associated with a quicker recovery).”

Probably not a substantial claim: may not pass S3 (load-bearing).

No clinical units dropped out during the study period.

“No clinical units dropped out during the study period.”

Classified as probably not a substantial claim.

Each hospital encounter was analyzed independently.

“We analyzed each hospital encounter independently.”

Classified as probably not a substantial claim.

The original analysis plan was modified from analyzing outcomes at both the unit and individual-patient level to analyzing outcomes only at the individual-patient level per hospital encounter.

“However, we modified our original analysis plan, which was to analyze outcomes both at the unit level and individual-patient level per hospital encounter (that is, each hospital admission), to instead analyze outcomes only at the individual-patient level per hospital encounter.”

• AssumptionRests on an assumption: Patient movement (transfers) between clinical units occurs frequently.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Mortality may result from a series of events extending beyond a single hospital encounter, but the use of a cluster-randomized controlled trial helps mitigate unknown confounders.

“Finally, eighth, while our discussion like other EWS interventional studies in hospital settings attributes mortality to a single encounter, we acknowledge that mortality may result from a series of events extending beyond a single hospital encounter. However, the use of a cluster-randomized controlled trial helps mitigate unknown confounders.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Institutional review boards at Columbia University and Brigham and Women's Hospital approved the protocol before trial initiation with a waiver of consent.

“Institutional review boards at Columbia University, United States, and Brigham and Women’s Hospital, United States, approved the protocol before trial initiation with a waiver of consent.”

Classified as probably not a substantial claim.

Outcomes were assessed at the individual hospital-patient encounter level due to patient movement across units during hospitalization.

“While randomization occurred at the unit level, outcomes were assessed at the individual hospital–patient encounter level due to patient movement across units during their hospitalization.”

• AssumptionRests on an assumption: Patient movement (transfers) between clinical units occurs frequently.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The updated Quan et al. version of the Charlson Comorbidity Index was used because it is recognized as the most widely used comorbidity scoring system and could be used reliably across both study sites.

“We used the updated Quan et al. version of the Charlson Comorbidity Index68 because it is recognized as the most widely used comorbidity scoring system69 and, therefore, could be used reliably across both study sites.”

• cites [68] Quan, H. et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med. Care. 43, 1130–1139 (2005). sourceNot checked — no full text of the source
• cites [69] Figueiredo, S. in Stroke Engine (eds Zeltzer, L. et al.) (Canadian Partnership for Stroke Recovery Heart & Stroke Foundation, 2009); https://strokengine.ca/en/assessments/charlson-comorbidity-index-cciNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The trial was complicated by disruptions at the sites due to the COVID-19 pandemic, which forced a 1-year postponement at one of the sites due to implementation delays.

“First, the trial was complicated by disruptions at the sites due to the COVID-19 pandemic, which forced a 1-year postponement at one of our sites due to implementation delays.”

Classified as probably not a substantial claim. May combine two claims.

All measures, including patient characteristics in Table 1, were analyzed for each hospital-patient encounter and not each patient because the reported characteristics could change between hospital encounters.

“To this point, all measures, including patient characteristics in Table 1, were analyzed for each hospital–patient encounter (and not each patient) because the reported characteristics could change between hospital encounters (for example, Charlson Comorbidity Index is assessed at each admission; discharge disposition is per encounter).”

Probably not a substantial claim: may not pass S3 (load-bearing).

A sensitivity analysis was performed for imbalance in COVID-19 diagnoses across study arms by excluding patient encounters with a COVID-19 diagnosis and repeating the survival analysis.

“Given that our study took place during the pandemic, we performed a sensitivity analysis for imbalance in COVID-19 diagnoses per patient–hospital encounters across study arms by excluding patient encounters with a COVID-19 diagnosis and repeating the survival analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study design as a cluster-randomized trial never changed.

“Our study design as a cluster-randomized trial never changed.”

May combine two claims.

The usual-care arm included 12 ICUs compared to nine ICUs in the intervention arm, but baseline characteristics between groups indicated no differences and analyses adjusted for confounders.

“Third, the usual-care arm included 12 ICUs compared to nine ICUs in the intervention arm, although evaluation of baseline characteristics between groups indicated no differences and analyses adjusted for confounders.”

Classified as probably not a substantial claim.

By targeting the entire clinical team rather than a solitary provider, CONCERN EWS promotes shared situational awareness.

“By targeting the entire clinical team rather than a solitary provider, CONCERN EWS promotes shared situational awareness.”

May combine two claims.

To address variability in EHR structures, the study employed two approaches: using simple and ubiquitous observational variables as a proxy for nursing surveillance, and using FHIR methods to retrieve real-time data.

“Sixth, data may be stored in different ways in other EHRs. Due to this variability in EHR structures, we purposely employed two approaches: (1) we used observational variables that are a proxy for nursing surveillance and are relatively simple and ubiquitous (for example, vital signs and note-writing patterns), which our prior research identified had strong predictive relationships for”

Classified as probably not a substantial claim. May combine two claims.

Transparency and explainability, achieved by displaying the CONCERN score and factors driving the prediction, may overcome the dissonance between nurses' identification of patient risk based on subtle clinical indicators and prescribing providers' expectations of changes in physiological data, thereby prompting earlier decisions to escalate medical intervention.

“Mortality increases when care escalation is delayed47; therefore, transparency and explainability, achieved by displaying the CONCERN score and factors driving the prediction, may overcome the dissonance between nurses’ identification of patient risk based on subtle clinical indicators and prescribing providers’ expectations of changes in physiological data, thereby prompting earlier decisions to”

Classified as probably not a substantial claim.

Coprimary outcomes were in-hospital mortality and length of stay.

“Coprimary outcomes were in-hospital mortality and LOS.”