CoChat
Sign Up
Cited by 21Sepsis Diagnosis and TreatmentMachine Learning in HealthcareAcute Kidney Injury Research

Multicenter Prospective Validation of an Updated Proprietary Sepsis Prediction Model

Open access
Wong, Andrew, Currey, Danielle, Schwinne, Megan, Park-Egan, Brenna, Meyer, Sean, Gutting, Andrew, Cao, Jie, Khan, Sharaf +8 more·2026·JAMA Network Open
DOI 10.1001/jamanetworkopen.2026.0181PMC PMC12949446
Open this paper in CoChat — free accountOpen PDF

Abstract

Importance: The Epic Sepsis Model version 2 (ESM v2) is a widely implemented proprietary sepsis prediction model, but no multicenter, external validation of its performance has been reported to guide adoption and use. Objective: To conduct a multicenter validation of the ESM v2 to compare performance against the original ESM v1, outline differences across heterogenous clinical sites, and compare model performance against clinician recognition of sepsis. Design, Setting, and Participants: This prognostic study included adult inpatient encounters at 4 large US health systems between August 31, 2023, and March 11, 2025. At each site, data were collected for a consecutive period immediately following new model implementation. Data were analyzed from July 23 to August 19, 2025. Main Outcomes and Measures: Sepsis was defined using Sepsis-3 clinical consensus criteria. Model discrimination was assessed using area under the receiver operating characteristic curve (AUROC) at the encounter level and prediction level with 4-hour, 12-hour, and hospitalization-wide time horizons. Performance against clinician recognition of sepsis was measured using antibiotics, lactate, and body culture orders. Results: Of 227 091 inpatient encounters, 7401 (3.3%; median [IQR] age, 65 [54-75] years; 3359 [45.4%] female; 2.7% Asian; 24.6% Black; 64.6% White; 7.1% Hispanic ethnicity) met sepsis criteria and 219 690 (96.7%; median [IQR] age, 48 [33-65] years; 123 563 [56.2%] female; 2.5% Asian; 38.8% Black; 49.6% White; 9.6% Hispanic ethnicity) did not. At the encounter level, the AUROC ranged from 0.82 (95% CI, 0.81-0.83) to 0.92 (95% CI, 0.92-0.93) across study sites. At the prediction level with a 12-hour time horizon, the AUROC ranged from 0.75 (95% CI, 0.74-0.75) to 0.85 (95% CI, 0.85-0.85). Comparison against clinician recognition of sepsis yielded a minor decrease in performance, with the resulting encounter-level AUROC ranging from 0.80 (95% CI, 0.79-0.81) to 0.90 (95% CI, 0.89-0.90) across sites. Positive predictive values remained low, from 0.13 (95% CI, 0.13-0.14) to 0.26 (95% CI, 0.25-0.27), with a high number needed to evaluate and high alert burden. Conclusions and Relevance: In this prognostic study of a new sepsis prediction model, a multicenter prospective validation performed across 4 major US health systems found improved discrimination for the early prediction of sepsis but noted high institutional variability, low positive predictive value, and high alert burden.

AI snapshot

Written from the paper’s full text.

TL;DR

This multicenter prospective validation of the Epic Sepsis Model version 2 (ESM v2) across 227,091 inpatient encounters at 4 US health systems found encounter-level AUROC between 0.82 and 0.92, but with high institutional variability, low positive predictive value, and high alert burden. ESM v2 outperformed ESM v1 across all metrics and sites, though performance was lower against clinician recognition of sepsis.

Key findings

  • Among 227,091 encounters, 7,401 (3.3%) met Sepsis-3 criteria. Encounter-level AUROC for ESM v2 ranged from 0.82 (95% CI, 0.81-0.83) at Michigan to 0.92 (95% CI, 0.92-0.93) at MetroHealth. At the 60% sensitivity threshold, specificity ranged from 0.83 to 0.96, PPV from 0.13 to 0.26, and NPV from 0.97 to 0.99; threshold scores ranged from 14 to 37 across sites.
  • At the prediction level with a 4-hour horizon, AUROC ranged from 0.75 to 0.86, sensitivity from 0.25 to 0.44, PPV from 0.02 to 0.04, and NNE from 24 to 69. At 12 hours, NNE ranged from 21 to 35; at the hospitalization-wide horizon, NNE improved to 4 to 8.
  • ESM v2 outperformed ESM v1 across all performance metrics, thresholds, and sites. At 60% sensitivity, ESM v1 had encounter-level AUROC from 0.65 to 0.84 and PPV from 0.07 to 0.14.
  • Against a composite clinician recognition definition, ESM v2 AUROC ranged from 0.80 to 0.90, with median prediction lead time ahead of clinician recognition from 1.4 to 7.1 hours. A fairness audit found no major deviations suggesting disproportionately better or worse performance by age, sex, race, or ethnicity; performance was closely associated with baseline sepsis rate. Alert burden was high, and an 8-hour silencing strategy reduced total alerts but low PPV and high NNE persisted.

Methods

  • Prospective validation at 4 large US health systems (University of Michigan, Oregon Health & Science University, Emory Healthcare, MetroHealth) using consecutive post-implementation periods. Included adults aged 18 years or older evaluated in the ED or hospitalized with at least 1 prediction score from both models; excluded patients who met sepsis criteria, were discharged, or died before a model score.
  • Sepsis outcome used Sepsis-3 criteria: suspected infection defined as body fluid culture plus first dose of at least 2 antibiotic doses from 72 hours before to 24 hours after culture order; organ dysfunction defined as a 2-point or greater change in SOFA score from encounter baseline between 48 hours before and 24 hours after suspected infection. Models generated scores every 15 minutes; ESM v2 was fine-tuned on site-specific historical data before implementation, and ESM v1 scores were calculated silently after implementation.
  • Analysis assessed AUROC, and at each institution’s 60% sensitivity threshold calculated specificity, PPV, NPV, and NNE, with 95% CIs from 1000 bootstrap resamples. Performance was analyzed at encounter level and prediction level with 4-hour, 12-hour, and hospitalization-wide horizons; calibration plots, sensitivity analyses against clinician recognition indicators, a fairness audit, and alert burden simulations with an 8-hour silencing strategy were also performed.

Limitations

  • The paper states that Sepsis-3 criteria have imperfect concordance with manual physician review, limiting analysis to cases that received cultures and antibiotic orders; postimplementation integration may introduce confirmation bias or model drift; the study did not differentiate ED, floor, and ICU sepsis; all sites fine-tuned the model, so base-model performance without fine-tuning is unknown; and the observational design was not intended to evaluate mortality or other clinical outcomes.
  • The text also makes evident that the first-hour no-CBC exclusion removed 9.8% to 30.1% of sepsis-positive encounters across sites, potentially limiting early prediction for a substantial proportion of sepsis cases.
  • The paper notes that simultaneous silent testing of both models was not feasible because sites depended on the models for clinical care, and the proprietary, site-fine-tuned nature of ESM v2 may limit generalizability of these performance estimates.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1001/jamanetworkopen.2026.0181

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

18 of 21 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured

  • Study design and conductcitation

    This study used Sepsis-3 clinical criteria to label sepsis cases, an approach that has imperfect concordance with the criterion standard approach of manual physician review.

    “First, this study used Sepsis-3 clinical criteria to label sepsis cases, an approach that has imperfect concordance with the criterion standard approach of manual physician review.25,26,27,28”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [25] Lindner, 2022, Ground truth labels challenge the validity of sepsis consensus definitions in critical illness., J Transl Med source

      The Results section ('Validation of Sepsis-1/2 and Sepsis-3 clinical criteria against ground truth labels'; page unavailable) reports only fair agreement between Sepsis-3 criteria and expert sepsis-onset labels, supporting the imperfect-concordance point. The Methods describe those reference labels as daily judgments by senior physicians using a questionnaire, not sepsis labels established through manual physician chart review. The cited paper also cannot establish how the citing study labeled its own cases. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [26] Henry, 2019, Comparison of automated sepsis identification methods and electronic health record-based sepsis phenotyping: improving case identification accuracy by accounting for confounding comorbid conditions., Crit Care Explor source

      The Discussion reports that automated Sepsis-3 can identify many false-positive cases, supporting imperfect agreement with physician assessment. But it says no gold standard for sepsis exists and physician case review is subjective. The paper also reports a direct physician-review comparison for ESP, not Sepsis-3, so it does not establish the stated criterion-standard comparison for Sepsis-3. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [27] Rhee, 2019, Variation in identifying sepsis and organ dysfunction using administrative versus electronic clinical data and impact on hospital outcome comparisons., Crit Care Med source

      The Discussion says the paper’s EHR-based clinical criteria are an imperfect reference standard and that there is no true gold standard for sepsis. But the paper did not review medical records to confirm its criteria (Discussion, limitations), so it does not establish their concordance with manual physician review or describe that review as the criterion standard. The citing study’s own use of Sepsis-3 is not a finding attributed to this source. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    Patient complexity and baseline sepsis rates affect the performance of sepsis prediction models.

    “These factors support prior evidence that patient complexity and baseline sepsis rates affect the performance of sepsis prediction models.13”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [13] Lyons, 2023, Factors associated with variability in the performance of a proprietary sepsis prediction model across 9 networked hospitals in the US., JAMA Intern Med source
  • Interpretationcitation

    The overall AUROC of the new model was generally lower than that reported in other frontier sepsis prediction models, such as COMPOSER, Targeted Real-time Early Warning System, and SepsisWatch.

    “While the overall performance (eg, AUROC) of the new model was generally lower than that reported in other frontier sepsis prediction models, such as COMPOSER,18 Targeted Real-time Early Warning System,19 and SepsisWatch,20”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [18] Boussina, 2024, Impact of a deep learning sepsis prediction model on quality of care and survival., NPJ Digit Med source
    • cites [19] Henry, 2022, Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing., Nat Med source

      The 'Targeted Real-time Early Warning System (TREWS)' section identifies TREWS as a machine-learning sepsis early warning system, so the citation supports its identification as an example. The Discussion reports sensitivity and precision, but our copy does not report a TREWS AUROC; it therefore does not substantiate the implied AUROC comparison with the citing paper's new model. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [20] Valan, 2025, Evaluating sepsis watch generalizability through multisite external validation of a sepsis machine learning model., NPJ Digit Med source
  • Interpretationcitation

    The new model outperforms other more commonly deployed prediction models and clinical prediction scores, such as Systemic Inflammatory Response Syndrome, SOFA, and Quick SOFA.

    “it outperforms other more commonly deployed prediction models21 and clinical prediction scores, such as Systemic Inflammatory Response Syndrome, SOFA, and Quick SOFA.22”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [21] Wallgren, 2021, Performance of NEWS2, RETTS, clinical judgment and the Predict Sepsis screening tools with respect to identification of sepsis among ambulance patients with suspected infection: a prospective cohort study., Scand J Trauma Resusc Emerg Med source

      The Results section compares Predict Sepsis tools with NEWS2, RETTS and clinical judgment, but does not compare them with SIRS, SOFA or qSOFA. A search for those named scores found SOFA used to define the sepsis outcome, not as a competing prediction score. Our copy therefore does not establish the attributed outperformance claim. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [22] Schertz, 2023, Sepsis prediction model for determining sepsis vs SIRS, qSOFA, and SOFA., JAMA Netw Open source

      The Findings report that the SPM had better balanced accuracy than SIRS, SOFA, and qSOFA at a predicting sepsis score threshold of 8 or greater. The Conclusions describe that advantage as marginal and note poorer timeliness, so the unqualified claim that it outperforms these scores overstates the result. The separate claim about other prediction models is attached to citation 21, not this source. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Premisecitation

    The Epic Sepsis Model version 1 was found to have poor discrimination and calibration in several external validations across different health systems.

    “Despite its widespread deployment, this model was found to have poor discrimination and calibration in several external validations across different health systems.12,13,14,15”
    Quote in the paperSubstantial claim3 of 4 sources checked
    • cites [12] Wong, 2021, External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients., JAMA Intern Med source

      The Results and Discussion report poor discrimination and calibration for the Epic Sepsis Model in this external validation. The Methods and Discussion specify that it was conducted at a single academic health system, Michigan Medicine; this paper therefore does not establish the claim about several external validations across different health systems. The passages do not identify the model as version 1. Page numbers are unavailable in our copy. (LLM, conf=0.89)

      Experimental — accuracy not yet measured
    • cites [13] Lyons, 2023, Factors associated with variability in the performance of a proprietary sepsis prediction model across 9 networked hospitals in the US., JAMA Intern Med source

      The Results report that ESM discrimination varied across nine hospitals in one network, with a C-statistic of 0.55 at the largest hospital. The Methods describe measuring discrimination, not calibration; the Discussion only suggests that increased COVID-19-related alerting may reflect miscalibration. This paper therefore supports poor discrimination at some hospitals, but not poor calibration established by several validations across different health systems. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [14] Ostermayer, 2024, External validation of the Epic sepsis predictive model in 2 county emergency departments., JAMIA Open source

      The Discussion reports poor diagnostic performance for ESPMv1 and says the results align with prior external validations. But the Limitations specify that this study covered one health system with two emergency departments, and our copy does not report poor calibration or establish the claim about several validations across different health systems. No page numbers are provided in the copy. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [15] Kamran, 2024, Evaluation of sepsis prediction models before onset of treatment., NEJM AI source
  • Classified as probably not a substantial claim. 15
  • May combine two claims. 20
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim.

    Premisecitation

    Prior sepsis research has suggested physiologic and epidemiologic heterogeneity between community-acquired and hospital-acquired sepsis populations.

    “Prior sepsis research has suggested physiologic and epidemiologic heterogeneity between community-acquired and hospital-acquired sepsis populations30”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [30] Markwart, 2020, Epidemiology and burden of sepsis acquired in hospitals and intensive care units: a systematic review and meta-analysis., Intensive Care Med source

      The Introduction distinguishes community-acquired from hospital-acquired sepsis, and the Discussion reports that patients with hospital-acquired sepsis have longer hospital or ICU stays than patients with community-acquired sepsis. This supports a difference between the populations, but the passages do not establish physiologic heterogeneity between them; the reported inter-study heterogeneity concerns variation across studies, not that comparison. Page numbers were not available in our copy. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Hospital-onset sepsis is more clinically complex and harder to predict than community-acquired sepsis.

    “prior research suggesting that hospital-onset sepsis is more clinically complex and harder to predict than community-acquired sepsis.23,24”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [23] Rhee, 2019, Epidemiology of hospital-onset versus community-onset sepsis in U.S. hospitals and association with mortality: a retrospective analysis using electronic clinical data., Crit Care Med source

      The Results (Patient Characteristics; page not provided) show that hospital-onset sepsis patients had more comorbidities and higher SOFA scores, supporting the description of greater clinical complexity. The Discussion (page not provided) suggests hospital-onset sepsis may be slower to recognize, but it does not establish that it is harder to predict. The paper also distinguishes community-onset sepsis from infections necessarily acquired in the community. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [24] Page, 2015, Community-, healthcare-, and hospital-acquired severe sepsis hospitalizations in the University HealthSystem Consortium., Crit Care Med source

      The Discussion (page unavailable) says hospital-acquired and community-acquired severe sepsis likely differ in clinical presentation and may require different detection strategies; it also notes that inpatient wards often lack frequent monitoring. This supports a distinction in detection challenges, but the paper does not establish that hospital-onset sepsis is more clinically complex or harder to predict than community-acquired sepsis. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    The Sepsis-3 consensus framework for sepsis coding and research recommends using concomitant orders for blood cultures and antibiotics within a specified period to define suspected infection.

    “The Sepsis-3 consensus framework for sepsis coding and research recommends using concomitant orders for blood cultures and antibiotics within a specified period to define suspected infection.4”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [4] Singer, 2016, The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3)., JAMA source

      In Results/Recommendations, the paper describes a research cohort in which suspected infection was identified by cultures being sampled and antibiotics being received. It does not recommend concomitant orders within a specified period as a definition for coding and research; the Implications section instead discusses proxies such as antibiotic commencement or clinical probability of infection. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Postimplementation model drift due to changes in clinician behavior is a widely recognized challenge in the evaluation of clinical prediction models with no standard solution.

    “Postimplementation model drift due to changes in clinician behavior is a widely recognized challenge in the evaluation of clinical prediction models with no standard solution.29”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [29] Ansari, 2025, Challenges in the postmarket surveillance of clinical prediction models., NEJM AI source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The reporting of this study was guided by TRIPOD+AI reporting guidelines.

    “The reporting of this study was guided by Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis updated for artificial intelligence (TRIPOD+AI) reporting guidelines.16”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [16] Collins, 2024, TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods., BMJ source

  • May combine two claims.

    Findingown data

    Higher rates of sepsis were observed in older patients, male patients, White patients, and patients with non-Hispanic ethnicity.

    “Higher rates of sepsis were observed in older patients, male patients, White patients, and patients with non-Hispanic ethnicity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Higher sepsis rates were observed at tertiary care centers.

    “The baseline sepsis incidence ranged from 2.2% to 7.1% across sites, with higher sepsis rates observed at tertiary care centers (Table 2).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    A theoretical 8-hour silencing strategy substantially decreased total alert burden, but low PPV and high NNE persisted.

    “A theoretical 8-hour silencing strategy substantially decreased total alert burden, but low PPV and high NNE persisted.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Our analysis found low PPV and high NNE associated with this model.

    “Our analysis found low PPV and high NNE associated with this model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    The new model maintained a moderate level of performance when compared against clinician recognition of sepsis, where the original model did not.

    “The new model also maintained a moderate level of performance when compared against clinician recognition of sepsis, where the original model did not.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At the threshold matching a sensitivity of 0.60, the original ESM v1 had an encounter-level AUROC ranging from 0.65 to 0.84 and a PPV ranging from 0.07 to 0.14.

    “At the threshold matching a sensitivity of 0.60, the original had an encounter-level AUROC ranging from 0.65 (95% CI, 0.64-0.65) to 0.84 (95% CI, 0.83-0.85) and a PPV ranging from 0.07 (95% CI, 0.07-0.08) to 0.14 (95% CI, 0.14-0.15).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Tertiary care sites had higher baseline sepsis rates, longer median times to sepsis positivity, longer median length of stay, a higher number of predictions per patient, and lower overall model performance against clinician recognition.

    “Tertiary care sites had higher baseline sepsis rates, longer median times to sepsis positivity, longer median length of stay, a higher number of predictions per patient, and lower overall model performance against clinician recognition.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ESM v2 performance varied significantly across study sites, including in AUROC, thresholds matching 60% sensitivity, prediction lead time, and performance against clinician suspicion.

    “with high variation in performance among institutions.”
    • Also stated in: § Discussion · plain
      “High variability was noted in model performance across study sites”
    • Also stated in: § Discussion · plain
      “Discriminative accuracy (AUROC, 0.82-0.92), thresholds matching 60% sensitivity (14 to 37), prediction lead time (1.5-9.1 hours) and decrease in model performance against clinician suspicion all varied significantly across institutions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The new model outperformed ESM v1 across all performance metrics, at all model thresholds, and at all study sites.

    “The new model outperformed the ESM v1 across all performance metrics, at all model thresholds, and at all study sites.”
    • Also stated in: § Discussion · plain
      “ESM v2 performance was superior to that of ESM v1 across all metrics.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingreasoning

    The new model demonstrated stronger performance at institutions with earlier sepsis onset times, a higher proportion of sepsis cases occurring in the ED, and lower sepsis incidence rates.

    “The new model demonstrated stronger performance at institutions with earlier sepsis onset times and a higher proportion of sepsis cases occurring in the ED, which is consistent with community-acquired cases.”
    • Also stated in: § Strengths and Limitations · hedged
      “This is consistent with our findings that suggested superior performance of this model at sites with lower sepsis incidence rates and earlier times to sepsis positivity.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Comparison of sepsis incidence rates following model integration against historical rates found no significant difference at our study sites, suggesting that introduction of the new model did not significantly change sepsis labeling rates.

    “First, comparison of sepsis incidence rates following model integration against historical rates found no significant difference at our study sites, suggesting that introduction of the new model did not significantly change sepsis labeling rates.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    Use of this clinical sepsis definition limited the analysis to sepsis cases that eventually received cultures and antibiotic orders and did not include any sepsis cases that did not eventually receive treatment.

    “Use of this clinical sepsis definition limited the analysis to sepsis cases that eventually received cultures and antibiotic orders and did not include any sepsis cases that did not eventually receive treatment.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    At each site, data were collected for a consecutive time period immediately following model implementation.

    “At each site, data were collected for a consecutive time period immediately following model implementation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The analysis of model performance against multiple clinician recognition metrics and at multiple time horizons provides a nuanced perspective of the real-world clinical trade-offs among model precision, timeliness, and false-positive rates.

    “Furthermore, the analysis of model performance against multiple clinician recognition metrics and at multiple time horizons provides a nuanced perspective of the real-world clinical trade-offs among model precision, timeliness, and false-positive rates.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Automated sepsis prediction tools often lack generalizability and are commonly implemented by health systems without internal validation.

    “However, such tools often lack generalizability and are commonly implemented by health systems without internal validation,8 potentially compromising sepsis care through missed or delayed treatment.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [8] Nong, 2025, Current use and evaluation of artificial intelligence and predictive models in US hospitals., Health Aff (Millwood) source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The study was approved by the institutional review board at each participating site, and informed consent was waived due to a minimal risk exemption for secondary research.

    “This prognostic study was approved by the institutional review board at each participating site. The requirement for informed consent was waived due to a minimal risk exemption for secondary research.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Changes in alert performance may have affected clinician behavior, including the frequency and timing of antibiotic or body culture orders.

    “Changes in alert performance may have affected clinician behavior, including the frequency and timing of antibiotic or body culture orders.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Sepsis prediction models may be more effective in identifying early-onset community-acquired sepsis cases in the ED than on the floor and intensive care unit.

    “and that sepsis prediction models may be more effective in identifying early-onset community-acquired sepsis cases in the ED than on the floor and intensive care unit.31”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31] Tarabichi, 2022, Improving timeliness of antibiotic administration using a provider and pharmacist facing sepsis early warning system in the emergency department setting: a randomized controlled quality improvement initiative., Crit Care Med source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    A fairness audit was performed to determine if the model had variable performance across subpopulations stratified by age, sex, race, and ethnicity.

    “A fairness audit was performed to determine if the model had variable performance across subpopulations stratified by age, sex, race, and ethnicity.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The first hour CBC exclusion rate may have affected model performance differently across sites.

    “Finally, the first hour CBC exclusion rate may have affected model performance differently across sites.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The new sepsis prediction model cannot produce accurate predictions in the first hour of admission in absence of a CBC result.
    • Rests on an assumption: Health systems differ in their ability to consistently draw and run a CBC within the first hour of care due to variations in clinical workflows, patient volume, and laboratory access.

  • May combine two claims.

    Study design and conductreasoning

    This may have introduced confirmation bias in our Sepsis-3 abstracted outcome labels and inflated model accuracy.

    “This may have introduced confirmation bias in our Sepsis-3 abstracted outcome labels and inflated model accuracy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Because all study sites performed model fine-tuning prior to implementation, we cannot comment on the performance of the base model without fine-tuning.

    “Fourth, because all our study sites performed model fine-tuning prior to implementation, we cannot comment on the performance of the base model without fine-tuning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    This study analyzed observational data collected immediately after implementation of the new model at each study site and was not designed to evaluate its impact on clinical outcomes, such as mortality.

    “Fifth, this study analyzed observational data collected immediately after implementation of the new model at each study site and was not designed to evaluate its impact on clinical outcomes, such as mortality.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The standardized evaluation approach using Sepsis-3 criteria and a 60% sensitivity benchmark allowed for a clear comparison of performance between sites and models.

    “The standardized evaluation approach using Sepsis-3 criteria and a 60% sensitivity benchmark allowed for a clear comparison of performance between sites and models.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The use of Sepsis-3 abstraction for sepsis labeling allowed for a standardized approach to large-scale model validation across multiple study sites but limited the model performance analysis to clinician-centered outcomes.

    “The use of Sepsis-3 abstraction for sepsis labeling allowed for a standardized approach to large-scale model validation across multiple study sites but limited the model performance analysis to clinician-centered outcomes”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    All study sites previously used the original model for sepsis alerting, and no changes to the clinical workflow were made following implementation of the new model other than which model triggered sepsis alerts.

    “Second, all study sites previously used the original model for sepsis alerting, and no changes to the clinical workflow were made following implementation of the new model other than which model triggered sepsis alerts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Implementation of alert silencing strategies can help reduce total alert burden, but cannot improve alert accuracy.

    “The implementation of alert silencing strategies can help reduce total alert burden, but cannot improve alert accuracy.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    This study reported on model performance across the entire hospitalization and did not differentiate between sepsis diagnosed in the ED, on the inpatient floor, and in the intensive care unit.

    “Third, this study reported on model performance across the entire hospitalization and did not differentiate between sepsis diagnosed in the ED, on the inpatient floor, and in the intensive care unit.”
    Quote in the paperSubstantial claim
33 of 33

  • The paper relies on this without support.

    An external multicenter validation of the Epic Sepsis Model version 2 had not been reported before this study.

    “Despite broad adoption, an external multicenter validation of the new sepsis prediction model has yet to be reported.”

  • The paper relies on this without support.

    The new sepsis prediction model is currently deployed at hundreds of US hospitals.

    “the new sepsis prediction model is currently deployed at hundreds of US hospitals”

  • The paper relies on this without support.

    The new sepsis prediction model cannot produce accurate predictions in the first hour of admission in absence of a CBC result.

    “Per developer guidelines, the new sepsis prediction model cannot produce accurate predictions in the first hour of admission in absence of a CBC result.”

  • The paper relies on this without support.

    Health systems differ in their ability to consistently draw and run a CBC within the first hour of care due to variations in clinical workflows, patient volume, and laboratory access.

    “Health systems differ in their ability to consistently draw and run a CBC within the first hour of care due to variations in clinical workflows, patient volume, and laboratory access.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • OHSU sepsis in-hospital mortality percentage is too low

    Minor

    Table 1 reports 309 in-hospital deaths among 2326 OHSU sepsis encounters. That is 13.3%, not the reported 12%.

    Reported 12 · recomputed 13.284609

  • OHSU sepsis 30-day mortality percentage is too low

    Minor

    Table 1 reports 330 deaths within 30 days among 2326 OHSU sepsis encounters. That is 14.2%, not the reported 13.0%.

    Reported 13 · recomputed 14.187446

  • Clinician-recognition lead-time IQR differs from Table 3

    Minor

    The text gives the lower end of the cross-site lead-time range an upper IQR bound of 4.0 hours, while Table 3 gives 5.2 hours for the corresponding Emory value.

  • Discussion gives a different prediction lead-time range

    Minor

    The discussion states a cross-institution prediction lead-time range of 1.5 to 9.1 hours. Table 2 reports site medians ranging from 1.9 to 10.3 hours.

  • Sepsis-positivity time range uses a prediction lead-time value

    Minor

    The text gives 10.3 hours as the upper end of the median time from presentation to sepsis positivity. Table 2 instead gives 8.7 hours as the highest median sepsis-positivity time; 10.3 hours is its OHSU prediction lead time.

5 of 5

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Wong, A., Currey, D., Schwinne, M., Park-Egan, B., Meyer, S., Gutting, A., Cao, J., Khan, S., Dantes, R., Pan, T., Buchman, T., Singh, K., Bhavani, S. V., Lyons, P. G., Sjoding, M. W., & Tarabichi, Y.
Affiliations
  • University of Michigan (US)
  • MetroHealth (US)
  • Emory University (US)
  • Oregon Health & Science University (US)
  • UC San Diego Health System (US)
Corresponding author
Wong, Andrew
ORCID
Venue
JAMA Network Open
Publisher
American Medical Association (AMA)
Published
2026-02-27
Volume
9
Issue
2
Pages
e260181
ISSN
2574-3805 (ISSN-L)
DOI
10.1001/jamanetworkopen.2026.0181
PMC
PMC12949446
OpenAlex
W7131855346
Semantic Scholar
d13076c7b24c9600443063b2a42da33eb1c8cd75
Cited by
21
References
31
Citations by year
2026: 21
FWCI
85.31
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JAMA Network Open (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by)
Funders
National Institutes of Health (1k08ca270383-01a1, 5r01hl158626-04)
Topics
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Nephrology › Acute Kidney Injury Research
Keywords
sepsis prediction model, sepsis, external validation, model discrimination, Sepsis-3 criteria, inpatient encounters, lactate, body culture, model adoption, sepsis mortality, sepsis-related organ dysfunction
MeSH terms
Prediction Algorithms, Adult, Aged, Female, Humans, Male, Middle Aged, Predictive Value of Tests, Prognosis, Prospective Studies, ROC Curve, United States, Sepsis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-27.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-27.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review