CoChat
Sign Up
Cited by 4Electronic Health Records SystemsMachine Learning in HealthcareArtificial Intelligence in Healthcare and Education

A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools

Open access
Patel, Hardik, Crusco, Salvatore, Hansen, Derek, Morales, Jaclyn, Patale, Tara, Chiuzan, Codruta, Sands, Mark, Naidich, Jason +4 more·2026·Journal of General Internal Medicine
DOI 10.1007/s11606-026-10381-y
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Clinical Decision Support (CDS) tools integrated with Electronic Health Records increasingly guide clinical practice. Epic Systems, storing more than 325 million patient records, offers various proprietary predictive models to healthcare systems. Despite widespread adoption, no systematic review has examined these tools’ real-world performance compared to vendor-reported metrics. Methods: This study was prospectively registered on PROSPERO (CRD420251148571). We systematically searched PubMed, Scopus, and Embase (January 2018 to August 2025) for external validations of Epic’s CDS tools. We pooled Area Under the Receiver Operating Characteristic Curve (AUROC) values using random-effects models and assessed heterogeneity using Higgins’ I 2 . Results: We included 22 studies in our systematic review, validating Epic CDS tools on a total of over 2.3 million patients and 34 sites. The Epic Deterioration Index (EDI, 7 studies, 542,983 patients) achieved pooled AUROC 0.79 [95% CI 0.76–0.80]; Epic Sepsis Model (ESM, 3 studies, 922,754 patients) 0.65 [0.61–0.70]; Epic Unplanned Readmission Model (EURM, 5 studies, 145,595 patients) 0.70 [0.68–0.73]; Epic End-of-Life Care Index (EEOL-CI, 3 studies, 217,885 patients) 0.76 [0.67–0.83]; and Epic Risk of Patient No-Show (ERPNS, 2 studies, 93,863 patients) 0.62 [0.54–0.68]. All models showed high heterogeneity (I 2 ≥ 93%, p < 0.001). For ESM, EURM, and EEOL-CI, Epic’s reported confidence intervals did not overlap with our pooled estimates, with Epic reporting consistently higher performance. Two additional models (ED-to-Inpatients and ICU Mortality) had one study each. Conclusion: Epic’s CDS tools demonstrated modest real-world performance, with none exceeding AUROC 0.79. Three models (ESM, EURM, EEOL-CI) underperformed Epic’s reported ranges. High heterogeneity across sites emphasizes the need for local validation before clinical deployment.

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic review and meta-analysis found that externally validated Epic clinical decision support tools had modest real-world discrimination, with no pooled AUROC exceeding 0.79. For several models, Epic’s vendor-reported performance was higher than pooled external validation estimates, and heterogeneity was high across sites.

Key findings

  • Across 22 studies and over 2.3 million patients, pooled AUROCs were: Epic Deterioration Index (EDI) 0.79 (7 studies; 542,983 patients; abstract 95% CI 0.76–0.80, results 0.77–0.80); Epic Sepsis Model (ESM) 0.65 (3 studies; 922,754 patients; 0.61–0.70); Epic Unplanned Readmission Model (EURM) 0.70 (5 studies; 145,595 patients; 0.68–0.73); Epic End-of-Life Care Index (EEOL-CI) 0.76 (3 studies; 217,885 patients; 0.67–0.83); and Epic Risk of Patient No-Show (ERPNS) 0.62 (2 studies; 93,863 patients; abstract 0.54–0.68, results 0.61 [0.55–0.68]).
  • All five meta-analyzed models showed high heterogeneity (I² ≥ 93%, p < 0.001).
  • For ESM, EURM, and EEOL-CI, Epic’s reported confidence intervals did not overlap the pooled estimates, with Epic reporting consistently higher performance. Epic-reported values included ESM 95% CI 0.76–0.83, EURM 0.73 (0.73–0.74), EEOL-CI 0.89 (0.89–0.90), EDI 0.80 (0.75–0.84), and ERPNS 0.77 without a CI.
  • Two additional models had one study each: ED-to-Inpatients AUROC 0.84 (329,194 patients) and ICU Mortality AUROC 0.89 (63,775 patients).

Methods

  • Prospectively registered on PROSPERO (CRD420251148571); searched PubMed, Scopus, and Embase from January 2018 to August 2025 for external validations of Epic CDS tools, limited to English, following PRISMA and using Covidence.
  • Included peer-reviewed external validation studies of Epic proprietary predictive models; excluded duplicates, abstracts, articles without full text, preprints, editorials, locally developed models, studies without AUROC, and studies focused on downstream interventions.
  • Data were extracted in duplicate; risk of bias assessed with PROBAST; random-effects meta-analysis performed in R 4.3.3 with metafor 4.8.0, pooling AUROC and 95% CI, testing heterogeneity with Cochran’s Q and quantifying with Higgins’ I²; meta-analysis conducted for models with more than one article.

Limitations

  • The authors state significant heterogeneity across studies, meaning the average AUROC is not necessarily reflective of performance at a particular site; they also note AUROC does not account for operating threshold, calibration metrics were not included because most studies did not report them, publication bias is possible, and excluding abstracts or papers without AUROC could inflate pooled estimates.
  • The authors note risk-of-bias concerns in some included studies, including high selection bias in Singh 2021 and Cummings 2021 (COVID) for EDI, and significant concerns in Hwang 2021, Hwang 2023, and Drew 2023 for EURM.
  • Evident from the text: the abstract and results report different site counts (34 vs. 40) and different confidence intervals for EDI and ERPNS, though the authors do not discuss these discrepancies as limitations.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11606-026-10381-y

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

22 of 32 cited sources checked · 9 supported · 1 flagged · accuracy not yet measured

  • Study design and conductreasoning

    Cummings 2021 (COVID) only included patients with COVID-19.

    “Similarly, Cummings 2021 3 (COVID) only included patients with COVID-19.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [3] Cummings BC, Ansari S, Motyka JR, et al. Predicting intensive care transfers and other unforeseen events: Analytic model validation study and comparison to existing methods. JMIR Med Inform. 2021;9(4). https://doi.org/10.2196/25066. source

      The Introduction states that the study trained and tuned PICTURE on a large non–COVID-19 cohort and validated it in both non–COVID-19 patients and patients with COVID-19. The Methods > Setting and Study Population section identifies the COVID-19 patients as one of four cohorts, not the only patients included. Page numbers are not available in the supplied copy. (LLM, conf=0.91)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Interpretationreasoning

    Hwang 2021, conducted at Swiss Hospitals with differences in demographic populations and disease prevalence, may not reflect performance in the United States.

    “Hwang 2021 27 is a validation study conducted at Swiss Hospitals with innate differences amongst demographic populations and disease prevalence, and as such, results of this validation study may not reflect performance in the United States.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [27] Hwang AB, Schuepfer G, Pietrini M, Boes S. External Validation of EPIC’s Risk of Unplanned Readmission Model, the LACE+ Index and SQLape as Predictors of Unplanned Hospital Readmissions: A Monocentric, Retrospective, Diagnostic Cohort Study in Switzerland. Vol 16. 2021. https://doi.org/10.1371/journal.pone.0258338. source

      The Discussion (Limitations; page not available) describes a single-center Swiss validation study and warns that its findings may not generalize to organizations outside Switzerland, which supports the caution about U.S. performance. It does not establish differences in demographics or disease prevalence between its patients and U.S. patients; the claim also calls the setting 'Swiss Hospitals' although the validation was conducted at one hospital. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Study design and conductreasoning

    Singh 2021 had a high risk of selection bias due to restricting validation to COVID-19 patients.

    “Singh 2021 21 had a high risk of selection bias due to restricting validation to COVID-19 patients.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [21] Singh K, Valley TS, Tang S, et al. Evaluating a widely implemented proprietary deterioration index model among hospitalized patients with COVID-19. Ann Am Thorac Soc. 2021;18(7):1129-1137. https://doi.org/10.1513/AnnalsATS.202006-698OC. source

      The Methods (Study Cohort) restrict the cohort to adults hospitalized with COVID-19. The Discussion says thresholds may differ for patients hospitalized with other conditions and calls for broader validation, but it does not assess this restriction as a high risk of selection bias. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • Findingcitation

    The Epic ICU Mortality model had an AUROC of 0.89 (95% CI 0.89-0.89).

    “Tan 2020 47 Mayo Clinic Prospective ICU 63,775 3.5% 0.89 (0.89, 0.89)”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [47] Tan EM, Kashyap R, Olson IC, O’Horo JC. Validation of a Retrospective Computing Model for Mortality Risk in the Intensive Care Unit. Mayo Clin Proc Innov Qual Outcomes. 2020;4(5):575-582. https://doi.org/10.1016/j.mayocpiqo.2020.09.001 source

      The Abstract Results report an AUROC of 89.7% (95% CI, 89.5%–89.9%) for Epic’s ICU mortality risk model, not the stated 0.89 (0.89–0.89). The Abstract Methods also describe a retrospective cohort study, whereas the citing row labels it prospective. (LLM, conf=0.90)

      Experimental — accuracy not yet measured
  • Study design and conductcitation

    Drew 2023 had significant concerns with regard to patient selection because only patients receiving outpatient parenteral antibiotics were included.

    “Similar to Hwang 2023, 28 Drew 2023 26 also had significant concerns with regard to patient selection; only patients receiving outpatient parenteral antibiotics were included.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [28] Hwang S, Urbanowicz R, Lynch S, et al. Toward Predicting 30-Day Readmission Among Oncology Patients: Identifying Timely and Actionable Risk Factors. JCO Clin Cancer Inform. 2023;(7):1-10. https://doi.org/10.1200/cci.22.00097. source

      Section 3.1 (page unavailable) describes a selected oncology cohort at one institute and excludes patients who moved away, died within 30 days, or were readmitted elsewhere. This supports the existence of patient-selection restrictions, but Hwang does not characterize them as significant concerns. The outpatient parenteral antibiotic restriction is attributed to Drew, not Hwang. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [26] Drew R, Brenneman E, Funaro J, et al. Electronic health record-based readmission risk model performance for patients undergoing outpatient parenteral antibiotic therapy (OPAT). PLOS Digit Health. 2023;2(8):e0000323. https://doi.org/10.1371/journal.pdig.0000323. source
  • May combine two claims. 20
  • Classified as probably not a substantial claim. 20

  • May combine two claims.

    Premisecitation

    For the EDI, using a threshold of 6, Wong 2021 reports a PPV of 12% while Kamran 2024 reports a PPV of 8%, despite similar AUROC values (0.63 and 0.62).

    “For example, using a threshold of 6 for the EDI, Wong 2021 35 reports a PPV of 12% while Kamran 2024 36 reports a PPV of 8% despite having similar AUROC values (0.63 and 0.62).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Wong 2021 source

      The Results (ESM Performance) report a positive predictive value of 12% at a score threshold of 6 and a hospitalization-level AUC of 0.63. However, Wong et al. evaluated the Epic Sepsis Model (ESM), not the EDI. The Kamran 2024 figures are not attributed to Wong and are outside this verdict. (LLM, conf=0.91)

      Experimental — accuracy not yet measured
    • cites Kamran 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Epic Systems stores more than 325 million patient records with a 41.3% inpatient market share as of 2025.

    “Epic Systems, a leading provider in the EHR industry, stores more than 325 million patient records with a 41.3% inpatient market share as of 2025.[8][9][10]”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites [8] About | Epic. https://www.epic.com/about/. Accessed 29 Dec 2025.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [9] Discover the Most Common EHR Systems in Hospitals. https://www.definitivehc.com/blog/most-common-inpatient-ehr-systems. Accessed 29 Dec 2025.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [10] Waxler S, Blazek P, White D, et al. Generative Medical Event Models Improve with Scale. arXiv. Preprint posted online November 7, 2025:arXiv:2508.12104. https://doi.org/10.48550/arXiv.2508.12104. source

      The Introduction (p. 1) says Epic Cosmos aggregates records for more than 300 million patients as of August 2025, supporting the general point that Epic-associated systems hold records at this scale. It does not establish more than 325 million records or a 41.3% inpatient market share; the inpatient percentage in Data Pre-Processing (p. 21) describes encounter types, not market share. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    CDS tool performance has been shown to drift when validated in hospitals and regions outside of the development sites and over time.

    “Importantly, CDS tool performance has been shown to drift when validated in hospitals and regions outside of the development sites 14,15 and over time.16,17”
    Quote in the paperSubstantial claim4 of 4 sources checked
    • cites [14] van Klaveren D, Zanos TP, Nelson J, et al. Prognostic models for COVID-19 needed updating to warrant transportability over time and space. BMC Med. 2022;20(1):1-11. https://doi.org/10.1186/s12916-022-02651-3. source
    • cites [15] Wessler BS, Nelson J, Park JG, et al. External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature. Circ Cardiovasc Qual Outcomes. 2021;14(8):E007858. https://doi.org/10.1161/CIRCOUTCOMES.121.007858. source
    • cites [16] Davis SE, Greevy RA, Lasko TA, Walsh CG, Matheny ME. Detection of calibration drift in clinical prediction models to inform model updating. J Biomed Inform. 2020;112:103611. https://doi.org/10.1016/J.JBI.2020.103611. source
    • cites [17] Levy TJ, Coppa K, Cang J, et al. Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients. Nat Commun 2022 131. 2022;13(1):1–14. https://doi.org/10.1038/s41467-022-34646-2. source

  • May combine two claims.

    Premisecitation

    False positive rates can contribute to unnecessary diagnostics and antimicrobial use, potentially avoidable ICU transfers, and alert fatigue.

    “False positive rates, on the other hand, can contribute to unnecessary diagnostics and antimicrobial use, potentially avoidable intensive care unit (ICU) transfers, and alert fatigue.6,7”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [6] Bedoya AD, Clement ME, Phelan M, Steorts RC, O’Brien C, Goldstein BA. Minimal Impact of Implemented Early Warning Score and Best Practice Alert for Patient Deterioration. Crit Care Med. 2019;47(1):49-55. https://doi.org/10.1097/CCM.0000000000003439. source

      The Discussion reports that alerts requiring no clinical intervention contributed to alert fatigue. It does not report that false positives caused unnecessary diagnostics, antimicrobial use, or avoidable ICU transfers; the Results report no change in ICU transfer rates after implementation. Page numbers are not available in our copy. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [7] Fallon A, Haralambides K, Mazzillo J, Gleber C. Addressing Alert Fatigue by Replacing a Burdensome Interruptive Alert with Passive Clinical Decision Support. Appl Clin Inform. 2024;15(1):101-110. https://doi.org/10.1055/a-2226-8144. source

      The Background and Significance section says unnecessary interruptive alerts pose a risk of alert fatigue, and the Abstract reports that redesigning a frequently firing COVID alert reduced alert burden. The paper does not link false-positive rates to unnecessary diagnostics, antimicrobial use, or avoidable ICU transfers. Page numbers were not available for these passages. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Study design and conductreasoning

    Hwang 2023 is a validation study only including oncology patients, introducing significant bias with regard to patient selection.

    “Hwang 2023 28 is a validation study only including oncology patients, introducing significant bias with regard to patient selection.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [28] Hwang S, Urbanowicz R, Lynch S, et al. Toward Predicting 30-Day Readmission Among Oncology Patients: Identifying Timely and Actionable Risk Factors. JCO Clin Cancer Inform. 2023;(7):1-10. https://doi.org/10.1200/cci.22.00097. source

      The Cohort and Dataset section describes a cohort restricted to oncology patients, supporting that part of the claim. But the Conclusions say the authors developed and validated models, so this was not a validation-only study. The Limitations section notes that the models were not evaluated at another hospital; it does not establish significant patient-selection bias. Our copy provides no page numbers for these passages. (LLM, conf=0.89)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    For the EEOL-CI, at a threshold of 45, Gensheimer 2025 reports sensitivity, specificity, and PPV of 0.63, 0.70, and 0.38, while Frechman 2025 reports 0.23, 0.98, and 0.22.

    “In the EEOL-CI, for a threshold of 45, Gensheimer 2025 37 reports a sensitivity, specificity, and PPV of 0.63, 0.70, and 0.38, respectively, while Frechman 2025 38 reports 0.23, 0.98, and 0.22, respectively.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Gensheimer 2025 source
    • cites Frechman 2025 source

  • May combine two claims.

    Premisecitation

    CDS tools directly influence patient care, with sepsis alerts prompting earlier antibiotic administration and fluid resuscitation, and deterioration scores used to trigger rapid response evaluations, intensified monitoring, and escalation of care.

    “These tools directly influence patient care, with sepsis alerts prompting earlier antibiotic administration and fluid resuscitation, and deterioration scores used to trigger rapid response evaluations, intensified monitoring, and escalation of care.[3][4][5]”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [3] Cummings BC, Ansari S, Motyka JR, et al. Predicting intensive care transfers and other unforeseen events: Analytic model validation study and comparison to existing methods. JMIR Med Inform. 2021;9(4). https://doi.org/10.2196/25066. source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites [4] Cummings BC, Blackmer JM, Motyka JR, et al. External Validation and Comparison of a General Ward Deterioration Index between Diversely Different Health Systems. Crit Care Med. 2023;51(6):775-786. https://doi.org/10.1097/CCM.0000000000005837. source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites [5] Liu VX, Fielding-Singh V, Greene JD, et al. The Timing of Early Antibiotics and Hospital Mortality in Sepsis. Am J Respir Crit Care Med. 2017;196(7):856-863. https://doi.org/10.1164/rccm.201609-1848OC. source

      Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    In a general patient population like Frechman 2025, high specificity (0.98) may be desirable to avoid false positives, while in a high-risk mortality population like Gensheimer 2025 (metastatic cancer), higher sensitivity (0.63) may be favored.

    “In a general patient population like Frechman 2025, 38 a high specificity (0.98) may be desirable to avoid false positives, while Gensheimer 2025 37 was in a population with a high risk of mortality (metastatic cancer) and may favor a higher sensitivity (0.63).”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Frechman 2025 source
    • cites Gensheimer 2025 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Database screening followed by deletion of duplicate results yielded 2348 studies.

    “Database screening followed by deletion of duplicate results yielded 2348 studies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Most studies in our analysis did not report calibration metrics, so we were unable to include them.

    “However, most studies in our analysis did not report these metrics, so we were unable to include them.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For ESM, EURM, and EEOL-CI, Epic's reported confidence intervals did not overlap with the pooled estimates, with Epic reporting consistently higher performance.

    “For ESM, EURM, and EEOL-CI, Epic's reported confidence intervals did not overlap with our pooled estimates, with Epic reporting consistently higher performance.”
    • Also stated in: § DISCUSSION · plain
      “For three models (ESM, EURM, and EEOL-CI), Epic's reported CI does not overlap with ours, with Epic's intervals being consistently higher than our pooled estimates.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Epic reported an ERPNS AUROC of 0.77 without a CI.

    “Epic reported an AUROC of 0.77 without a CI.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] Galaxy - Cognitive Computing Model Brief: Risk of Patient No-Show (Version 1). https://galaxy.epic.com/?#Browse/page=1!68!50!3530079&from=Galaxy-Redirect. (Requires institutional access). Accessed 29 Dec 2025.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Epic reported EEOL-CI AUROC at three different sites with a pooled AUROC of 0.89 (95% CI 0.89-0.90).

    “Epic reported AUROC at three different sites with a pooled AUROC of 0.89 [0.89-0.90].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29] Galaxy - Cognitive Computing Model Brief: End of Life Care Index. https://galaxy.epic.com/?#Browse/page=1!68!95!100039705&from=Galaxy-Redirect. (Requires institutional access). Accessed 29 Dec 2025.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The authors systematically searched PubMed, Scopus, and Embase from January 2018 to August 2025 for external validations of Epic's CDS tools.

    “We systematically searched PubMed, Scopus, and Embase (January 2018 to August 2025) for external validations of Epic's CDS tools.”
    • Also stated in: § Search Strategy · plain
      “In collaboration with an information specialist, J.M., we developed a search strategy and queried the PubMed, Scopus, and Embase databases for articles related to validated proprietary Epic CDS tools from January 2018 to August 2025 (Supplementary Appendix A).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The review protocol was prospectively registered on PROSPERO (CRD420251148571).

    “This study was prospectively registered on PROSPERO (CRD420251148571).”
    • Also stated in: § Search Strategy · plain
      “The review protocol was registered in PROSPERO (CRD420251148571).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Epic reported EDI AUROC at three different sites with a pooled AUROC of 0.80 (95% CI 0.75-0.84).

    “Epic reported AUROC for the EDI at three different sites with a pooled AUROC of 0.80 [95% CI, 0.75-0.84].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] Galaxy - Cognitive Computing Model Brief: Deterioration Index. https://galaxy.epic.com/?#Browse/page=1!68!50!3883949&from=Galaxy-Redirect. (Requires. institutional access). Accessed 29 Dec 2025.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    The net benefit of two models with the same AUROC can differ significantly based on the ROC curve and operating threshold, and AUROC does not reflect differences in performance in patient populations with different prevalences.

    “The net benefit of two models with the same AUROC can differ significantly based on the ROC curve and operating threshold, and AUROC does not reflect differences in performance in patient populations with different prevalences (e.g., a lower prevalence will yield more 9).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Four independent reviewers screened article titles and abstracts.

    “Four independent reviewers (H.P., S.C., J.M., D.H.) screened article titles and abstracts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Model calibration metrics (slope, Brier score) can provide a more holistic picture of a model's performance.

    “Model calibration metrics (slope, Brier score) can provide a more holistic picture of a model's performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Data collection was performed in duplicate by two reviewers, with a third reviewer adjudicating discrepancies.

    “Data collection (coding/extraction of AUROC and secondary metrics) was performed in duplicate by two reviewers (H.P. and S.C.), with a third reviewer (D.H.) adjudicating discrepancies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Epic's version 2 development of the ESM recommends localizing the algorithm on at least 35,000 hospital or emergency department encounters.

    “Notably, with Epic's version 2 development of the ESM, they recommend localizing the algorithm on at least 35,000 hospital or emergency department encounters.34”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [34] Galaxy - Cognitive Computing Model Brief: Early Detection of Sepsis (Version 2). https://galaxy.epic.com/?#Browse/page=1!68!50!100170819&from=Galaxy-Redirect. (Requires institutional access). Accessed 12 Sept 2025.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The suitability of operating points depends on the benefit and risk trade-off.

    “The suitability of these operating points also depends on the benefit and risk trade-off.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    For many clinical use cases, especially high-stakes applications like sepsis detection or ICU transfer, this level of discrimination may result in excessive false positives or missed cases, potentially limiting clinical utility.

    “For many clinical use cases, especially high-stakes applications like sepsis detection or ICU transfer, this level of discrimination may result in excessive false positives or missed cases, potentially limiting clinical utility.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The authors used Covidence systematic review software to adhere to PRISMA guidelines throughout the systematic review and meta-analysis.

    “We utilized the Covidence systematic review software to strictly adhere to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines 18 throughout the systematic review and meta-analysis.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] PRISMA 2020 checklist. PRISMA statement. https://www.prisma-statement.org/prisma-2020-checklist. Accessed 6 Jan 2026.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Full-text review was performed in duplicate by at least two reviewers.

    “Full-text review was performed in duplicate by at least two reviewers (H.P., S.C., and D.H.).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Epic reported their Sepsis v1 model achieved an AUROC between 0.76 and 0.83.

    “Epic reported their Sepsis v1 model achieved an AUROC between 0.76 and 0.83.12”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] Galaxy - Cognitive Computing Model Brief: Early Detection of Sepsis (Version 1). https://galaxy.epic.com/?#Browse/page=1!68!50!3289911&from=Galaxy-Redirect (Requires institutional access).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    High heterogeneity among studies and the impact of threshold selection mean hospitals must carefully tailor models and operating points for optimal patient care.

    “High heterogeneity among studies and the impact of threshold selection mean hospitals must carefully tailor models and operating points for optimal patient care.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Epic provides general guidelines on how to set an operating threshold that a hospital system can be comfortable with.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    A study was considered low risk only when all four PROBAST domains showed low risk.

    “We considered a study low risk only when all four domains showed low risk.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Two reviewers were required to agree for inclusion at each stage.

    “Two reviewers were required to agree for inclusion at each stage.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Risk of bias was assessed using the PROBAST tool.

    “We assessed risk of bias using the Prediction model Risk Of Bias ASsessment (PROBAST) tool”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Local validation and customization to specific populations are needed for Epic's CDS models.

    “In summary, Epic's CDS models showed lower and highly variable AUROC performance in real-world validation studies than their own reported metrics, emphasizing the need for local validation and customization to specific populations.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Two reviewers independently evaluated each study and rated domains as low, high, or unclear risk.

    “Two reviewers (H.P. and S.C.) independently evaluated each study and rated domains as low, high, or unclear risk.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given

    AUROC does not depend on the operating threshold, so it can be used to compare discriminative ability across many studies.

    “We selected AUROC because it does not depend on the operating threshold, so it can be used to compare discriminative ability across many studies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Studies with null or unfavorable performance may be less likely to be published or to report AUROC, and our exclusion of abstracts and papers that did not report AUROC could preferentially remove such studies, potentially inflating pooled estimates.

    “Studies with null or unfavorable performance may be less likely to be published or to report AUROC, and our exclusion of abstracts and papers that did not report AUROC could preferentially remove such studies, potentially inflating pooled estimates.”
    Quote in the paperNot a substantial claim
34 of 34

  • The paper relies on this without support.

    AUROC performance can vary significantly when models are run on populations that differ from the groups they were trained on.

    “However, performance metrics such as AUROC can vary significantly when the models are run on populations that differ from the groups they were trained on.”

  • The paper relies on this without support.

    Epic trains these models on data from a small number of hospitals.

    “Epic trains these models on data from a small number of hospitals.”

  • The paper relies on this without support.

    Epic provides general guidelines on how to set an operating threshold that a hospital system can be comfortable with.

    “Epic provides general guidelines on how to set an operating threshold that a hospital system can be comfortable with.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • The review reports both 34 and 40 sites

    Major

    The abstract reports 34 sites, while the study characteristics section reports 40 sites for the included studies.

  • The EDI pooled confidence interval differs between sections

    Minor

    The abstract gives the EDI interval as 0.76-0.80, but the results give 0.77-0.80 for the same pooled estimate.

  • The no-show model's pooled estimate differs between sections

    Minor

    The abstract reports an AUROC of 0.62 with a lower confidence limit of 0.54; the results report 0.61 and 0.55, respectively.

  • The threshold example names the wrong model

    Minor

    The discussion calls Wong 2021 and Kamran 2024 examples of EDI performance, whereas Table 9 lists both under Sepsis.

  • Two included models exceed the stated maximum AUROC

    Minor

    The conclusion says none of Epic's CDS tools exceeded an AUROC of 0.79, but the included single-study validations report 0.84 for ED-to-Inpatient and 0.89 for ICU Mortality.

5 of 5

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Patel, H., Crusco, S., Hansen, D., Morales, J., Patale, T., Chiuzan, C., Sands, M., Naidich, J., Myers, K., Davidson, K. W., Hirsch, J. S., & Zanos, T. P.
Affiliations
Northwell Health (US)
Corresponding author
Zanos, Theodoros P.
ORCID
Venue
Journal of General Internal Medicine
Publisher
Springer Science and Business Media LLC
Published
2026-03-31
ISSN
  • 0884-8734 (ISSN-L)
  • 1525-1497
DOI
10.1007/s11606-026-10381-y
OpenAlex
W7147754297
Cited by
4
References
38
Citations by year
2026: 4
FWCI
9.16
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of General Internal Medicine (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
National Institute of Nursing Research (R01NR020774)
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
clinical decision support, Epic Systems, external validation, electronic health records, prediction model, end-of-life care, real-world performance, area under the receiver operating characteristic, ICU mortality prediction, local validation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-27.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-27.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review