Validation of a Retrospective Computing Model for Mortality Risk in the Intensive Care Unit
Open accessAbstract
Objective To compare the predictive performance of Epic Systems Corporation's proprietary intensive care unit (ICU) mortality risk model (IMRM) with that of the Acute Physiology and Chronic Health Evaluation (APACHE) IV score. Methods This is a retrospective cohort study of patients treated from January 1, 2008, through January 1, 2018. This single-center study was performed at Mayo Clinic (Rochester, MN), a tertiary care teaching and referral center. The primary outcome was death in the ICU. Discrimination of each risk model for hospital mortality was assessed by comparing area under the receiver operating characteristic curve (AUROC). Results The cohort mostly comprised older patients (median age, 64 years) and men (56.7%). The mortality rate of the cohort was 3.5% (2251 of 63,775 patients). The AUROC for mortality prediction was 89.7% (95% CI, 89.5% to 89.9%) for the IMRM, which was significantly greater than the AUROC of 88.2% (95% CI, 87.9% to 88.4%) for APACHE IV ( P <.001). Conclusion The IMRM was superior to the commonly used APACHE IV score and may be easily integrated into electronic health records at any hospital using Epic software.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Epic ICU Mortality model had an AUROC of 0.89 (95% CI 0.89-0.89).
“Tan 2020 47 Mayo Clinic Prospective ICU 63,775 3.5% 0.89 (0.89, 0.89)”
From A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools (2026) · cites this paper as [47]
The Abstract Results report an AUROC of 89.7% (95% CI, 89.5%–89.9%) for Epic’s ICU mortality risk model, not the stated 0.89 (0.89–0.89). The Abstract Methods also describe a retrospective cohort study, whereas the citing row labels it prospective. (LLM, conf=0.90)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · see citing works
ASSESSMENT OF COMA AND IMPAIRED CONSCIOUSNESS
Teasdale, G., & Jennett, B. (1974) · The Lancet · doi:10.1016/s0140-6736(74)91639-0 · Cited by 13,215 · cited 2× · see citing works
“6 for men 11,500 (6,600-16,300) 3,400-9,600 164,000 (112,000-223,000) 135,000-371,000”
BackgroundZimmerman, J. E., Kramer, A. A., McNair, D. S., & Malila, F. M. (2006) · Critical Care Medicine · doi:10.1097/01.ccm.0000215112.84523.f0 · Cited by 1,764 · see citing works
Moreno, R. P., Metnitz, P., Almeida, E., Jordan, B., Bauer, P., Campos, R. A., Iapichino, G., Edbrooke, D. L., Capuzzo, M., Gall, J. L., et al. (2005) · Intensive Care Medicine · doi:10.1007/s00134-005-2763-5 · Cited by 1,414 · see citing works
Lemeshow, S. A. (1993) · JAMA · doi:10.1001/jama.1993.03510200084037 · Cited by 1,030 · see citing works
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312 · see citing works
Multidisciplinary Management of Sedation and Analgesia in Critical Care
Sessler, C. N., Grap, M. J., & Brophy, G. M. (2001) · Seminars in Respiratory and Critical Care Medicine · doi:10.1055/s-2001-13834 · Cited by 140 · see citing works
Bennett, C. E., Wright, R. S., Jentzer, J. C., Gajic, O., Murphree, D. H., Murphy, J. G., Mankad, S. V., Wiley, B. M., Bell, M. R., & Barsness, G. W. (2018) · Journal of Critical Care · doi:10.1016/j.jcrc.2018.12.012 · Cited by 115 · see citing works
Cowen, M. E., Dusseau, D. J., Toth, B. G., Guisinger, C., Zodet, M. W., & Shyr, Y. (1998) · Medical Care · doi:10.1097/00005650-199807000-00016 · Cited by 110 · see citing works
Kashyap, R., Singh, T. D., Rayes, H., O’Horo, J. C., Wilson, G. A., Bauer, P. R., & Gajic, O. (2019) · Journal of Critical Care · doi:10.1016/j.jcrc.2019.01.005 · Cited by 23 · see citing works
Self-citationSmischney, N. J., Velagapudi, V. M., Onigkeit, J. A., Pickering, B. W., Herasevich, V., & Kashyap, R. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-55 · Cited by 19 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- [2]Kuhn M. Johnson K. Applied Predictive Modeling 2013 Springer New York, NY
- [3]Bach F.R. Bolasso: model consistent lasso estimation through the bootstrap. Paper presented at: Proceedings of the 25th International Conference on Machine Learning July 5-9, 2008 Finland Helsinki
Cited by
Sorted by citations
A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools
Patel, H., Crusco, S., Hansen, D., Morales, J., Patale, T., Chiuzan, C., Sands, M., Naidich, J., Myers, K., Davidson, K. W., et al. (2026) · Journal of General Internal Medicine · doi:10.1007/s11606-026-10381-y · Cited by 4 · see citing works
- Authors
- Tan, E. M., Kashyap, R., Olson, I. C., & O’Horo, J. C.
- Affiliations
- Mayo Clinic (US)
- Epic Systems (United States) (US)
- Corresponding author
- O’Horo, John C.
- Venue
- Mayo Clinic Proceedings: Innovations, Quality & Outcomes
- Publisher
- Elsevier BV
- Published
- 2020-10
- Volume
- 4
- Issue
- 5
- Pages
- 575-582
- ISSN
- 2542-4548 (ISSN-L)
- PMC
- PMC7560567
- OpenAlex
- W3091867920
- Cited by
- 5
- References
- 11
- Citations by year
- 2026: 2 · 2025: 1 · 2021: 2
- FWCI
- 0.36
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Mayo Clinic Proceedings Innovations Quality & Outcomes (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Mayo Clinic Critical Care Independent Multidisciplinary Program (IMP) Research Subcommittee
- CCaTS (UL1 TR000135)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Endocrinology, Diabetes and Metabolism › Hyperglycemia and glycemic control in critically ill and hospitalized patients
- Keywords
- ICU mortality, mortality prediction, APACHE IV score, electronic health records, model discrimination, critically ill patients, prediction model validation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
