Comparison of 1-year mortality predictions from vendor-supplied versus academic model for cancer patients
Open accessAbstract
Purpose: The Epic End of Life Care Index (EOLCI) predicts 1-year mortality for a general adult population using medical record data. It is deployed at various medical centers, but we are not aware of an independent validation. We evaluated its performance for predicting 1-year mortality in patients with metastatic cancer, comparing it against an academic machine learning model designed for cancer patients. We focused on this patient population because of their high short-term mortality risk and because we had access to the comparator model predictions. Materials and Methods: This retrospective analysis included adult outpatients with metastatic cancer from four outpatient sites. Performance metrics included AUC for 1-year mortality and positive predictive value of high-risk score. Results: There were 1,399 patients included. Median age at first EOLCI prediction was 67 and 55% were female. A total of 1,283 patients were evaluable for 1-year mortality; of these, 297 (23%) died within 1 year. AUC for 1-year mortality for EOLCI and academic model was 0.73 (95% CI [0.70–0.76]) and 0.82 (95% CI [0.80–0.85]), respectively. Positive predictive value was 0.38 and 0.65, respectively. Conclusion: The EOLCI’s discrimination performance was lower than the vendor-stated value (AUC of 0.86) and the academic model’s performance. Vendor-supplied machine learning models should be independently validated, particularly in specialized patient populations, to ensure accuracy and reliability.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For the EEOL-CI, at a threshold of 45, Gensheimer 2025 reports sensitivity, specificity, and PPV of 0.63, 0.70, and 0.38, while Frechman 2025 reports 0.23, 0.98, and 0.22.
“In the EEOL-CI, for a threshold of 45, Gensheimer 2025 37 reports a sensitivity, specificity, and PPV of 0.63, 0.70, and 0.38, respectively, while Frechman 2025 38 reports 0.23, 0.98, and 0.22, respectively.”
From A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools (2026) · cites this paper as Gensheimer 2025
In a general patient population like Frechman 2025, high specificity (0.98) may be desirable to avoid false positives, while in a high-risk mortality population like Gensheimer 2025 (metastatic cancer), higher sensitivity (0.63) may be favored.
“In a general patient population like Frechman 2025, 38 a high specificity (0.98) may be desirable to avoid false positives, while Gensheimer 2025 37 was in a population with a high risk of mortality (metastatic cancer) and may favor a higher sensitivity (0.63).”
From A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools (2026) · cites this paper as Gensheimer 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 33 · Cited by 5
Research works printed but not linked: 27
- Bestvina CM Polite BN Implementation of advance care planning in oncology: a review of the literature Journal of Oncology Practice 2017 13 10 657 662 10.1200/JOP.2017.021246 28586259 · DOI
- Buchan TA Ross HJ McDonald M Billia F Delgado D Duero Posada JG Luk A Guyatt GH Alba AC Physician judgement vs model-predicted prognosis in patients with heart failure Canadian Journal of Cardiology 2020 36 1 84 91 10.1016/j.cjca.2019.07.623 31735429 · DOI
- Byrd TF Southwell B Ravishankar A Tran T Kc A Phelan T Melton-Meaux GB Usher MG Scheppmann D Switzer S Simon G Tignanelli CJ Validation of a proprietary deterioration index model and performance in hospitalized adults JAMA Network Open 2023 6 7 e2324176 10.1001/jamanetworkopen.2023.24176 37486632 PMC10366696 · DOI
- Curtis MD Griffith SD Tucker M Taylor MD Capra WB Carrigan G Holzman B Torres AZ You P Arnieri B Abernethy AP Development and validation of a high-quality composite real-world mortality endpoint Health Services Research 2018 53 6 4460 4476 10.1111/1475-6773.12872 29756355 PMC6232402 · DOI
- Gann PH Daviglus ML Dyer AR Stamler J Heart rate and prostate cancer mortality: results of a prospective analysis Cancer Epidemiology Biomarkers Prevention 1995 4 6 611 616 8547827
- Gensheimer MF Aggarwal S Benson KRK Carter JN Henry AS Wood DJ Soltys SG Hancock S Pollom E Shah NH Chang DT Automated model versus treating physician for predicting survival time of patients with metastatic cancer Journal of the American Medical Informatics Association 2021 28 6 1108 1116 10.1093/jamia/ocaa290 33313792 PMC8200275 · DOI
- Gensheimer MF Gupta D Patel MI Fardeen T Hildebrand R Teuteberg W Seevaratnam B Asuncion MK Alves N Rogers B Hansen J DeNofrio J Shah NH Parikh D Neal J Fan AC Moore K Ruiz S Li C Khaki AR Pagtama J Chien J Brown T Tisch AH Das M Srinivas S Roy M Wakelee H Myall NJ Huang J Shah S Lee H Ramchandran K Use of machine learning and lay care coaches to increase advance care planning conversations for patients with metastatic cancer JCO Oncology Practice 2023 19 2 e176 e184 10.1200/OP.22.00128 36395436 PMC12360162 · DOI
- Gensheimer MF Henry AS Wood DJ Hastie TJ Aggarwal S Dudley SA Pradhan P Banerjee I Cho E Ramchandran K Pollom E Koong AC Rubin DL Chang DT Automated survival prediction in metastatic cancer patients using high-dimensional electronic medical record data JNCI: Journal of the National Cancer Institute 2019 111 6 568 574 10.1093/jnci/djy178 30346554 PMC6579743 · DOI
- Gensheimer MF Narasimhan B A scalable discrete-time survival model for neural networks PeerJ 2019 7 8 e6257 10.7717/peerj.6257 30701130 PMC6348952 · DOI
- Gensheimer MF Narasimhan B Henry AS Wood DJ Rubin DL Accuracy of electronic medical record follow-up data for estimating the survival time of patients with cancer JCO Clinical Cancer Informatics 2022 6 e2200019 10.1200/CCI.22.00019 35802836 PMC9296186 · DOI
- Gensheimer MF Teuteberg W Patel MI Gupta D Noroozi M Ling X Fardeen T Seevaratnam B Lu Y Alves N Rogers B Asuncion MK DeNofrio J Hansen J Shah NH Chen T Cabebe E Blayney DW Colevas AD Ramchandran K Automated patient selection and care coaches to increase advance care planning for patients with cancer JNCI: Journal of the National Cancer Institute 2024 30 djae243 10.1093/jnci/djae243 PMC12990145 39348179 · DOI
- Gervasi SS Chen IY Smith-McLallen A Sontag D Obermeyer Z Vennera M Chawla R The potential for bias in machine learning and opportunities for health insurers to address it: article examines the potential for bias in machine learning and opportunities for health insurers to address it Health Affairs 2022 41 2 212 218 10.1377/hlthaff.2021.01287 35130064 · DOI
- Grandhige AP Mortality risk predictive index to prompt earlier palliative medicine involvement Journal of Pain and Symptom Management 2024 67 5 e517 10.1016/j.jpainsymman.2024.02.283 · DOI
- Lee DH Park S Lim SM Lee MK Giovannucci EL Kim JH Kim SI Jeon JY Resting heart rate as a prognostic factor for mortality in patients with breast cancer Breast Cancer Research and Treatment 2016 159 2 375 384 10.1007/s10549-016-3938-1 27544225 · DOI · Paper page
- Levin MA Lin H Prabhakar G McCormick PJ Egorova NN Alive or dead: validity of the social security administration death master file after 2011 Health Services Research 2019 54 1 24 33 10.1111/1475-6773.13069 30520023 PMC6338290 · DOI · Paper page
- Li RC Asch SM Shah NH Developing a delivery science for artificial intelligence in healthcare NPJ Digital Medicine 2020 3 1 107 10.1038/s41746-020-00318-y 32885053 PMC7443141 · DOI
- Lu J Sattler A Wang S Khaki AR Callahan A Fleming S Fong R Ehlert B Li RC Shieh L Ramchandran K Gensheimer MF Chobot S Pfohl S Li S Shum K Parikh N Desai P Seevaratnam B Hanson M Smith M Xu Y Gokhale A Lin S Pfeffer MA Teuteberg W Shah NH Considerations in the reliability and fairness audits of predictive models for advance care planning Frontiers in Digital Health 2022 4 943768 10.3389/fdgth.2022.943768 36339512 PMC9634737 · DOI
- McGinn T Putting meaning into meaningful use: a roadmap to successful integration of evidence at the point of care JMIR Medical Informatics 2016 4 2 e16 10.2196/medinform.4553 27199223 PMC4891572 · DOI
- Peralta AAA Desai P Datta S A study linking patient EHR data to external death data at Stanford Medicine 2022 https://arxiv.org/abs/2211.01488 · arXiv
- Raclin T Narasimhan B Henry S Wood D Rubin DL Gensheimer MF Use of EMR data to validate performance of machine learning prognostic models for patients with cancer JCO Oncology Practice 2023 19 11_Suppl 512 10.1200/OP.2023.19.11_suppl.512 · DOI
- Reps JM Rijnbeek P Cuthbert A Ryan PB Pratt N Schuemie M An empirical analysis of dealing with patients who are lost to follow-up when developing prognostic models using a cohort design BMC Medical Informatics and Decision Making 2021 21 1 43 10.1186/s12911-021-01408-x 33549087 PMC7866757 · DOI
- Sahiner B Chen W Samala RK Petrick N Data drift in medical machine learning: implications and potential remedies The British Journal of Radiology 2023 96 1150 20220878 10.1259/bjr.20220878 36971405 PMC10546450 · DOI · Paper page
- Shah NH Milstein A Bagley SC Making machine learning models clinically useful Journal of the American Medical Association 2019 322 14 1351 10.1001/jama.2019.10306 31393527 · DOI · Paper page
- Shah T Patel-Teague S Kroupa L Meyer AND Singh H Impact of a national QI programme on reducing electronic health record notifications to clinicians BMJ Quality & Safety 2019 28 1 10 14 10.1136/bmjqs-2017-007447 PMC6365918 29507122 · DOI
- Watson J Hutyra CA Clancy SM Chandiramani A Bedoya A Ilangovan K Nderitu N Poon EG Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers? JAMIA Open 2020 3 2 167 172 10.1093/jamiaopen/ooz046 32734155 PMC7382631 · DOI · Paper page
- Wu YC Lee WC Alternative performance measures for prediction models PLOS ONE 2014 9 3 e91249 10.1371/journal.pone.0091249 24608868 PMC3946724 · DOI
- Zhang A Xing L Zou J Wu JC Shifting machine learning for healthcare from development to deployment and from models to data Nature Biomedical Engineering 2022 6 12 1330 1345 10.1038/s41551-022-00898-y PMC12063568 35788685 · DOI · Paper page
- Authors
- Gensheimer, M. F., Lu, J., & Ramchandran, K.
- Affiliations
- Stanford Medicine (US)
- Corresponding author
- Gensheimer, Michael F.
- Venue
- PeerJ
- Publisher
- PeerJ
- Published
- 2025-02-11
- Volume
- 13
- Pages
- e18958
- ISSN
- 2167-8359 (ISSN-L)
- PMC
- PMC11827575
- OpenAlex
- W4407380525
- Cited by
- 5
- References
- 33
- Citations by year
- 2026: 4 · 2025: 1
- FWCI
- 4.52
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PeerJ (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Cancer Institute (5P30CA124435)
- National Institutes of Health/National Center for Research Resources (UL1 RR025744)
- Stanford University School of Medicine MedScholars
- Department of Medicine and Stanford Healthcare
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Palliative Care and End-of-Life Issues
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- 1-year mortality prediction, metastatic cancer, end-of-life care, model validation, model discrimination, positive predictive value
- MeSH terms
- Machine Learning, Adult, Aged, Aged, 80 and over, Female, Humans, Male, Middle Aged, Neoplasms, Retrospective Studies, Terminal Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
