External validation of a proprietary risk model for 1-year mortality in community-dwelling adults aged 65 years or older
Open accessAbstract
Objective: To examine the discrimination, calibration, and algorithmic fairness of the Epic End of Life Care Index (EOL-CI). Materials and Methods: We assessed the EOL-CI’s performance by estimating area under the receiver operating characteristic curve (AUC), sensitivity, and positive and negative predictive values in community-dwelling adults ≥65 years of age in a single health system in the Southeastern United States. Algorithmic fairness was examined by comparing the model’s performance across sex, race, and ethnicity subgroups. Using a machine learning approach, we also explored local re-calibration of the EOL-CI considering additional information on past hospitalizations and frailty. Results: Among 215 731 patients (median age = 74 years, 57% female, 12% of Black race), 10% were classified as medium risk (15-44) and 3% as high risk (≥45) by the EOL-CI. The observed 1-year mortality rate was 3%. The EOL-CI had an AUC 0.82 for 1-year mortality, with a positive predictive value of 22%. Predictive performance was generally similar across sex and race subgroups, though the EOL-CI displayed better performance with increasing age and in older adults with 2 or more outpatient encounters in the past 24 months. Local re-calibration of the EOL-CI was required to provide absolute estimates of mortality risk, and calibration was further improved when the EOL-CI was augmented with data on inpatient hospitalizations and frailty. Discussion: The EOL-CI demonstrates reasonable discrimination, albeit with better performance in older adults and in those with greater health system contact. Conclusion: Local refinement and calibration of the EOL-CI score is required to provide direct estimates of prognosis, with the goal of making the EOL-CI a more a valuable tool at the point of care for identifying patients who would benefit from targeted palliative care interventions and proactive care planning.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In a general patient population like Frechman 2025, high specificity (0.98) may be desirable to avoid false positives, while in a high-risk mortality population like Gensheimer 2025 (metastatic cancer), higher sensitivity (0.63) may be favored.
“In a general patient population like Frechman 2025, 38 a high specificity (0.98) may be desirable to avoid false positives, while Gensheimer 2025 37 was in a population with a high risk of mortality (metastatic cancer) and may favor a higher sensitivity (0.63).”
From A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools (2026) · cites this paper as Frechman 2025
For the EEOL-CI, at a threshold of 45, Gensheimer 2025 reports sensitivity, specificity, and PPV of 0.63, 0.70, and 0.38, while Frechman 2025 reports 0.23, 0.98, and 0.22.
“In the EEOL-CI, for a threshold of 45, Gensheimer 2025 37 reports a sensitivity, specificity, and PPV of 0.63, 0.70, and 0.38, respectively, while Frechman 2025 38 reports 0.23, 0.98, and 0.22, respectively.”
From A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools (2026) · cites this paper as Frechman 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · see citing works
Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data
Quan, H., Sundararajan, V., Halfon, P., Fong, A., Burnand, B., Luthi, J.-C., Saunders, L. D., Beck, C. A., Feasby, T. E., & Ghali, W. A. (2005) · Medical Care · doi:10.1097/01.mlr.0000182534.19832.83 · Cited by 10,669 · see citing works
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1× · see citing works
“51 Fourth, another potential limitation is the risk of data drift.”
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718 · cited 1× · see citing works
“25 The study was approved by the institutional review board at Wake Forest University Health Sciences (IRB00109077) under a waiver of HIPAA authorization.”
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
Precision medicine, AI, and the future of personalized health care
Johnson, K. B., Wei, W., Weeraratne, D., Frisse, M. E., Misulis, K., Rhee, K., Zhao, J., & Snowdon, J. L. (2021) · Clinical and Translational Science · doi:10.1111/cts.12884 · Cited by 1,854 · see citing works
Christakis, N. A. (2000) · BMJ · doi:10.1136/bmj.320.7233.469 · Cited by 1,425 · see citing works
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · see citing works
A systematic review of physicians' survival predictions in terminally ill cancer patients
Glare, P. A., Virik, K., Jones, M., Hudson, M., Eychmuller, S., Simes, J., & Christakis, N. (2003) · BMJ · doi:10.1136/bmj.327.7408.195 · Cited by 1,025 · see citing works
Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995 · see citing works
Change in End-of-Life Care for Medicare Beneficiaries
Teno, J. M., Gozalo, P. L., Bynum, J., Leland, N. E., Miller, S. C., Morden, N. E., Scupp, T., Goodman, D. C., & Mor, V. (2013) · JAMA · doi:10.1001/jama.2012.207624 · Cited by 974 · see citing works
AI can be sexist and racist — it’s time to make it fair
Zou, J., & Schiebinger, L. (2018) · Nature · doi:10.1038/d41586-018-05707-8 · Cited by 765 · see citing works
Improving palliative care with deep learning
Avati, A., Jung, K., Harman, S. M., Downing, N. L., Ng, A. Y., & Shah, N. H. (2017) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2017.8217669 · Cited by 445 · see citing works
van den Goorbergh, R., van Smeden, M., Timmerman, D., & CALSTER, B. V. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac093 · Cited by 375 · see citing works
United States Life Tables, 2020
Arias, E., & Xu, J. (2022) · doi:10.15620/cdc:118055 · Cited by 245 · see citing works
United States Life Tables, 2021.
Arias, E., Xu, J., & Kd, K. (2010) · PubMed · Cited by 224 · see citing works
Schonberg, M. A., Davis, R. B., McCarthy, E. P., & Marcantonio, E. R. (2009) · Journal of General Internal Medicine · doi:10.1007/s11606-009-1073-y · Cited by 214 · see citing works
Policy Changes Key To Promoting Sustainability And Growth Of The Specialty Palliative Care Workforce
Kamal, A. H., Wolf, S. P., Troy, J. D., Leff, V., Dahlin, C. M., Rotella, J. D., Handzo, G. F., Rodgers, P. E., & Myers, E. R. (2019) · Health Affairs · doi:10.1377/hlthaff.2019.00018 · Cited by 198 · see citing works
Analyzing Competing Risk Data Using theRtimeregPackage
Scheike, T., & Zhang, M. (2011) · Journal of Statistical Software · doi:10.18637/jss.v038.i02 · Cited by 170 · see citing works
External validation of AI models in health should be replaced with recurring local validation
Youssef, A., Pencina, M., Thakur, A., Zhu, T., Clifton, D., & Shah, N. H. (2023) · Nature Medicine · doi:10.1038/s41591-023-02540-z · Cited by 170 · see citing works
Predictive modeling of U.S. health care spending in late life
Liran Einav, B., Finkelstein, A. N., Mullainathan, S., & Obermeyer, Z. (2018) · Science · doi:10.1126/science.aar5045 · Cited by 164 · see citing works
The care planning umbrella: The evolution of advance care planning
Hickman, S. E., Lum, H. D., Walling, A. M., Savoy, A., & Sudore, R. L. (2023) · Journal of the American Geriatrics Society · doi:10.1111/jgs.18287 · Cited by 147 · see citing works
Frailty Screening Using the Electronic Health Record Within a Medicare Accountable Care Organization
Pajewski, N. M., Lenoir, K. M., Wells, B. J., Williamson, J. D., & Callahan, K. E. (2019) · The Journals of Gerontology Series A · doi:10.1093/gerona/glz017 · Cited by 140 · see citing works
Self-citationPeng, Y., Zhong, G., Zhou, X., Guan, L., & Zhou, L. (2022) · BMC Geriatrics · doi:10.1186/s12877-022-03404-w · Cited by 136 · see citing works
Riley, R. D., Collins, P. G. S., Ensor, J., Archer, L., Booth, S., Mozumder, S. I., Rutherford, M. J., van Smeden, M., Lambert, P. C., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9275 · Cited by 116 · see citing works
Navigating and Communicating about Serious Illness and End of Life
Jackson, V. A., & Emanuel, L. L. (2023) · New England Journal of Medicine · doi:10.1056/nejmcp2304436 · Cited by 106 · see citing works
Phelan, M., Bhavsar, N. A., & Goldstein, B. A. (2017) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.5334/egems.243 · Cited by 95 · see citing works
Perspectives on validation of clinical predictive algorithms
de Hond, A., Shah, V. B., Kant, I. M. J., CALSTER, B. V., Steyerberg, E. W., & Hernandez‐Boussard, T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00832-9 · Cited by 95 · see citing works
Courtright, K. R., Chivers, C., Becker, M., Regli, S. H., Pepper, L. C., Draugelis, M. E., & O’Connor, N. R. (2019) · Journal of General Internal Medicine · doi:10.1007/s11606-019-05169-2 · Cited by 81 · see citing works
Evaluation of sepsis prediction models before onset of treatment
Kamran, F., Tjandra, D., Heiler, A., Virzi, J., Singh, K., King, J. E., Valley, T. S., & Wiens, J. (2024) · NEJM AI · doi:10.1056/aioa2300032 · Cited by 74 · see citing works
Sahni, N., Simon, G., & Arora, R. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-018-4316-y · Cited by 65 · see citing works
Guidelines and standard frameworks for artificial intelligence in medicine: a systematic review
Shiferaw, K. B., Roloff, M., Balaur, I., Welter, D., Waltemath, D., & Zeleke, A. A. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae155 · Cited by 57 · see citing works
Algorithmic fairness of machine learning models for Alzheimer disease progression
Yuan, C., Linn, K. A., & Hubbard, R. A. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.42203 · Cited by 49 · see citing works
Celi, L. A., Citi, L., Ghassemi, M., & Pollard, T. (2019) · PLoS ONE · doi:10.1371/journal.pone.0210232 · Cited by 45 · see citing works
End-of-Life Care, Not End-of-Life Spending
Jha, A. K. (2018) · JAMA · doi:10.1001/jama.2018.11177 · Cited by 45 · see citing works
Life expectancy estimates based on comorbidities and frailty to inform preventive care
Schoenborn, N. L., Blackford, A. L., Joshu, C. E., Boyd, C. M., & Varadhan, R. (2021) · Journal of the American Geriatrics Society · doi:10.1111/jgs.17468 · Cited by 42 · see citing works
Piccininni, M., Wechsung, M., van Calster, B., Rohmann, J., Konigorski, S., & van Smeden, M. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104666 · Cited by 42 · see citing works
Considerations in the reliability and fairness audits of predictive models for advance care planning
Lu, J., Sattler, A. L., Wang, S. X. Y., Khaki, A. R., Callahan, A., Fleming, S. L., Fong, R., Ehlert, B., Li, R., Shieh, L. Y., et al. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.943768 · Cited by 41 · see citing works
Oblique random survival forests
Jaeger, B. C., Long, D. L., Long, D. M., Sims, M., Szychowski, J. M., Min, Y., McClure, L. A., Howard, G., & Simon, N. (2019) · The Annals of Applied Statistics · doi:10.1214/19-aoas1261 · Cited by 39 · see citing works
Self-citationOrkaby, A. R., Callahan, K. E., Driver, J. A., Hudson, K., Clegg, A. P., & Pajewski, N. M. (2024) · Age and Ageing · doi:10.1093/ageing/afae025 · Cited by 33 · see citing works
Self-citationAccelerated and Interpretable Oblique Random Survival Forests
Jaeger, B. C., Welden, S., Lenoir, K. M., Speiser, J. L., Segar, M. W., Pandey, A., & Pajewski, N. M. (2023) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2023.2231048 · Cited by 28 · see citing works
Self-citationEvaluation of AI Solutions in Health Care Organizations — The OPTICA Tool
Dagan, N., Devons-Sberro, S., Paz, Z., Zoller, L., Sommer, A., Shaham, G., Shahar, N., Ohana, R., Weinstein, O., Netzer, D., et al. (2024) · NEJM AI · doi:10.1056/aics2300269 · Cited by 27 · see citing works
Guo, P. R. F. A. (2021) · The American Journal of Managed Care · doi:10.37765/ajmc.2021.88578 · Cited by 21 · see citing works
Ho, L. T. F., Pugh, C., Seth, S., Arakelyan, S., Lone, N., Lyall, M. J., Anand, A., Fleuriot, J., Galdi, P., & Guthrie, B. (2024) · The Lancet Healthy Longevity · doi:10.1016/s2666-7568(23)00264-7 · Cited by 20 · see citing works
Regional Variation in Billed Advance Care Planning Visits
Skolarus, L. E., Lin, C. C., Kerber, K. A., & Burke, J. F. (2020) · Journal of the American Geriatrics Society · doi:10.1111/jgs.16730 · Cited by 14 · see citing works
Teeple, S., Chivers, C., Linn, K. A., Halpern, S. D., Eneanya, N. D., Draugelis, M. E., & Courtright, K. R. (2023) · BMJ Quality & Safety · doi:10.1136/bmjqs-2022-015173 · Cited by 7 · see citing works
Roberts, R. L., Cherry, K. D., Mohan, D. P., Statler, T. M., Kirkendall, E. S., Moses, A., McCraw, J., Brown, A. E., Fofanova, T. Y., & Gabbard, J. (2024) · JMIR Aging · doi:10.2196/54128 · Cited by 7 · see citing works
Self-citationGabbard, J., Brenes, G. A., Callahan, K. E., Dharod, A., Bundy, R. A., Foley, K. L., Moses, A., Williamson, J. D., & Pajewski, N. M. (2024) · Journal of the American Geriatrics Society · doi:10.1111/jgs.19100 · Cited by 6 · see citing works
Self-citationRedefining Health Care Utilization as a Quality Measure for Goal Concordance at the End of Life
Nayfeh, A., Kamra, M., & Fowler, R. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.3835 · Cited by 3 · see citing works
Mortality Risk Predictive Index to Prompt Earlier Palliative Medicine Involvement
Grandhige, A. (2024) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2024.02.283 · Cited by 2 · see citing works
Flagged first, then by citations
2 of the 52 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [45]Gotanda H, Walling AM, Zhang JJ, et al Timing and setting of billed advance care planning among Medicare decedents in 2017-2019. J Am Geriatr Soc. 2023; 71: 3237- 3243. 10.1111/jgs.1847 37335260 PMC10592584 · DOI
Cited by
Sorted by citations
A Systematic Review and Meta-analysis of Externally Validated Epic Clinical Decision Support Tools
Patel, H., Crusco, S., Hansen, D., Morales, J., Patale, T., Chiuzan, C., Sands, M., Naidich, J., Myers, K., Davidson, K. W., et al. (2026) · Journal of General Internal Medicine · doi:10.1007/s11606-026-10381-y · Cited by 4 · see citing works
Accuracy and Equity of the End-of-Life Care Index in Predicting 1-Year Mortality
Kohn, R. A., Courtright, K. R., Grau‐Sepulveda, M. V., Olsen, M. K., Madden, V. L., Sewell, B. D., Sheu, D., Ahmad, Y. S., Auriemma, C. L., Nimetz, A., et al. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.33038 · Cited by 1 · see citing works
Kohn, R. A., Courtright, K. R., Grau‐Sepulveda, M. V., Olsen, M. K., Madden, V. L., Sewell, B. D., Sheu, D., Ahmad, Y. S., Auriemma, C. L., Nimetz, A., et al. (2026) · medRxiv · doi:10.64898/2026.01.02.26343350
- Authors
- Frechman, E., Jaeger, B. C., Kowalkowski, M., Williamson, J. D., Lenoir, K. M., Palakshappa, J. A., Wells, B. J., Callahan, K. E., Pajewski, N. M., & Gabbard, J. L.
- Affiliations
- Wake Forest University Health Sciences (US)
- Corresponding author
- Frechman, Erica
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2025-04-29
- Volume
- 32
- Issue
- 7
- Pages
- 1110-1119
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC12199354
- OpenAlex
- W4409962252
- Cited by
- 3
- References
- 52
- Citations by year
- 2026: 3
- FWCI
- 2.73
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institute on Aging
- National Institutes of Health (U54AG063546, UL1TR004929, K23AG070234, K23AG073529)
- Imbedded Pragmatic Alzheimer's Disease
- AD-Related Dementias Clinical Trials Collaboratory
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Palliative Care and End-of-Life Issues
- Health Sciences › Medicine › Geriatrics and Gerontology › Frailty in Older Adults
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- mortality prediction, one-year mortality, older adults, community-dwelling adults, algorithmic fairness, model calibration, frailty, hospitalization history, end-of-life care, palliative care, health disparities
- MeSH terms
- Machine Learning, Aged, Aged, 80 and over, Algorithms, Female, Humans, Male, Mortality, ROC Curve, Terminal Care, Southeastern United States, Risk Assessment, Area Under Curve, Independent Living
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
