Improving hospital readmission prediction using individualized utility analysis
Open accessPublished version of a preprint
Abstract
Machine learning (ML) models for allocating readmission-mitigating interventions are typically selected according to their discriminative ability, which may not necessarily translate into utility in allocation of resources. Our objective was to determine whether ML models for allocating readmission-mitigating interventions have different usefulness based on their overall utility and discriminative ability. We conducted a retrospective utility analysis of ML models using claims data acquired from the Optum Clinformatics Data Mart, including 513,495 commercially-insured inpatients (mean [SD] age 69 [19] years; 294,895 [57%] Female) over the period January 2016 through January 2017 from all 50 states with mean 90 day cost of $11,552. Utility analysis estimates the cost, in dollars, of allocating interventions for lowering readmission risk based on the reduction in the 90-day cost. Allocating readmission-mitigating interventions based on a GBDT model trained to predict readmissions achieved an estimated utility gain of $104 per patient, and an AUC of 0.76 (95% CI 0.76, 0.77); allocating interventions based on a model trained to predict cost as a proxy achieved a higher utility of $175.94 per patient, and an AUC of 0.62 (95% CI 0.61, 0.62). A hybrid model combining both intervention strategies is comparable with the best models on either metric. Estimated utility varies by intervention cost and efficacy, with each model performing the best under different intervention settings. We demonstrate that machine learning models may be ranked differently based on overall utility and discriminative ability. Machine learning models for allocation of limited health resources should consider directly optimizing for utility.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The utility of an AI model-guided workflow is highly dependent on the benefits and costs of possible outcomes, and thus the inputs to the APLUS simulation tool are specific to the use case.
“The utility of the workflow is highly dependent on the benefits and costs of possible outcomes 10-12 , and thus the inputs to APLUS are specific to the use case.”
From Standing on FURM Ground: A Framework for Evaluating Fair, Useful, and Reliable AI Models in Health Care Systems (2024) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data
Quan, H., Sundararajan, V., Halfon, P., Fong, A., Burnand, B., Luthi, J.-C., Saunders, L. D., Beck, C. A., Feasby, T. E., & Ghali, W. A. (2005) · Medical Care · doi:10.1097/01.mlr.0000182534.19832.83 · Cited by 10,669 · see citing works
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · see citing works
17. A Value for n-Person Games
Shapley, L. S. (1953) · Contributions to the Theory of Games (AM-28), Volume II · doi:10.1515/9781400881970-018 · Cited by 6,950 · see citing works
Friedman, J., Hastie, T., & Tibshirani, R. (2000) · The Annals of Statistics · doi:10.1214/aos/1016218223 · Cited by 6,884 · see citing works
Rehospitalizations among Patients in the Medicare Fee-for-Service Program
Jencks, S. F., Williams, M. V., & Coleman, E. A. (2009) · New England Journal of Medicine · doi:10.1056/nejmsa0803563 · Cited by 5,164 · see citing works
Alba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J., McGinn, T., & Guyatt, G. (2017) · JAMA · doi:10.1001/jama.2017.12126 · Cited by 1,631 · see citing works
Unintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · see citing works
van Walraven, C., Dhalla, I. A., Bell, C. M., Etchells, E. E., Stiell, I. G., Zarnke, K. B., Austin, P. C., & Forster, A. J. (2010) · Canadian Medical Association Journal · doi:10.1503/cmaj.091117 · Cited by 966 · see citing works
Matching on the Estimated Propensity Score
Abadie, A., & Imbens, G. W. (2016) · Econometrica · doi:10.3982/ecta11293 · Cited by 902 · see citing works
Hospital Readmissions Reduction Program
McIlvennan, C. K., Eapen, Z. J., & Allen, L. A. (2015) · Circulation · doi:10.1161/circulationaha.114.010270 · Cited by 690 · see citing works
Reducing Hospital Readmission Rates: Current Strategies and Future Directions
Kripalani, S. B., Theobald, C. N., Anctil, B., & Vasilevskis, E. E. (2013) · Annual Review of Medicine · doi:10.1146/annurev-med-022613-090415 · Cited by 576 · see citing works
Making Machine Learning Models Clinically Useful
Shah, N. H., Milstein, A., & Bagley, P. (2019) · JAMA · doi:10.1001/jama.2019.10306 · Cited by 294 · see citing works
Self-citationStatistical Primer: developing and validating a risk prediction model†
Grant, S. W., Collins, P. G. S., & Nashef, S. A. M. (2018) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezy180 · Cited by 254 · see citing works
Applications of Machine Learning in Real-Life Digital Health Interventions: Review of the Literature
Triantafyllidis, A. K., & Tsanas, A. (2019) · Journal of Medical Internet Research · doi:10.2196/12286 · Cited by 243 · see citing works
Donzé, J. D., Williams, M. V., Robinson, E. J., Zimlichman, E., Aujesky, D., Vasilevskis, E. E., Kripalani, S. B., Metlay, J. P., Wallington, T., Fletcher, G. S., et al. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.8462 · Cited by 241 · see citing works
Artificial Intelligence in Health Care: Will the Value Match the Hype?
Emanuel, E. J., & Wachter, R. M. (2019) · JAMA · doi:10.1001/jama.2019.4914 · Cited by 237 · see citing works
Predictive models for hospital readmission risk: A systematic review of methods
Artetxe, A., Beristain, A., & Graña, M. (2018) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2018.06.006 · Cited by 205 · see citing works
Predictive modeling of U.S. health care spending in late life
Liran Einav, B., Finkelstein, A. N., Mullainathan, S., & Obermeyer, Z. (2018) · Science · doi:10.1126/science.aar5045 · Cited by 164 · see citing works
Hospital-Readmission Risk — Isolating Hospital Effects from Patient Effects
Krumholz, H. M., Wang, K., Lin, Z., Dharmarajan, K., Horwitz, L. I., Ross, J. S., Drye, E. E., Bernheim, S. M., & Normand, S. T. (2017) · New England Journal of Medicine · doi:10.1056/nejmsa1702321 · Cited by 123 · see citing works
Bayati, M., Braverman, M., Gillam, M., Mack, K. M., Ruiz, G., Smith, M. S., & Horvitz, E. (2014) · PLoS ONE · doi:10.1371/journal.pone.0109264 · Cited by 122 · see citing works
Crown, W. H., Büyükkaramikli, N., Thokala, P., Morton, A., Sır, M. Y., Marshall, D. A., Tosh, J. C., Padula, W. V., IJzerman, M. J., Wong, P. K., et al. (2017) · Value in Health · doi:10.1016/j.jval.2017.01.013 · Cited by 109 · see citing works
Economic Evaluation of Quality Improvement Interventions Designed to Prevent Hospital Readmission
Nuckols, T. K., Keeler, E. B., Morton, S. C., Anderson, L., Doyle, B. J., Pevnick, J. M., Booth, M. S., Shanman, R., Arifkhanova, A., & Shekelle, P. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2017.1136 · Cited by 107 · see citing works
A framework for making predictive models useful in practice
Jung, K., Kashyap, S., Avati, A., Harman, S. M., Shaw, H., Li, R., Smith, M. A., Shum, K., Javitz, J., Vetteth, Y., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa318 · Cited by 82 · see citing works
Self-citationDevelopment and Validation of an Algorithm to Identify Planned Readmissions From Claims Data
Horwitz, L. I., Grady, J. N., Cohen, D. B., et al. (2015) · Journal of Hospital Medicine · doi:10.1002/jhm.2416 · Cited by 64 · see citing works
The number needed to benefit: estimating the value of predictive analytics in healthcare
Liu, V. X., Bates, D. W., Wiens, J., & Shah, N. H. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz088 · Cited by 59 · see citing works
Self-citationGPU-acceleration for Large-scale Tree BoostingPreprint
Zhang, H., Si, S., & Hsieh, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.08359 · Cited by 54 · see citing works
Crown, W. H., Büyükkaramikli, N., Sır, M. Y., Thokala, P., Morton, A., Marshall, D. A., Tosh, J. C., IJzerman, M. J., Padula, W. V., & Pasupathy, K. S. (2018) · Value in Health · doi:10.1016/j.jval.2018.05.003 · Cited by 53 · see citing works
Individualized Cost-Effectiveness Analysis
Ioannidis, J. P. A., & Garber, A. M. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1001058 · Cited by 50 · see citing works
The Proliferation of Reports on Clinical Scoring Systems
Challener, D. W., Prokop, L. J., & AbuSaleh, O. (2019) · JAMA · doi:10.1001/jama.2019.5284 · Cited by 45 · see citing works
Nair, R. M., Lak, H. M., Hasan, S., Gunasekaran, D., Babar, A., & Gopalakrishna, K. V. (2020) · Cureus · doi:10.7759/cureus.7420 · Cited by 41 · see citing works
Baky, V., Moran, D., Warwick, T., George, A., Williams, T., McWilliams, E., & Marine, J. E. (2018) · International Journal of Cardiology · doi:10.1016/j.ijcard.2017.10.036 · Cited by 28 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- G. Ke, Q. Meng, T. Finley, et al., LightGBM: a highly efficient gradient boosting decision tree, NIPS17 Proc 31st Int Conf Neural Inf Process Syst, Published online December 2017:3149-3157.
Cited by
Sorted by citations
Callahan, A., McElfresh, D., Banda, J. M., Bunney, G., Char, D., Chen, J., Corbin, C. K., Dash, D., Downing, N. L., Jain, S. S., et al. (2024) · NEJM Catalyst · doi:10.1056/cat.24.0131 · Cited by 61 · see citing works
Self-citation
- Authors
- Ko, M., Chen, E., Agrawal, A., Rajpurkar, P., Avati, A., Ng, A., Basu, S., & Shah, N. H.
- Affiliations
- Stanford University (US)
- Harvard University (US)
- Corresponding author
- Shah, Nigam H.
- ORCID
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2021-07
- Volume
- 119
- Pages
- 103826
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W3165347509
- Cited by
- 15
- References
- 31
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 1 · 2023: 5 · 2022: 5 · 2021: 1
- FWCI
- 1.06
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion, cc-by-nc-nd)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Keywords
- hospital readmission prediction, resource allocation, discriminatory power, gradient boosting decision tree, cost prediction
- MeSH terms
- Machine Learning, Aged, Female, Humans, Patient Readmission, Retrospective Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
