A framework for making predictive models useful in practice
Open accessPublished version of a preprint
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Objective: To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality. Materials and Methods: We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models’ predictions. Factors included nonclinical reasons that make ACP inappropriate: limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP. Results: Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model’s predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care. Discussion: The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit. Conclusion: An analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Early Palliative Care for Patients with Metastatic Non–Small-Cell Lung Cancer
Temel, J. S., Greer, J. A., Muzikansky, A., Gallagher, E. R., Admane, S., Jackson, V. A., Dahlin, C. M., Blinderman, C. D., Jacobsen, J., Pirl, W. F., et al. (2010) · New England Journal of Medicine · doi:10.1056/nejmoa1000678 · Cited by 7,482
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“8 12–14 In many cases the overall benefit of using predictive models for improving healthcare will be critically determined by the considerations of implementation costs, actionability, safety, and utility.”
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579 · cited 1×
“The factors analyzed were: (1) ineligibility of candidates due to non-clinical reasons, (2) limited capacity to offer ACP, (3) failure to complete ACP due to early discharge, and (4) availability of an outpatient pathway to follow up on missed cases.”
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648
Self-citationBig Data and Machine Learning in Health Care
Beam, A. L., & Kohane, I. S. (2018) · JAMA · doi:10.1001/jama.2017.18391 · Cited by 2,506
Implementing Machine Learning in Health Care — Addressing Ethical Challenges
Char, D. S., Shah, N. H., & Magnus, D. (2018) · New England Journal of Medicine · doi:10.1056/nejmp1714229 · Cited by 1,857 · cited 1×
“8 12–14 In many cases the overall benefit of using predictive models for improving healthcare will be critically determined by the considerations of implementation costs, actionability, safety, and utility.”
Self-citationBig Data In Health Care: Using Analytics To Identify And Manage High-Risk And High-Cost Patients
Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A., & Escobar, G. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0041 · Cited by 1,235 · cited 1×
“Unlike prior work applying the cost– benefit framework to the clinical use of predictive models, 31,32 our focus is not on selecting a suitable decision threshold for triggering a fixed intervention.”
Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936
Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614
Research works printed but not linked: 31
- Obermeyer, 2020, Adoption of artificial intelligence and machine learning is increasing, but irrational exuberance remains, NEJM Catalyst, 1, 10.1056/CAT.19.1090 · DOI
- Emanuel, 2019, Artificial intelligence in health care: Will the value match the hype?, JAMA, 321, 2281, 10.1001/jama.2019.4914 · DOI · Paper page
- Morse, 2020, Estimate the hidden deployment cost of predictive models to improve patient care, Nat Med, 26, 18, 10.1038/s41591-019-0651-8 · DOI
- Sendak, 2019, Machine learning in health care: A critical appraisal of challenges and opportunities, eGEMs, 7, 1, 10.5334/egems.287 · DOI · Paper page
- Einav, 2018, Predictive modeling of US health care spending in late life, Science, 360, 1462, 10.1126/science.aar5045 · DOI · Paper page
- Smith, 2014, Evidence on the cost and cost-effectiveness of palliative care: a literature review, Palliat Med, 28, 130, 10.1177/0269216313493466 · DOI · Paper page
- Sullivan, 2019, Association of early palliative care use with survival and place of death among patients with advanced lung cancer receiving care in the Veterans Health Administration, JAMA Oncol, 5, 1702, 10.1001/jamaoncol.2019.3105 · DOI · Paper page
- Verret, 2013, The value of palliative care, Healthc Finance Manage, 67, 50
- Osagiede, 2018, Palliative care use among patients with solid cancer tumors: a national cancer data base study, J Palliat Care, 33, 149, 10.1177/0825859718777320 · DOI
- Evans, 2020, Palliative care consultation is underutilized in critically ill general surgery patients, Am J Hosp Palliat Care, 37, 149, 10.1177/1049909119864025 · doi:10.1177/1049909119864025
- Rubens, 2019, Palliative care consultation trends among hospitalized patients with advanced cancer in the United States, 2005 to 2014, Am J Hosp Palliat Care, 36, 294, 10.1177/1049909118809975 · DOI
- Wiskar, 2018, Palliative care in heart failure, Trends Cardiovasc Med, 28, 445, 10.1016/j.tcm.2018.02.008 · DOI
- Parikh, 2019, Machine learning approaches to predict 6-month mortality among patients with cancer, JAMA Netw Open, 2, e1915997, 10.1001/jamanetworkopen.2019.15997 · DOI · Paper page
- Courtright, 2019, Electronic health record mortality prediction model for targeted palliative care among hospitalized medical patients: a pilot quasi-experimental study, J Gen Intern MED, 34, 1841, 10.1007/s11606-019-05169-2 · DOI
- Liu, 2019, The number needed to benefit: estimating the value of predictive analytics in healthcare, J Am Med Inform Assoc, 26, 1655, 10.1093/jamia/ocz088 · DOI · Paper page
- McIntosh, 1999, Recent advances in the methods of cost-benefit analysis in healthcare. Matching the art to the science, Pharmacoeconomics, 15, 357, 10.2165/00019053-199915040-00003 · DOI
- Weiner, 2018, Interactive cost-benefit analysis: providing real-world financial context to predictive analytics, AMIA Annu Symp Proc, 2018, 1076
- Pantilat, 2010, Hospital-based palliative medicine consultation: a randomized controlled trial, Arch Intern Med, 170, 2038, 10.1001/archinternmed.2010.460 · DOI
- Ke, 2017, Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17), 3149
- Lowe, 2009, STRIDE—An integrated standards-based translational research informatics platform, AMIA Annu Symp Proc, 2009, 391
- Scrucca, 2011, Model-based SIR for dimension reduction, Comput Stat Data Anal, 55, 3010, 10.1016/j.csda.2011.05.006 · DOI
- Shilo, 2020, Axes of a revolution: challenges and promises of big data in healthcare, Nat Med, 26, 29, 10.1038/s41591-019-0727-5 · DOI · Paper page
- Challener, 2019, The proliferation of reports on clinical scoring systems: issues about uptake and clinical utility, JAMA, 321, 2405, 10.1001/jama.2019.5284 · DOI · Paper page
- Escobar, 2012, Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record, J Hosp Med, 7, 388, 10.1002/jhm.1929 · DOI · Paper page
- Escobar, 2016, Early detection, prevention, and mitigation of critical illness outside intensive care settings, J Hosp Med, 11 (Suppl 1, S5, 10.1002/jhm.2653 · DOI
- Escobar, 2016, Piloting electronic medical record-based early detection of inpatient deterioration in community hospitals, J Hosp Med, 11 (Suppl 1, S18, 10.1002/jhm.2652 · DOI · Paper page
- Dummett, 2016, Incorporating an early detection system into routine clinical practice in two community hospitals, J Hosp Med, 11 (Suppl 1, S25, 10.1002/jhm.2661 · DOI · Paper page
- Crown, 2017, Constrained optimization methods in health services research-an introduction: Report 1 of the ISPOR optimization methods emerging good practices task force, Value Health, 20, 310, 10.1016/j.jval.2017.01.013 · DOI
- Crown, 2018, Application of constrained optimization methods in health services research: report 2 of the ISPOR optimization methods emerging good practices task force, Value Health, 21, 1019, 10.1016/j.jval.2018.05.003 · DOI
- Li, 2020, Developing a delivery science for, NPJ Digit Med, 3, 10.1038/s41746-020-00318-y · DOI · Paper page
- Beaulieu-Jones, 2019, Trends and focus of machine learning applications for health research, JAMA Netw Open, 2, e1914051, 10.1001/jamanetworkopen.2019.14051 · DOI
Cited by
Algorithmic fairness in artificial intelligence for medicine and healthcare
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01056-8 · Cited by 750
Marwaha, J. S., Landman, A. B., Brat, G. A., Dunn, T., & Gordon, W. J. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00557-1 · Cited by 241
“A digital health tool that is useful in practice is one that is able to trigger a speci fi c work fl ow: the insight that the tool provides must be translated into an actual human intervention in order to realize the tool ’ s value 25 .”
MethodAn empirical characterization of fair machine learning for clinical risk prediction
Pfohl, S. R., Foryciarz, A., & Shah, N. H. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103621 · Cited by 164
Self-citationMarwaha, J. S., & Kvedar, J. C. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00572-2 · Cited by 46
Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings
Davis, S. E., Walsh, C. G., & Matheny, M. E. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.958284 · Cited by 33
Metadata
- Authors
- Jung, K., Kashyap, S., Avati, A., Harman, S., Shaw, H., Li, R., Smith, M., Shum, K., Javitz, J., Vetteth, Y., Seto, T., Bagley, S. C., & Shah, N. H.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-12-22
- Volume
- 28
- Issue
- 6
- Pages
- 1149-1158
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W3117058671
- Cited by
- 78
- References
- 15
- FWCI
- 6.20
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Funders
- Stanford Medicine Department of Medicine, an endowment from Debra and Mark Leslie, and innovations funds from Stanford Healthcare
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A framework for making predictive models useful in practice. https://app.cochat.ai/papers/10.1093/jamia/ocaa318
- The paper
- Jung, K., Kashyap, S., Avati, A., Harman, S., Shaw, H., Li, R., Smith, M., Shum, K., Javitz, J., Vetteth, Y., Seto, T., Bagley, S. C., & Shah, N. H. (2021). A framework for making predictive models useful in practice. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocaa318
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.