CoChat

A framework for making predictive models useful in practice

Open access
Jung, Kenneth, Kashyap, Sehj, Avati, Anand, Harman, Stephanie, Shaw, Heather, Li, Ron, Smith, Margaret, Shum, Kenny, Javitz, Jacob, Vetteth, Yohan, Seto, Tina, Bagley, Steven C, Shah, Nigam H · 2021 · Journal of the American Medical Informatics Association

Published version of a preprint

Cited by 78

The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.

Abstract

Objective: To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality. Materials and Methods: We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models’ predictions. Factors included nonclinical reasons that make ACP inappropriate: limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP. Results: Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model’s predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care. Discussion: The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit. Conclusion: An analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocaa318

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 15 references

Research works printed but not linked: 31

  1. Obermeyer, 2020, Adoption of artificial intelligence and machine learning is increasing, but irrational exuberance remains, NEJM Catalyst, 1, 10.1056/CAT.19.1090 · DOI
  2. Emanuel, 2019, Artificial intelligence in health care: Will the value match the hype?, JAMA, 321, 2281, 10.1001/jama.2019.4914 · DOI · Paper page
  3. Morse, 2020, Estimate the hidden deployment cost of predictive models to improve patient care, Nat Med, 26, 18, 10.1038/s41591-019-0651-8 · DOI
  4. Sendak, 2019, Machine learning in health care: A critical appraisal of challenges and opportunities, eGEMs, 7, 1, 10.5334/egems.287 · DOI · Paper page
  5. Einav, 2018, Predictive modeling of US health care spending in late life, Science, 360, 1462, 10.1126/science.aar5045 · DOI · Paper page
  6. Smith, 2014, Evidence on the cost and cost-effectiveness of palliative care: a literature review, Palliat Med, 28, 130, 10.1177/0269216313493466 · DOI · Paper page
  7. Sullivan, 2019, Association of early palliative care use with survival and place of death among patients with advanced lung cancer receiving care in the Veterans Health Administration, JAMA Oncol, 5, 1702, 10.1001/jamaoncol.2019.3105 · DOI · Paper page
  8. Verret, 2013, The value of palliative care, Healthc Finance Manage, 67, 50
  9. Osagiede, 2018, Palliative care use among patients with solid cancer tumors: a national cancer data base study, J Palliat Care, 33, 149, 10.1177/0825859718777320 · DOI
  10. Evans, 2020, Palliative care consultation is underutilized in critically ill general surgery patients, Am J Hosp Palliat Care, 37, 149, 10.1177/1049909119864025 · doi:10.1177/1049909119864025
  11. Rubens, 2019, Palliative care consultation trends among hospitalized patients with advanced cancer in the United States, 2005 to 2014, Am J Hosp Palliat Care, 36, 294, 10.1177/1049909118809975 · DOI
  12. Wiskar, 2018, Palliative care in heart failure, Trends Cardiovasc Med, 28, 445, 10.1016/j.tcm.2018.02.008 · DOI
  13. Parikh, 2019, Machine learning approaches to predict 6-month mortality among patients with cancer, JAMA Netw Open, 2, e1915997, 10.1001/jamanetworkopen.2019.15997 · DOI · Paper page
  14. Courtright, 2019, Electronic health record mortality prediction model for targeted palliative care among hospitalized medical patients: a pilot quasi-experimental study, J Gen Intern MED, 34, 1841, 10.1007/s11606-019-05169-2 · DOI
  15. Liu, 2019, The number needed to benefit: estimating the value of predictive analytics in healthcare, J Am Med Inform Assoc, 26, 1655, 10.1093/jamia/ocz088 · DOI · Paper page
  16. McIntosh, 1999, Recent advances in the methods of cost-benefit analysis in healthcare. Matching the art to the science, Pharmacoeconomics, 15, 357, 10.2165/00019053-199915040-00003 · DOI
  17. Weiner, 2018, Interactive cost-benefit analysis: providing real-world financial context to predictive analytics, AMIA Annu Symp Proc, 2018, 1076
  18. Pantilat, 2010, Hospital-based palliative medicine consultation: a randomized controlled trial, Arch Intern Med, 170, 2038, 10.1001/archinternmed.2010.460 · DOI
  19. Ke, 2017, Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17), 3149
  20. Lowe, 2009, STRIDE—An integrated standards-based translational research informatics platform, AMIA Annu Symp Proc, 2009, 391
  21. Scrucca, 2011, Model-based SIR for dimension reduction, Comput Stat Data Anal, 55, 3010, 10.1016/j.csda.2011.05.006 · DOI
  22. Shilo, 2020, Axes of a revolution: challenges and promises of big data in healthcare, Nat Med, 26, 29, 10.1038/s41591-019-0727-5 · DOI · Paper page
  23. Challener, 2019, The proliferation of reports on clinical scoring systems: issues about uptake and clinical utility, JAMA, 321, 2405, 10.1001/jama.2019.5284 · DOI · Paper page
  24. Escobar, 2012, Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record, J Hosp Med, 7, 388, 10.1002/jhm.1929 · DOI · Paper page
  25. Escobar, 2016, Early detection, prevention, and mitigation of critical illness outside intensive care settings, J Hosp Med, 11 (Suppl 1, S5, 10.1002/jhm.2653 · DOI
  26. Escobar, 2016, Piloting electronic medical record-based early detection of inpatient deterioration in community hospitals, J Hosp Med, 11 (Suppl 1, S18, 10.1002/jhm.2652 · DOI · Paper page
  27. Dummett, 2016, Incorporating an early detection system into routine clinical practice in two community hospitals, J Hosp Med, 11 (Suppl 1, S25, 10.1002/jhm.2661 · DOI · Paper page
  28. Crown, 2017, Constrained optimization methods in health services research-an introduction: Report 1 of the ISPOR optimization methods emerging good practices task force, Value Health, 20, 310, 10.1016/j.jval.2017.01.013 · DOI
  29. Crown, 2018, Application of constrained optimization methods in health services research: report 2 of the ISPOR optimization methods emerging good practices task force, Value Health, 21, 1019, 10.1016/j.jval.2018.05.003 · DOI
  30. Li, 2020, Developing a delivery science for, NPJ Digit Med, 3, 10.1038/s41746-020-00318-y · DOI · Paper page
  31. Beaulieu-Jones, 2019, Trends and focus of machine learning applications for health research, JAMA Netw Open, 2, e1914051, 10.1001/jamanetworkopen.2019.14051 · DOI

Cited by

Metadata

Authors
Jung, K., Kashyap, S., Avati, A., Harman, S., Shaw, H., Li, R., Smith, M., Shum, K., Javitz, J., Vetteth, Y., Seto, T., Bagley, S. C., & Shah, N. H.
Venue
Journal of the American Medical Informatics Association
Published
2020-12-22
Volume
28
Issue
6
Pages
1149-1158
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W3117058671
Cited by
78
References
15
FWCI
6.20
Type
journal article
Language
en
Open access
hybrid
Funders
Stanford Medicine Department of Medicine, an endowment from Debra and Mark Leslie, and innovations funds from Stanford Healthcare

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.