CoChat
Sign Up
Cited by 26Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareMachine Learning in Healthcare

APLUS: A Python library for usefulness simulations of machine learning models in healthcare

Open access
Wornow, Michael, Gyang Ross, Elsie, Callahan, Alison, Shah, Nigam H.·2023·Journal of Biomedical Informatics
DOI 10.1016/j.jbi.2023.104319PMC PMC10309067
Open this paper in CoChat — free account

Abstract

Despite the creation of thousands of machine learning (ML) models, the promise of improving patient care with ML remains largely unrealized. Adoption into clinical practice is lagging, in large part due to disconnects between how ML practitioners evaluate models and what is required for their successful integration into care delivery. Models are just one component of care delivery workflows whose constraints determine clinicians' abilities to act on models' outputs. However, methods to evaluate the usefulness of models in the context of their corresponding workflows are currently limited. To bridge this gap we developed APLUS, a reusable framework for quantitatively assessing via simulation the utility gained from integrating a model into a clinical workflow. We describe the APLUS simulation engine and workflow specification language, and apply it to evaluate a novel ML-based screening pathway for detecting peripheral artery disease at Stanford Health Care.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2023.104319

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Clinical workflows are capacity constrained-a nursing team may only be able to act on a handful of model predictions per day.

    “In reality, clinical workflows are capacity constrained—a nursing team may only be able to act on a handful of model predictions per day79,80.”

    From The shaky foundations of large language models and foundation models for electronic health records (2023) · cites this paper as [80]

  • The authors contribute three novel elements to existing testing and evaluation regimens: pre-deployment ethical review for identification of stakeholder value mismatches, financial sustainability analysis, and the use of simulation to estimate achievable benefit under a given work capacity.

    “We contribute three novel elements to existing testing and evaluation regimens: pre-deployment ethical review for identification of stakeholder value mismatches, financial sustainability analysis, and the use of simulation to estimate achievable benefit under a given work capacity 9 .”

    From Standing on FURM Ground: A Framework for Evaluating Fair, Useful, and Reliable AI Models in Health Care Systems (2024) · cites this paper as [9]

  • By freely sharing the assessment process, underlying methods, and open source tools used to quantify realizable benefits, the authors make it possible for other healthcare systems to conduct actionable evaluations of candidate AI systems.

    “By freely sharing the assessment process, underlying methods 12 and open source tools 9 used to quantify realizable benefits, we make it possible for other healthcare systems to conduct actionable evaluations of candidate AI systems.”

    From Standing on FURM Ground: A Framework for Evaluating Fair, Useful, and Reliable AI Models in Health Care Systems (2024) · cites this paper as [9]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 62 · Cited by 26

Research works printed but not linked: 53

  1. [1]Topol EJ, High-performance medicine: the convergence of human and artificial intelligence, Nat. Med 25 (2019) 44– 56, 10.1038/s41591-018-0300-7. 30617339 · DOI · Paper page
  2. [2]Lee D, Yoon SN, Application of Artificial Intelligence-Based Technologies in the Healthcare Industry: Opportunities and Challenges, Int. J. Environ. Res. Public Health 18 (2021) 271, 10.3390/ijerph18010271. 33401373 PMC7795119 · DOI · Paper page
  3. [3]Ngiam KY, Khor IW, Big data and machine learning algorithms for health-care delivery, Lancet Oncol. 20 (2019) e262– e273, 10.1016/S1470-2045(19)30149-4. 31044724 · DOI · Paper page
  4. [4]Miotto R, Wang F, Wang S, Jiang X, Dudley JT, Deep learning for healthcare: review, opportunities and challenges, Brief. Bioinform 19 (2018) 1236– 1246, 10.1093/bib/bbx044. 28481991 PMC6455466 · DOI · Paper page
  5. [5]Obermeyer Z, Weinstein JN, Adoption of Artificial Intelligence and Machine Learning Is Increasing, but Irrational Exuberance Remains, NEJM Catalyst. 1 (2020) CAT.19.1090. 10.1056/CAT.19.1090. · DOI
  6. [6]Shah N, Making Machine Learning Models Clinically Useful, J. Am. Med. Assoc 322 (2019) 1351, 10.1001/jama.2019.10306. 31393527 · DOI · Paper page
  7. [8]Challener DW, Prokop LJ, Abu-Saleh O, The Proliferation of Reports on Clinical Scoring Systems: Issues About Uptake and Clinical Utility, J. Am. Med. Assoc 321 (2019) 2405– 2406, 10.1001/jama.2019.5284. 31125046 · DOI
  8. [9]Jung K, Kashyap S, Avati A, Harman S, Shaw H, Li R, Smith M, Shum K, Javitz J, Vetteth Y, Seto T, Bagley SC, Shah NH, A framework for making predictive models useful in practice, J. Am. Med. Inform. Assoc 28 (2021) 1149– 1158, 10.1093/jamia/ocaa318. 33355350 PMC8200271 · DOI
  9. [10]Dummett BA, Adams C, Scruth E, Liu V, Guo M, Escobar GJ, Incorporating an Early Detection System Into Routine Clinical Practice in Two Community Hospitals, J. Hosp. Med 11 (2016) S25– S31, 10.1002/jhm.2661. 27805798 PMC5568844 · DOI
  10. [11]Greenes RA, Bates DW, Kawamoto K, Middleton B, Osheroff J, Shahar Y, Clinical decision support models and frameworks: Seeking to address research issues underlying implementation successes and failures, J. Biomed. Inform 78 (2018) 134– 143, 10.1016/j.jbi.2017.12.005. 29246790 · DOI
  11. [12]Kannampallil TG, Schauer GF, Cohen T, Patel VL, Considering complexity in healthcare systems, J. Biomed. Inform 44 (2011) 943– 947, 10.1016/j.jbi.2011.06.006. 21763459 · DOI
  12. [13]Seneviratne MG, Shah NH, Chu L, Bridging the implementation gap of machine learning in healthcare, BMJ Innovations. 6 (2020), 10.1136/bmjinnov-2019-000359. · DOI · Paper page
  13. [14]Morse KE, Bagley SC, Shah NH, Estimate the hidden deployment cost of predictive models to improve patient care, Nat. Med 26 (2020) 18– 19, 10.1038/s41591-019-0651-8. 31932778 · DOI
  14. [15]Li RC, Smith M, Lu J, Avati A, Wang S, Teuteberg WG, Shum K, Hong G, Seevaratnam B, Westphal J, Dougherty M, Rao P, Asch S, Lin S, Sharp C, Shieh L, Shah NH, Using AI to Empower Collaborative Team Workflows: Two Implementations for Advance Care Planning and Care Escalation, NEJM Catalyst. 3 (n.d.) CAT.21.0457. 10.1056/CAT.21.0457. · DOI
  15. [16]Baker SG, Decision Curves and Relative Utility Curves, Med. Decis. Making 39 (2019) 489– 490, 10.1177/0272989X19850762. 31104590 PMC6786943 · DOI
  16. [17]Vickers AJ, Elkin EB, Decision Curve Analysis: A Novel Method for Evaluating Prediction Models, Med. Decis. Making 26 (2006) 565– 574, 10.1177/0272989X06295361. 17099194 PMC2577036 · DOI · Paper page
  17. [18]Keane PA, Topol EJ, With an eye to AI and autonomous diagnosis, NPJ Digital Med. 1 (2018) 40, 10.1038/s41746-018-0048-y. PMC6550235 31304321 · DOI · Paper page
  18. [19]Kerr KF, Brown MD, Zhu K, Janes H, Assessing the Clinical Impact of Risk Prediction Models With Decision Curves: Guidance for Correct Interpretation and Appropriate Use, J. Clin. Oncol 34 (2016) 2534– 2540, 10.1200/JCO.2015.65.5654. 27247223 PMC4962736 · DOI · Paper page
  19. [20]Baker SG, Cook NR, Vickers A, Kramer BS, Using relative utility curves to evaluate risk prediction, J. R. Stat. Soc. A. Stat. Soc 172 (2009) 729– 748, 10.1111/j.1467-985X.2009.00592.x. PMC2804257 20069131 · DOI · Paper page
  20. [21]Connell A, Black G, Montgomery H, Martin P, Nightingale C, King D, Karthikesalingam A, Hughes C, Back T, Ayoub K, Suleyman M, Jones G, Cross J, Stanley S, Emerson M, Merrick C, Rees G, Laing C, Raine R, Implementation of a Digitally Enabled Care Pathway (Part 2): Qualitative Analysis of Experiences of Health Care Professionals, J. Med. Internet Res 21 (2019) e13143. 31368443 PMC6693304
  21. [22]Wiens J, Saria S, Sendak M, Ghassemi M, Liu VX, Doshi-Velez F, Jung K, Heller K, Kale D, Saeed M, Ossorio PN, Thadaney-Israni S, Goldenberg A, Do no harm: a roadmap for responsible machine learning for health care, Nat. Med 25 (2019) 1337– 1340, 10.1038/s41591-019-0548-6. 31427808 · DOI · Paper page
  22. [23]Sendak MP, Balu S, Schulman KA, Barriers to Achieving Economies of Scale in Analysis of EHR Data, A Cautionary Tale, Applied Clinical Informatics. 8 (2017) 826– 831, 10.4338/ACI-2017-03-CR-0046. 28837212 PMC6220705 · DOI · Paper page
  23. [24]Hamrock E, Paige K, Parks J, Scheulen J, Levin S, Discrete event simulation for healthcare organizations: a tool for decision making, J. Healthc. Manag 58 (2013), 110– 124; discussion 124–125. 23650696
  24. [25]Vázquez-Serrano JI, Peimbert-García RE, Cárdenas-Barrón LE, Discrete-Event Simulation Modeling in Healthcare: A Comprehensive Review, Int. J. Environ. Res. Public Health 18 (2021) 12262, 10.3390/ijerph182212262. 34832016 PMC8625660 · DOI
  25. [26]Zhang X, Application of discrete event simulation in health care: a systematic review, BMC Health Serv. Res 18 (2018) 687, 10.1186/s12913-018-3456-4. 30180848 PMC6123911 · DOI
  26. [27]Jacobson SH, Hall SN, Swisher JR, Discrete-Event Simulation of Health Care Systems, in: Hall RW (Ed.), Patient Flow: Reducing Delay in Healthcare Delivery, Springer US, Boston, MA, 2006: pp. 211– 252. 10.1007/978-0-387-33636-7_8. · DOI
  27. [28]Kovalchuk SV, Funkner AA, Metsker OG, Yakovlev AN, Simulation of patient flow in multiple healthcare units using process and data mining techniques for model identification, J. Biomed. Inform 82 (2018) 128– 142, 10.1016/j.jbi.2018.05.004. 29753874 · DOI
  28. [29]Ko M, Chen E, Agrawal A, Rajpurkar P, Avati A, Ng A, Basu S, Shah NH, Improving hospital readmission prediction using individualized utility analysis, J. Biomed. Inform 119 (2021), 103826, 10.1016/j.jbi.2021.103826. 34087428 · DOI · Paper page
  29. [31]Mišíc VV, Rajaram K, Gabel E, A simulation-based evaluation of machine learning models for clinical decision support: application and analysis using hospital readmission, npj Digital Med. 4 (2021) 98, 10.1038/s41746-021-00468-7. PMC8203794 34127786 · DOI
  30. [32]Itoga NK, Minami HR, Chelvakumar M, Pearson K, Mell MM, Bendavid E, Owens DK, Cost-effectiveness analysis of asymptomatic peripheral artery disease screening with the ABI test, Vasc. Med 23 (2018) 97– 106, 10.1177/1358863X17745371. 29345540 PMC5893367 · DOI
  31. [33]Diao JA, Wedlund L, Kvedar J, Beyond performance metrics: modeling outcomes and cost for clinical machine learning, npj Digital Med. 4 (2021) 1– 2, 10.1038/s41746-021-00495-4. PMC8355228 34376781 · DOI
  32. [34]Ghanzouri I, Amal S, Ho V, Safarnejad L, Cabot J, Brown-Johnson CG, Leeper N, Asch S, Shah NH, Ross EG, Performance and usability testing of an automated tool for detection of peripheral artery disease using electronic health records, Sci. Rep 12 (2022) 13364, 10.1038/s41598-022-17180-5. 35922657 PMC9349186 · DOI
  33. [35]Cassandras CG, Lafortune S, Introduction to discrete event systems, Springer, New York, NY, 2010, p. 2, ed., corr. at 2. print.
  34. [36]Ghattas J, Peleg M, Soffer P, Denekamp Y, Learning the Context of a Clinical Process, in: Rinderle-Ma S, Sadiq S, Leymann F (Eds.), Business Process Management Workshops, Springer, Berlin, Heidelberg, 2010, pp. 545– 556, 10.1007/978-3-642-12186-9_53. · DOI
  35. [37]Choi J, Jansen K, Coenen A, Modeling a Nursing Guideline with Standard Terminology and Unified Modeling Language for a Nursing Decision Support System: A Case Study, AMIA Ann. Symp. Proc 2015 (2015) 426– 433. PMC4765648 26958174
  36. [38]Ferrante S, Bonacina S, Pinciroli F, Modeling stroke rehabilitation processes using the Unified Modeling Language (UML), Comput. Biol. Med 43 (2013) 1390– 1401, 10.1016/j.compbiomed.2013.07.012. 24034730 · DOI
  37. [39]Peleg M, 13 - Guidelines and workflow models, in: Greenes RA (Ed.), Clinical Decision Support, Academic Press, Burlington, 2007, pp. 281– 306, 10.1016/B978-012369377-8/50014-3. · DOI
  38. [40]Mulyar N, van der Aalst WMP, Peleg M, A pattern-based analysis of clinical computer-interpretable guideline modeling languages, J. Am. Med. Inform. Assoc 14 (2007) 781– 787, 10.1197/jamia.M2389. 17712087 PMC2213484 · DOI
  39. [41]Peleg M, Tu S, Manindroo A, Altman R, Modeling and analyzing biomedical processes using Work-flow/Petri Net models and tools, Stud. Health Technol. Inform 107 (2004) 74– 78, 10.3233/978-1-60750-949-3-74. 15360778 · DOI
  40. [42]Shahar Y, Young O, Shalom E, Galperin M, Mayaffit A, Moskovitch R, Hessing A, A framework for a distributed, hybrid, multiple-ontology clinical-guideline library, and automated guideline-support tools, J. Biomed. Inform 37 (2004) 325– 344, 10.1016/j.jbi.2004.07.001. 15488747 · DOI
  41. [44]De Roock E, Martin N, Process mining in healthcare – An updated perspective on the state of the art, J. Biomed. Inform 127 (2022), 103995, 10.1016/j.jbi.2022.103995. 35077900 · DOI
  42. [45]Vickers AJ, Van Calster B, Steyerberg EW, Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests, BMJ (2016), i6, 10.1136/bmj.i6. 26810254 PMC4724785 · DOI · Paper page
  43. [46]Baker SG, Putting Risk Prediction in Perspective: Relative Utility Curves, JNCI: Journal of the National Cancer Institute. 101 (2009) 1538– 1542, 10.1093/jnci/djp353. 19843888 PMC2778669 · DOI
  44. [49]Mahoney EM, Wang K, Cohen DJ, Hirsch AT, Alberts MJ, Eagle K, Mosse F, Jackson JD, Steg PG, Bhatt DL, One-Year Costs in Patients With a History of or at Risk for Atherothrombosis in the United States, Circ. Cardiovasc. Qual. Outcomes 1 (2008) 38– 45, 10.1161/CIRCOUTCOMES.108.775247. 20031786 · DOI
  45. [50]Aronow WS, Peripheral arterial disease of the lower extremities, Arch. Med. Sci 8 (2012) 375– 388, 10.5114/aoms.2012.28568. 22662015 PMC3361053 · DOI
  46. [51]Shu J, Santulli G, Update on peripheral artery disease: Epidemiology and evidence-based facts, Atherosclerosis 275 (2018) 379– 381, 10.1016/j.atherosclerosis.2018.05.033. 29843915 PMC6113064 · DOI · Paper page
  47. [52]Chongthawonsatid S, Dutsadeevettakul S, Validity and reliability of the ankle-brachial index by oscillometric blood pressure and automated ankle-brachial index, J. Res. Med. Sci 22 (2017) 44, 10.4103/jrms.JRMS_728_16. 28567064 PMC5426092 · DOI
  48. [53]US Preventive Services Task Force, Curry SJ, Krist AH, Owens DK, Barry MJ, Caughey AB, Davidson KW, Doubeni CA, Epling JW, Kemper AR, Kubik M, Landefeld CS, Mangione CM, Silverstein M, Simon MA, Tseng C-W, Wong JB, Screening for Peripheral Artery Disease and Cardiovascular Disease Risk Assessment With the Ankle-Brachial Index: US Preventive Services Task Force Recommendation Statement, JAMA 320 (2018) 177, 10.1001/jama.2018.8357. 29998344 · DOI
  49. [56]van der Sijs H, Aarts J, Vulto A, Berg M, Overriding of Drug Safety Alerts in Computerized Physician Order Entry, J. Am. Med. Inform. Assoc 13 (2006) 138– 147, 10.1197/jamia.M1809. 16357358 PMC1447540 · DOI · Paper page
  50. [58]Rojas E, Munoz-Gama J, Sepúlveda M, Capurro D, Process mining in healthcare: A literature review, J. Biomed. Inform 61 (2016) 224– 236, 10.1016/j.jbi.2016.04.007. 27109932 · DOI
  51. [59]Fujimoto R, Parallel and distributed simulation, in: Proceedings of the 2015 Winter Simulation Conference, IEEE Press, Huntington Beach, California, 2015: pp. 45– 59.
  52. [60]Jafer S, Liu Q, Wainer G, Synchronization methods in parallel and distributed discrete-event simulation, Simul. Model. Pract. Theory 30 (2013) 54– 73, 10.1016/j.simpat.2012.08.003. · DOI
  53. [61]Li RC, Asch SM, Shah NH, Developing a delivery science for artificial intelligence in healthcare, NPJ Digital Med. 3 (2020) 107, 10.1038/s41746-020-00318-y. PMC7443141 32885053 · DOI

Authors
Wornow, M., Gyang Ross, E., Callahan, A., & Shah, N. H.
Affiliations
  • Stanford University (US)
  • Stanford Medicine (US)
  • Stanford Health Care (US)
Corresponding author
Wornow, Michael
ORCID
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2023-03
Volume
139
Pages
104319
ISSN
1532-0464 (ISSN-L)
DOI
10.1016/j.jbi.2023.104319
PMC
PMC10309067
OpenAlex
W4320487057
Cited by
26
References
62
Citations by year
2026: 6 · 2025: 6 · 2024: 9 · 2023: 5
FWCI
1.04
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Informatics (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion, cc-by-nc-nd)
Funders
  • Gordon and Betty Moore Foundation
  • National Science Foundation
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
Aplus, clinical workflow integration, clinical decision support, simulation engine
MeSH terms
Machine Learning, Computer Simulation, Delivery of Health Care, Humans, Language, Workflow

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review