CoChat
Sign Up
Cited by 52Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareArtificial Intelligence in Healthcare

Machine Learning Operations in Health Care: A Scoping Review

Open access
Rajagopal, Anjali, Ayanian, Shant, Ryu, Alexander J., Qian, Ray, Legler, Sean R., Peeler, Eric A., Issa, Meltiady, Coons, Trevor J. +1 more·2024·Mayo Clinic Proceedings: Digital Health
DOI 10.1016/j.mcpdig.2024.06.009PMC PMC11975983
Open this paper in CoChat — free accountOpen PDF

Abstract

The use of machine learning tools in health care is rapidly expanding. However, the processes that support these tools in deployment, that is, machine learning operations, are still emerging. The purpose of this work was not only to provide a comprehensive synthesis of existing literature in the field but also to identify gaps and offer insights for adoption in clinical practice. A scoping review was conducted using the MEDLINE, PubMed, Google Scholar, Embase, and Scopus databases. We used MeSH and non-MeSH search terms to identify pertinent articles, with the authors performing 2 screening phases and assigning relevance scores: 148 English language articles most salient to the review were eligible for inclusion; 98 offered the most unique information and these were supplemented by 50 additional sources, yielding 148 references. From the 148 references, we distilled 7 key topic areas, based on a synthesis of the available literature and how that aligned with practitioner needs. The 7 topic areas were machine learning model monitoring; automated retraining systems; ethics, equity, and bias; clinical workflow integration; infrastructure, human resources, and technology stack; regulatory considerations; and financial considerations. This review provides an overview of best practices and knowledge gaps of this domain in health care and identifies the strengths and weaknesses of the literature, which may be useful to health care machine learning practitioners and consumers.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.mcpdig.2024.06.009

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 106

  1. [3]Mello M.M. Guha N. Understanding liability risk from using health care artificial intelligence tools N Engl J Med 390 3 2024 271 278 10.1056/NEJMhle2308901 38231630 · DOI · Paper page
  2. [5]Ebert C. Gallardo G. Hernantes J. Serrano N. DevOps IEEE Softw 33 3 2016 94 100 10.1109/MS.2016.68 · DOI
  3. [6]Stirbu V. Granlund T. Mikkonen T. Continuous design control for machine learning in certified medical systems Softw. Qual J 31 2 2023 307 333 10.1007/s11219-022-09601-5 · DOI
  4. [7]Sendak M. Gao M. Nichols M. Lin A. Balu S. Machine learning in health care: a critical appraisal of challenges and opportunities EGEMS (Wash DC) 7 1 2019 1 10.5334/egems.287 30705919 PMC6354017 · DOI
  5. [8]de Hond A.A.H. Leeuwenberg A.M. Hooft L. Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review NPJ Digit Med 5 1 2022 1 13 10.1038/s41746-021-00549-7 35013569 PMC8748878 · DOI · Paper page
  6. [9]Kelly C.J. Karthikesalingam A. Suleyman M. Corrado G. King D. Key challenges for delivering clinical impact with artificial intelligence BMC Med 17 1 2019 195 10.1186/s12916-019-1426-2 31665002 PMC6821018 · DOI · Paper page
  7. [10]Doyen S. Dadario N.B. 12 Plagues of AI in healthcare: a practical guide to current issues with using machine learning in a medical context Front Digit Health 4 2022 765406 PMC9110785 35592460
  8. [11]Rahmani K. Thapa R. Tsou P. Assessing the effects of data drift on the performance of machine learning models used in clinical sepsis prediction Int J Med Inform 173 2023 104930 10.1016/j.ijmedinf.2022.104930 36893656 · DOI
  9. [12]Sahiner B. Chen W. Samala R.K. Petrick N. Data drift in medical machine learning: implications and potential remedies Br J Radiol 96 1150 2023 20220878 10.1259/bjr.20220878 PMC10546450 36971405 · DOI · Paper page
  10. [13]Veeranki S.P.K. Kramer D. Hayn D. Is regular re-training of a predictive delirium model necessary after deployment in routine care? Stud Health Technol Inform 260 2019 186 191 31118336
  11. [14]Moreno-Torres J.G. Raeder T. Alaiz-Rodríguez R. Chawla N.V. Herrera F. A unifying view on dataset shift in classification Pattern Recognit 45 1 2012 521 530 10.1016/j.patcog.2011.06.019 · DOI · Paper page
  12. [15]Singer M. Deutschman C.S. Seymour C.W. The Third International consensus definitions for sepsis and septic shock (Sepsis-3) JAMA 315 8 2016 801 810 10.1001/jama.2016.0287 26903338 PMC4968574 · DOI · Paper page
  13. [16]Duckworth C. Chmiel F.P. Burns D.K. Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during COVID-19 Sci Rep 11 1 2021 23017 10.1038/s41598-021-02481-y PMC8626460 34837021 · DOI · Paper page
  14. [17]Vasilev Y. Vladzymyrskyy A. Arzamasov K. Clinical application of radiological AI for pulmonary nodule evaluation: replicability and susceptibility to the population shift caused by the COVID-19 pandemic Int J Med Inform 178 2023 105190 10.1016/j.ijmedinf.2023.105190 37603940 · DOI
  15. [18]Vela D. Sharp A. Zhang R. Nguyen T. Hoang A. Pianykh O.S. Temporal quality degradation in AI models Sci Rep 12 1 2022 11654 10.1038/s41598-022-15245-z PMC9270447 35803963 · DOI
  16. [19]Ryu A.J. Ayanian S. Qian R. A Clinician’s guide to running custom machine-learning models in an electronic health record environment Mayo Clin Proc 98 3 2023 445 450 10.1016/j.mayocp.2022.11.019 36868752 · DOI
  17. [20]van der Vegt A.H. Scott I.A. Dermawan K. Schnetler R.J. Kalke V.R. Lane P.J. Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework J Am Med Inform Assoc 30 9 2023 1503 1515 10.1093/jamia/ocad088 37208863 PMC10436156 · DOI · Paper page
  18. [21]Adam G.A. Chang C.H.K. Haibe-Kains B. Goldenberg A. Hidden risks of machine learning applied to healthcare: unintended feedback loops between models and future data causing model degradation Proceedings of the 5th Machine Learning for Healthcare Conference 2020 PMLR 710 731 https://proceedings.mlr.press/v126/adam20a.html
  19. [22]Vaid A. Sawant A. Suarez-Farinas M. Implications of the use of artificial intelligence predictive models in health care settings: a simulation study Ann Intern Med 176 10 2023 1358 1369 10.7326/M23-0949 37812781 · DOI
  20. [23]Bai E. Song S.L. Fraser H.S.F. Ranney M.L. A graphical toolkit for longitudinal dataset maintenance and predictive model training in health care Appl Clin Inform 13 1 2022 56 66 10.1055/s-0041-1740923 35172371 PMC8850007 · DOI
  21. [24]Waring J. Lindvall C. Umeton R. Automated machine learning: review of the state-of-the-art and opportunities for healthcare Artif Intell Med 104 2020 101822 10.1016/j.artmed.2020.101822 32499001 · DOI · Paper page
  22. [25]Davis S.E. Greevy R.A. Lasko T.A. Walsh C.G. Matheny M.E. Detection of calibration drift in clinical prediction models to inform model updating J Biomed Inform 112 2020 103611 10.1016/j.jbi.2020.103611 PMC8627243 33157313 · DOI · Paper page
  23. [27]Davis S.E. Greevy R.A. Jr. Fonnesbeck C. Lasko T.A. Walsh C.G. Matheny M.E. A nonparametric updating method to correct clinical prediction model drift J Am Med Inform Assoc 26 12 2019 1448 1457 10.1093/jamia/ocz127 31397478 PMC6857513 · DOI
  24. [29]Del Fiol G. Haug P.J. Infobuttons and classification models: a method for the automatic selection of on-line information resources to fulfill clinicians’ information needs J Biomed Inform 41 4 2008 655 666 10.1016/j.jbi.2007.11.007 18249041 PMC2644726 · DOI
  25. [30]Chen J. Zheng Y. Liang Y. Edge2Analysis: a novel AIoT platform for atrial fibrillation recognition and detection IEEE J Biomed Health Inform 26 12 2022 5772 5782 10.1109/JBHI.2022.3171918 35511842 · DOI
  26. [31]Toor A.A. Usman M. Younas F. M Fong A.C. Khan S.A. Fong S. Mining massive E-health data streams for IoMT enabled healthcare systems Sensors 20 7 2020 2131 10.3390/s20072131 32283841 PMC7180875 · DOI
  27. [32]Zhang X. Xue Y. Su X. A transfer learning approach to correct the temporal performance drift of clinical prediction models: retrospective cohort study JMIR Med Inform 10 11 2022 e38053 10.2196/38053 PMC9685506 36350705 · DOI
  28. [35]Wiens J. Saria S. Sendak M. Do no harm: a roadmap for responsible machine learning for health care Nat Med 25 9 2019 1337 1340 10.1038/s41591-019-0548-6 31427808 · DOI · Paper page
  29. [36]Mehrabi N. Morstatter F. Saxena N. Lerman K. Galstyan A. A survey on bias and fairness in machine learning ACM Comput Surv 54 6 2019 1 35 10.1145/3457607 · DOI · Paper page
  30. [37]Hardt M. Price E. Price E. Srebro N. Equality of opportunity in supervised learning. In: Advances in Neural Information Processing Systems. Vol 29. Curran Associates https://proceedings.neurips.cc/paper_files/paper/2016/hash/9d2682367c3935defcb1f9e247a97c0d-Abstract.html 2016
  31. [38]Grote T. Keeling G. Enabling fairness in healthcare through machine learning Ethics Inf Technol 24 3 2022 39 10.1007/s10676-022-09658-7 36060496 PMC9428374 · DOI
  32. [39]Rajkomar A. Hardt M. Howell M.D. Corrado G. Chin M.H. Ensuring fairness in machine learning to advance health equity Ann Intern Med 169 12 2018 866 872 10.7326/M18-1990 30508424 PMC6594166 · DOI · Paper page
  33. [40]Rueda J. Rodríguez J.D. Jounou I.P. Hortal-Carmona J. Ausín T. Rodríguez-Arias D. “Just” accuracy? Procedural fairness demands explainability in AI-based medical resource allocations AI Soc Published online December 21, 2022 10.1007/s00146-022-01614-9 PMC9769482 36573157 · DOI
  34. [42]Char D.S. Abràmoff M.D. Feudtner C. Identifying ethical considerations for machine learning healthcare applications Am J Bioeth 20 11 2020 7 17 10.1080/15265161.2020.1819469 PMC7737650 33103967 · DOI · Paper page
  35. [44]McCradden M.D. Anderson J.A. A Stephenson E. A research ethics framework for the clinical translation of healthcare machine learning Am J Bioeth 22 5 2022 8 22 10.1080/15265161.2021.2013977 35048782 · DOI
  36. [45]Kleppe A. Skrede O.J. De Raedt S. Liestøl K. Kerr D.J. Danielsen H.E. Designing deep learning studies in cancer diagnostics Nat Rev Cancer 21 3 2021 199 211 10.1038/s41568-020-00327-9 33514930 · DOI
  37. [46]Drukker K. Chen W. Gichoya J. Toward fairness in artificial intelligence for medical image analysis: identification and mitigation of potential biases in the roadmap from data collection to model deployment J Med Imaging (Bellingham) 10 6 2023 061104 10.1117/1.JMI.10.6.061104 PMC10129875 37125409 · DOI · Paper page
  38. [48]Cohen I.G. Amarasingham R. Shah A. Xie B. Lo B. The legal and ethical concerns that arise from using complex predictive analytics in health care Health Aff (Millwood) 33 7 2014 1139 1147 10.1377/hlthaff.2014.0048 25006139 · DOI · Paper page
  39. [49]Pleiss G. Raghavan M. Wu F. Kleinberg J. Weinberger K.Q. On fairness and calibration Advances in Neural Information Processing Systems 30 2017 Curran Associates https://proceedings.neurips.cc/paper_files/paper/2017/hash/b8b9c74ac526fffbeb2d39ab038d1cd7-Abstract.html
  40. [50]Chomutare T. Tejedor M. Svenning T.O. Artificial intelligence implementation in healthcare: a theory-based scoping review of barriers and facilitators Int J Environ Res Public Health 19 23 2022 16359 10.3390/ijerph192316359 PMC9738234 36498432 · DOI
  41. [51]Yogarajan V. Dobbie G. Leitch S. Data and model bias in artificial intelligence for healthcare applications in New Zealand Front Comput Sci 4 2022 10.3389/fcomp.2022.1070493 · DOI
  42. [52]Meetings examine impact of healthcare algorithms on racial and ethnic disparities in health and healthcare. Effective Health Care (EHC) Program https://effectivehealthcare.ahrq.gov/news/meetings 38147523
  43. [53]Kawamoto K. Finkelstein J. Del Fiol G. Implementing machine learning in the electronic health record: checklist of essential considerations Mayo Clin Proc 98 3 2023 366 369 10.1016/j.mayocp.2023.01.013 36868743 · DOI
  44. [54]Watson J. Hutyra C.A. Clancy S.M. Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers? JAMIA Open 3 2 2020 167 172 10.1093/jamiaopen/ooz046 32734155 PMC7382631 · DOI · Paper page
  45. [55]Schaarup J.F.R. Aggarwal R. Dalsgaard E.M. Perception of artificial intelligence-based solutions in healthcare among people with and without diabetes: a cross-sectional survey from the health in Central Denmark cohort Diabetes Epidemiol Manag 9 2023 100114 10.1016/j.deman.2022.100114 · DOI
  46. [56]McIntosh C. Conroy L. Tjong M.C. Clinical integration of machine learning for curative-intent radiation treatment of patients with prostate cancer Nat Med 27 6 2021 999 1005 10.1038/s41591-021-01359-w 34083812 · DOI
  47. [57]Wang S.M. Hogg H.D.J. Sangvai D. Development and integration of machine learning algorithm to identify peripheral arterial disease: multistakeholder qualitative study JMIR Form Res 7 2023 e43963 10.2196/43963 PMC10557008 37733427 · DOI
  48. [58]Ng R. Tan K.B. Implementing an individual-centric discharge process across Singapore public hospitals Int J Environ Res Public Health 18 16 2021 8700 10.3390/ijerph18168700 34444448 PMC8393960 · DOI
  49. [59]Engstrom C.J. Adelaine S. Liao F. Jacobsohn G.C. Patterson B.W. Operationalizing a real-time scoring model to predict fall risk among older adults in the emergency department Front Digit Health 4 2022 958663 https://www.frontiersin.org/articles/10.3389/fdgth.2022.958663 PMC9671211 36405416 · DOI
  50. [60]Moorman L.P. Principles for real-world implementation of bedside predictive analytics monitoring Appl Clin Inform 12 4 2021 888 896 10.1055/s-0041-1735183 34553360 PMC8458037 · DOI · Paper page
  51. [61]Wolff R.F. Moons K.G.M. Riley R.D. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies Ann Intern Med 170 1 2019 51 58 10.7326/M18-1376 30596875 · DOI · Paper page
  52. [62]Collins G.S. Reitsma J.B. Altman D.G. Moons K.G.M. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement Ann Intern Med 162 1 2015 55 63 10.7326/M14-0697 25560714 · DOI · Paper page
  53. [65]Bartels R. Dudink J. Haitjema S. Oberski D. van ’t Veen A. A perspective on a quality management system for AI/ML-based clinical decision support in hospital care Front Digit Health 4 2022 942588 10.3389/fdgth.2022.942588 PMC9299425 35873347 · DOI
  54. [67]Cai C.J. Winter S. Steiner D. Wilcox L. Terry M. “Hello AI”: uncovering the onboarding needs of medical practitioners for human-AI collaborative decision-making Proc ACM Hum-Comput Interact 3 CSCW 2019 1 24 10.1145/3359206 · DOI · Paper page
  55. [68]Verma A.A. Murray J. Greiner R. Implementing machine learning in medicine CMAJ 193 34 2021 E1351 E1357 10.1503/cmaj.202434 35213323 PMC8432320 · DOI
  56. [70]Hribar M.R. Read-Brown S. Goldstein I.H. Secondary use of electronic health record data for clinical workflow analysis J Am Med Inform Assoc 25 1 2018 40 46 10.1093/jamia/ocx098 29036581 PMC6080808 · DOI
  57. [71]Vankipuram A. Traub S. Patel V.L. A method for the analysis and visualization of clinical workflow in dynamic environments J Biomed Inform 79 2018 20 31 10.1016/j.jbi.2018.01.007 29410146 · DOI
  58. [72]Jarab A.S. Al-Qerem W. Alzoubi K.H. Artificial intelligence in pharmacy practice: attitude and willingness of the community pharmacists and the barriers for its implementation Saudi Pharm J 31 8 2023 101700 10.1016/j.jsps.2023.101700 PMC10404546 37555012 · DOI
  59. [73]Neher M. Petersson L. Nygren J.M. Svedberg P. Larsson I. Nilsen P. Innovation in healthcare: leadership perceptions about the innovation characteristics of artificial intelligence-a qualitative interview study with healthcare leaders in Sweden Implement Sci Commun 4 1 2023 81 10.1186/s43058-023-00458-8 37464420 PMC10354990 · DOI
  60. [74]Akudjedu T.N. Torre S. Khine R. Katsifarakis D. Newman D. Malamateniou C. Knowledge, perceptions, and expectations of artificial intelligence in radiography practice: a global radiography workforce survey J Med Imaging Radiat Sci 54 1 2023 104 116 10.1016/j.jmir.2022.11.016 36535859 · DOI
  61. [75]Al-Medfa M.K. Al-Ansari A.M.S. Darwish A.H. Qreeballa T.A. Jahrami H. Physicians’ attitudes and knowledge toward artificial intelligence in medicine: benefits and drawbacks Heliyon 9 4 2023 e14744 10.1016/j.heliyon.2023.e14744 PMC10073828 37035387 · DOI · Paper page
  62. [76]Chen Y. Wu Z. Wang P. Radiology residents’ perceptions of artificial intelligence: nationwide cross-sectional survey study J Med Internet Res 25 2023 e48249 10.2196/48249 PMC10623237 37856181 · DOI
  63. [77]Tanaka M. Matsumura S. Bito S. Roles and competencies of doctors in artificial intelligence implementation: qualitative analysis through physician interviews JMIR Form Res 7 2023 e46020 10.2196/46020 PMC10236283 37200074 · DOI
  64. [78]Ahmad M.N. Abdallah S.A. Abbasi S.A. Abdallah A.M. Student perspectives on the integration of artificial intelligence into healthcare services Digit Health 9 2023 20552076231174095 10.1177/20552076231174095 PMC10259127 37312954 · DOI
  65. [79]Abuzaid M.M. Elshami W. Fadden S.M. Integration of artificial intelligence into nursing practice Health Technol (Berl) 12 6 2022 1109 1115 10.1007/s12553-022-00697-0 36117522 PMC9470236 · DOI · Paper page
  66. [80]Sezgin E. Sirrianni J. Linwood S.L. Operationalizing and implementing pretrained, large artificial intelligence linguistic models in the US health care system: outlook of generative pretrained transformer 3 (GPT-3) as a service model JMIR Med Inform 10 2 2022 e32875 10.2196/32875 PMC8874824 35142635 · DOI
  67. [81]Hogg H.D.J. Al-Zubaidy M. Technology Enhanced Macular Services Study Reference Group Talks J. Denniston A.K. Kelly C.J. Stakeholder perspectives of clinical artificial intelligence implementation: systematic review of qualitative evidence J Med Internet Res 25 1 2023 e39742 10.2196/39742 PMC9875023 36626192 · DOI · Paper page
  68. [82]Kruse C.S. Smith B. Vanderlinden H. Nealand A. Security techniques for the electronic health records J Med Syst 41 8 2017 127 10.1007/s10916-017-0778-4 28733949 PMC5522514 · DOI · Paper page
  69. [83]Price W.N. Cohen I.G. Privacy in the age of medical big data Nat Med 25 1 2019 37 43 10.1038/s41591-018-0272-7 30617331 PMC6376961 · DOI · Paper page
  70. [84]Marinescu D.C. Cloud Computing: Theory and Practice 2022 Morgan Kaufmann
  71. [85]Darwish A. Hassanien A.E. Elhoseny M. Sangaiah A.K. Muhammad K. The impact of the hybrid platform of internet of things and cloud computing on healthcare systems: opportunities, challenges, and open problems J Ambient Intell Human Comput 10 10 2019 4151 4166 10.1007/s12652-017-0659-1 · DOI
  72. [86]Kuo A.M. Opportunities and challenges of cloud computing to improve health care services J Med Internet Res 13 3 2011 e67 10.2196/jmir.1867 PMC3222190 21937354 · DOI
  73. [88]Kwasniewska A. Raghava S. Davila C. Sevenier M. Gamba D. Ruminski J. Preferred benchmarking criteria for systematic taxonomy of embedded platforms (STEP) in human system interaction systems 2022 15th International Conference on Human System Interaction (HSI) 2022 IEEE 1 7 10.1109/HSI55341.2022.9869470 · DOI
  74. [89]Pianykh O.S. Guitron S. Parke D. Improving healthcare operations management with machine learning Nat Mach Intell 2 5 2020 266 273 10.1038/s42256-020-0176-3 · DOI · Paper page
  75. [90]Baier L. Jöhren F. Seebacher S. Challenges in the deployment and operation of machine learning in practice May 2019 Presented at: ECIS 2019—27th European Conference on Information Systems Stockholm
  76. [91]Vassiliadis P. Simitsis A. Skiadopoulos S. Conceptual modeling for ETL processes Proceedings of the 5th ACM International Workshop on Data Warehousing and OLAP. DOLAP ’02 2002 Association for Computing Machinery 14 21 10.1145/583890.583893 · DOI
  77. [92]Garg S. Pundir P. Rathee G. Gupta P.K. Garg S. Ahlawat S. On continuous integration/continuous delivery for automated deployment of machine learning models using MLOps 2021 IEEE Fourth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE) 2021 IEEE 25 28 10.1109/AIKE52691.2021.00010 · DOI
  78. [93]Granlund T. Stirbu V. Mikkonen T. Towards regulatory-compliant MLOps: Oravizio’s journey from a machine learning experiment to a deployed certified medical product SN Comput Sci 2 5 2021 342 10.1007/s42979-021-00726-1 · DOI · Paper page
  79. [94]Kleftakis S. Mavrogiorgou A. Mavrogiorgos K. Kiourtis A. Kyriazis D. Digital twin in healthcare through the eyes of the vitruvian man Chen Y.W. Tanaka S. Howlett R.J. Jain L.C. Innovation in Medicine and Healthcare. Smart Innovation, Systems and Technologies 2022 Springer Nature 75 85 10.1007/978-981-19-3440-7_7 · DOI
  80. [95]Soh J. Singh P. Machine learning operations Soh J. Singh P. Data Science Solutions on Azure: Tools and Techniques Using Databricks and MLOps 2020 Apress 259 279 10.1007/978-1-4842-6405-8_8 · DOI · Paper page
  81. [96]Bhavsar K. Vishwakarma H. Pawar B.R. Shinde S. Kimbahune S. Ghose A. A platform to enable algorithms as service model aimed at digital health service delivery 2022 14th International Conference on Communication Systems & Networks (COMSNETS) 2022 IEEE 241 245 10.1109/COMSNETS53615.2022.9668453 · DOI
  82. [97]Yoo J. Lee J. Min J.Y. Development of an interoperable and easily transferable clinical decision support system deployment platform: system design and development study J Med Internet Res 24 7 2022 e37928 10.2196/37928 PMC9377482 35896020 · DOI
  83. [98]Cresswell K. Rigby M. Magrabi F. The need to strengthen the evaluation of the impact of artificial intelligence-based decision support systems on healthcare provision Health Policy 136 2023 104889 10.1016/j.healthpol.2023.104889 37579545 · DOI
  84. [99]Colantonio S. Berti A. Buongiorno R. AI trustworthiness in prostate cancer imaging: a look at algorithmic and system transparency 2023 IEEE EMBS Special Topic Conference on Data Science and Engineering in Healthcare, Medicine and Biology 2023 IEEE 79 80 10.1109/IEEECONF58974.2023.10404432 · DOI
  85. [101]Tarabichi Y. Higginbotham J. Riley N. Kaelber D.C. Watts B. Reducing disparities in no show rates using predictive model-driven live appointment reminders for at-risk patients: a randomized controlled quality improvement initiative J Gen Intern Med 38 13 2023 2921 2927 10.1007/s11606-023-08209-0 37126125 PMC10150669 · DOI
  86. [102]Assadi A. Laussen P.C. Goodwin A.J. An integration engineering framework for machine learning in healthcare Front Digit Health 4 2022 932411 10.3389/fdgth.2022.932411 PMC9386122 35990013 · DOI
  87. [103]Sperrin M. Martin G.P. Sisk R. Peek N. Missing data should be handled differently for prediction than for description or causal explanation J Clin Epidemiol 125 2020 183 187 10.1016/j.jclinepi.2020.03.028 32540389 · DOI
  88. [104]Li R.C. Asch S.M. Shah N.H. Developing a delivery science for artificial intelligence in healthcare NPJ Digit Med 3 1 2020 1 3 10.1038/s41746-020-00318-y 32885053 PMC7443141 · DOI
  89. [105]Wiesenfeld B.M. Aphinyanaphongs Y. Nov O. AI model transferability in healthcare: a sociotechnical perspective Nat Mach Intell 4 10 2022 807 809 10.1038/s42256-022-00544-x · DOI
  90. [106]Hofer I.S. Burns M. Kendale S. Wanderer J.P. Realistically integrating machine learning into clinical practice: a road map of opportunities, challenges, and a potential future Anesth Analg 130 5 2020 1115 1118 10.1213/ANE.0000000000004575 32287118 PMC7584400 · DOI
  91. [107]Shaw J. Rudzicz F. Jamieson T. Goldfarb A. Artificial intelligence and the implementation challenge J Med Internet Res 21 7 2019 e13659 10.2196/13659 PMC6652121 31293245 · DOI · Paper page
  92. [108]Shah P. Kendall F. Khozin S. Artificial intelligence and machine learning in clinical development: a translational perspective NPJ Digit Med 2 1 2019 69 10.1038/s41746-019-0148-3 31372505 PMC6659652 · DOI
  93. [109]Zhang J. Budhdeo S. William W. Moving towards vertically integrated artificial intelligence development NPJ Digit Med 5 1 2022 143 10.1038/s41746-022-00690-x 36104535 PMC9474277 · DOI · Paper page
  94. [110]Leming M.J. Bron E.E. Bruffaerts R. Challenges of implementing computer-aided diagnostic models for neuroimages in a clinical setting NPJ Digit Med 6 1 2023 129 10.1038/s41746-023-00868-x 37443276 PMC10345121 · DOI
  95. [115]Buiten M.C. Towards intelligent regulation of artificial intelligence Eur J Risk Regulat 10 1 2019 41 59 10.1017/err.2019.8 · DOI
  96. [118]Nilsen P. Reed J. Nair M. Realizing the potential of artificial intelligence in healthcare: learning from intervention, innovation, implementation and improvement sciences Front Health Serv 2 2022 961475 10.3389/frhs.2022.961475 PMC10012740 36925879 · DOI
  97. [119]Shashikumar S.P. Amrollahi F. Nemati S. Unsupervised detection and correction of model calibration shift at test-time Annu Int Conf IEEE Eng Med Biol Soc 2023 2023 1 4 10.1109/EMBC40787.2023.10341086 PMC10805329 38083174 · DOI
  98. [122]Pica F. AI, genetic testing: what washed up at Epic’s global conference. The Cap Times https://captimes.com/news/business/ai-genetic-testing-what-washed-up-at-epics-global-conference/article_4cc21fb9-79f6-533a-b4f7-300fddcce032.html
  99. [124]Liao F. Adelaine S. Afshar M. Patterson B.W. Governance of clinical AI applications to facilitate safe and equitable deployment in a large health system: key elements and early successes Front Digit Health 4 2022 931439 10.3389/fdgth.2022.931439 PMC9448877 36093386 · DOI
  100. [125]Naik N. Hameed B.M.Z. Shetty D.K. Legal and ethical consideration in artificial intelligence in healthcare: who takes responsibility? Front Surg 9 2022 862322 10.3389/fsurg.2022.862322 PMC8963864 35360424 · DOI · Paper page
  101. [127]Tolentino R. Baradaran A. Gore G. Pluye P. Abbasgholizadeh-Rahimi S. Curriculum frameworks and educational programs in artificial intelligence for medical students, residents, and practicing physicians: a scoping review protocol JBI Evid Synth 21 7 2023 1477 1484 10.11124/JBIES-22-00374 37434376 · DOI
  102. [128]Strubell E. Ganesh A. McCallum A. Energy and policy considerations for deep learning in NLP Preprint Posted online June 5, 2019 10.48550/arXiv.1906.02243 · DOI · Paper page
  103. [129]Sharir O. Peleg B. Shoham Y. The cost of training NLP models: a concise overview Preprint Published online April 19, 2020 10.48550/arXiv.2004.08900 · DOI · Paper page
  104. [130]Khanna N.N. Maindarkar M.A. Viswanathan V. Economics of artificial intelligence in healthcare: diagnosis vs. treatment Healthcare (Basel) 10 12 2022 2493 10.3390/healthcare10122493 36554017 PMC9777836 · DOI · Paper page
  105. [133]Kacew A.J. Strohbehn G.W. Saulsberry L. Artificial intelligence can cut costs while maintaining accuracy in colorectal cancer genotyping Front Oncol 11 2021 630953 10.3389/fonc.2021.630953 PMC8217761 34168975 · DOI
  106. [138]Campbell R.J. The five “rights” of clinical decision support J AHIMA 84 10 2013 42 47 [quiz 48] 24245088

Cited by

Sorted by citations

Authors
Rajagopal, A., Ayanian, S., Ryu, A. J., Qian, R., Legler, S. R., Peeler, E. A., Issa, M., Coons, T. J., & Kawamoto, K.
Affiliations
  • Mayo Clinic (US)
  • Cleveland Clinic Abu Dhabi (AE)
  • University of Utah (US)
Corresponding author
Ryu, Alexander J.
ORCID
Venue
Mayo Clinic Proceedings: Digital Health
Publisher
Elsevier BV
Published
2024-09
Volume
2
Issue
3
Pages
421-437
ISSN
2949-7612 (ISSN-L)
DOI
10.1016/j.mcpdig.2024.06.009
PMC
PMC11975983
OpenAlex
W4400618004
Cited by
52
References
18
Citations by year
2026: 29 · 2025: 20 · 2024: 3
FWCI
1.19
Type
journal article
Language
en
Open access
Licence
Locations
  • Mayo Clinic Proceedings Digital Health (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
MLOps, algorithmic bias, health care equity, clinical workflow integration, regulatory considerations, financial considerations, clinical adoption of AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review