Implementation of Machine Learning Applications in Health Care Organizations: Systematic Review of Empirical Studies
Open accessPublished version of a preprint
The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.
Abstract
Background: There is a growing enthusiasm for machine learning (ML) among academics and health care practitioners. Despite the transformative potential of ML-based applications for patient care, their uptake and implementation in health care organizations are sporadic. Numerous challenges currently impede or delay the widespread implementation of ML in clinical practice, and limited knowledge is available regarding how these challenges have been addressed. Objective: This work aimed to (1) examine the characteristics of ML-based applications and the implementation process in clinical practice, using the Consolidated Framework for Implementation Research (CFIR) for theoretical guidance and (2) synthesize the strategies adopted by health care organizations to foster successful implementation of ML. Methods: A systematic literature review was conducted based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. The search was conducted in PubMed, Scopus, and Web of Science over a 10-year period (2013-2023). The search strategy was built around 4 blocks of keywords (artificial intelligence, implementation, health care, and study type). Only empirical studies documenting the implementation of ML applications in clinical settings were considered. The implementation process was investigated using a thematic analysis and coding procedure. Results: Thirty-four studies were selected for data synthesis. Selected papers were relatively recent, with only 9% (3/34) of records published before 2019. ML-based applications were implemented mostly within hospitals (29/34, 85%). In terms of clinical workflow, ML-based applications supported mostly prognosis (20/34, 59%) and diagnosis (10/34, 29%). The implementation efforts were analyzed using CFIR domains. As for the inner setting domain, access to knowledge and information (12/34, 35%), information technology infrastructure (11/34, 32%), and organizational culture (9/34, 26%) were among the most observed dimensions influencing the success of implementation. As for the ML innovation itself, factors deemed relevant were its design (15/34, 44%), the relative advantage with respect to existing clinical practice (14/34, 41%), and perceived complexity (14/34, 41%). As for the other domains (ie, processes, roles, and outer setting), stakeholder engagement (12/34, 35%), reflecting and evaluating practices (11/34, 32%), and the presence of implementation leaders (9/34, 26%) were the main factors identified as important. Conclusions: This review sheds some light on the factors that are relevant and that should be accounted for in the implementation process of ML-based applications in health care. While the relevance of ML-specific dimensions, like trust, emerges clearly across several implementation domains, the evidence from this review highlighted that relevant implementation factors are not necessarily specific for ML but rather transversal for digital health technologies. More research is needed to further clarify the factors that are relevant to implementing ML-based applications at the organizational level and to support their uptake within health care organizations. Trial Registration: PROSPERO 403873; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=403873 International Registered Report Identifier (IRRID): RR2-10.2196/47971
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 1×
“The recently updated version of the Consolidated Framework for Implementation Research (CFIR), a commonly used model to assess factors influencing implementation and to explain barriers and facilitators to implementation effectiveness [27,28], was used as a criterion for inclusion.”
Valente, T. W. (2003) · Genetics in Medicine · doi:10.1097/01.gim.0000061743.67794.c4 · Cited by 13,723
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312
Rudd, K. E., Johnson, S. C., Agesa, K. M., Shackelford, K. A., Tsoi, D., Kievlan, D. R., Colombara, D. V., Ikuta, K. S., Kissoon, N., Finfer, S., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(19)32989-7 · Cited by 7,971 · cited 1×
“The only condition for which we observed a conspicuous number of studies was sepsis, a dysfunction accounting for around 20% of deaths worldwide [72], for which ML-based applications are proliferating [73], although no definitive causal link with reduced mortality has been demonstrated to date [74].”
Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations
GREENHALGH, T., ROBERT, G., MACFARLANE, F., BATE, P., & KYRIAKIDOU, O. (2004) · The Milbank Quarterly · doi:10.1111/j.0887-378x.2004.00325.x · Cited by 7,448 · cited 1×
“We adopted the definition of implementation as an “active and planned effort to mainstream innovation within an organization” [26], while health care organizations encompassed all entities delivering health services, including hospitals, outpatient centers, primary care facilities, and public…”
The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., et al. (2018) · Education for Information · doi:10.3233/efi-180221 · Cited by 4,737 · cited 1×
“Critical appraisal of the studies selected for data synthesis was performed using the Mixed Method Appraisal Tool (MMAT [29]), which has been designed specifically for systematic reviews that include heterogeneous studies, as it allows to assess the methodological quality of 5 types of study…”
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
The updated consolidated framework for implementation research based on user feedback
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022) · Implementation Science · doi:10.1186/s13012-022-01245-0 · Cited by 4,452 · cited 1×
“The recently updated version of the Consolidated Framework for Implementation Research (CFIR), a commonly used model to assess factors influencing implementation and to explain barriers and facilitators to implementation effectiveness [27,28], was used as a criterion for inclusion.”
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136 · cited 1×
“…that the outer setting domain was largely overlooked, although prior studies have highlighted the importance of external factors, such as data privacy and security laws, ethical issues, regulatory frameworks, and medical liability, in implementing ML applications in clinical practice [5,76-78].”
Human trust in artificial intelligence: Review of empirical research
Glikson, E., & Woolley, A. W. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0057 · Cited by 2,921 · cited 1×
“ML applications based on computational models are often characterized as opaque (ie, black boxes), introducing an extra layer of complexity to the trust relationship between end users and technological innovations [80].”
Big Data and Machine Learning in Health Care
Beam, A. L., & Kohane, I. S. (2018) · JAMA · doi:10.1001/jama.2017.18391 · Cited by 2,506
The practical implementation of artificial intelligence technologies in medicine
He, J., Baxter, S. L., Xu, J., Xu, J., Zhou, X., & Zhang, K. (2019) · Nature Medicine · doi:10.1038/s41591-018-0307-0 · Cited by 2,246 · cited 2×
“Challenges for the implementation of AI systems, without an exclusive focus on ML, have been previously outlined, with various interdependent factors at different stakeholder group levels [9,10].”
Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30123-2 · Cited by 1,883 · cited 1×
“Although some AI systems have already been shown to be equal or even superior in performance to HCPs [6], full automation of a broad range of human tasks is expected to occur only at later stages.”
Implementing Machine Learning in Health Care — Addressing Ethical Challenges
Char, D. S., Shah, N. H., & Magnus, D. (2018) · New England Journal of Medicine · doi:10.1056/nejmp1714229 · Cited by 1,857 · cited 1×
“Notions of patient confidentiality and privacy should be reimagined entirely as data must be shared across multiple institutions to maximize their value and allow for improved algorithms [14].”
Can we open the black box of AI?
Castelvecchi, D. (2016) · Nature · doi:10.1038/538020a · Cited by 1,773 · cited 1×
“ML algorithms are often associated with the so-called “black box” effect [11,12].”
The role of artificial intelligence in healthcare: a structured literature review
Secinaro, S., Calandra, D., Secinaro, A., Muthurangu, V., & Biancone, P. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01488-9 · Cited by 1,085
Sun, T. Q., & Medaglia, R. (2019) · Government Information Quarterly · doi:10.1016/j.giq.2018.09.008 · Cited by 948 · cited 9×
“Themes, such as data collection and quality [39,58,59], security [33], availability [36,38], and sharing [39,61], were highly described as challenges for the adoption of the application.”
InfluentialHengstler, M., Enkel, E., & Duelli, S. (2016) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2015.12.014 · Cited by 661 · cited 10×
“Themes, such as data collection and quality [39,58,59], security [33], availability [36,38], and sharing [39,61], were highly described as challenges for the adoption of the application.”
InfluentialO’Connor, S., Hanlon, P., O’Donnell, C. A., Garcia, S., Glanville, J., & Mair, F. S. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0359-3 · Cited by 636
Lebovitz, S., Lifshitz-Assaf, H., & Levina, N. (2022) · Organization Science · doi:10.1287/orsc.2021.1549 · Cited by 632 · cited 5×
“On the one hand, faster decision-making could be considered a potential advantage [47], and on the other hand, human-machine interaction could lead to a loss of efficiency compared to human intervention only [40,56].”
InfluentialSecules, S., McCall, C., Mejia, J. A., Beebe, C., Masters, A. S., L. Sánchez‐Peña, M., & Svyantek, M. (2021) · Journal of Engineering Education · doi:10.1002/jee.20377 · Cited by 463 · cited 1×
“Positionality refers to how individuals identify with and relate to different social dimensions such as gender, race, and ethnicity [25], and as such is a relevant aspect to consider in qualitative research.”
Artificial Intelligence in Health Care: Bibliometric Analysis
Guo, Y., Hao, Z., Zhao, S., Gong, J., & Yang, F. (2020) · Journal of Medical Internet Research · doi:10.2196/18228 · Cited by 461 · cited 1×
“To date, research on ML implementation has been predominantly conceptual in nature, with an underreporting of empirical investigations into the specifics and consequences of implementation processes in real-life settings [18,19].”
Clinical applications of machine learning algorithms: beyond the black box
Watson, D. S., Krutzinna, J., Bruce, I. N., Griffiths, C. E., McInnes, I. B., Barnes, M. R., & Floridi, L. (2019) · BMJ · doi:10.1136/bmj.l886 · Cited by 450 · cited 1×
“ML algorithms are often associated with the so-called “black box” effect [11,12].”
Petersson, L., Larsson, I., Nygren, J. M., Nilsen, P., Neher, M., Reed, J. E., Tyskbo, D., & Svedberg, P. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08215-8 · Cited by 434 · cited 1×
“…that the outer setting domain was largely overlooked, although prior studies have highlighted the importance of external factors, such as data privacy and security laws, ethical issues, regulatory frameworks, and medical liability, in implementing ML applications in clinical practice [5,76-78].”
Role of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review
Yin, J., Ngiam, K. Y., & Teo, H. H. (2021) · Journal of Medical Internet Research · doi:10.2196/25759 · Cited by 419 · cited 1×
“Different from prior work encompassing logic-based and rule-based applications [10,70,71], our study focused exclusively on ML-based applications.”
Artificial intelligence and the implementation challenge
Shaw, J., Rudzicz, F., Jamieson, T., & Goldfarb, A. (2019) · Journal of Medical Internet Research · doi:10.2196/13659 · Cited by 389 · cited 2×
“…that the outer setting domain was largely overlooked, although prior studies have highlighted the importance of external factors, such as data privacy and security laws, ethical issues, regulatory frameworks, and medical liability, in implementing ML applications in clinical practice [5,76-78].”
Strohm, L., Hehakaya, C., Ranschaert, E. R., Boon, W. P. C., & Moors, E. H. M. (2020) · European Radiology · doi:10.1007/s00330-020-06946-y · Cited by 329 · cited 10×
“Additionally, trust in performance could be fueled by experience, the application’s ability to identify cases missed by humans, and the consistency over time of recommendations [47].”
InfluentialSendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15182 · Cited by 287 · cited 7×
“Regardless, feedback collection was described as an iterative activity [46,48,60,61], which also positively influenced professional buy-in [47].”
InfluentialPrediction of sepsis patients using machine learning approach: a meta-analysis
Islam, M. M., Nasrin, T., Walther, B. A., Wu, C.-C., Yang, H.-C., & Li, Y.-C. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2018.12.027 · Cited by 259 · cited 1×
“The only condition for which we observed a conspicuous number of studies was sepsis, a dysfunction accounting for around 20% of deaths worldwide [72], for which ML-based applications are proliferating [73], although no definitive causal link with reduced mortality has been demonstrated to date [74].”
Applications of Machine Learning in Real-Life Digital Health Interventions: Review of the Literature
Triantafyllidis, A. K., & Tsanas, A. (2019) · Journal of Medical Internet Research · doi:10.2196/12286 · Cited by 243 · cited 1×
“Available studies have primarily focused on the quantitative impact of ML algorithms on health outcomes or accuracy, without examining the corresponding implementation processes [20].”
Gama, F., Tyskbo, D., Nygren, J., Barlow, J., Reed, J., & Svedberg, P. (2022) · Journal of Medical Internet Research · doi:10.2196/32215 · Cited by 216 · cited 1×
“Several implementation frameworks for health care technologies are on hand, but no widely recognized model addresses all the specific issues that are relevant to ML applications [15-17].”
Elements of Trust in Digital Health Systems: Scoping Review
Adjekum, A., Blasimme, A., & Vayena, E. (2018) · Journal of Medical Internet Research · doi:10.2196/11254 · Cited by 211 · cited 2×
“A recent review by Adjekum et al [79] categorized factors influencing trust in digital health systems into personal, technological, and institutional elements.”
Henry, K. E., Kornfield, R., Sridharan, A., Linton, R. C., Groh, C., Wang, T., Wu, A., Mutlu, B., & Saria, S. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00597-7 · Cited by 207 · cited 7×
“Professional communities and peers could also trigger external pressure that may positively impact the willingness to implement ML applications [54].”
InfluentialArtificial Intelligence Applications in Health Care Practice: Scoping Review
Sharma, M., Savage, C., Nair, M., Larsson, I., Svedberg, P., & Nygren, J. M. (2022) · Journal of Medical Internet Research · doi:10.2196/40238 · Cited by 198 · cited 3×
“This could impact the generalizability of our findings, as acknowledged in similar studies such as the study by Sharma et al [70].”
Developing a delivery science for artificial intelligence in healthcare
Li, R. C., Asch, S. M., & Shah, N. H. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00318-y · Cited by 193 · cited 1×
“To date, research on ML implementation has been predominantly conceptual in nature, with an underreporting of empirical investigations into the specifics and consequences of implementation processes in real-life settings [18,19].”
Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study
Sandhu, S., Lin, A. L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A. D., Balu, S., O'Brien, C., & Sendak, M. P. (2020) · Journal of Medical Internet Research · doi:10.2196/22421 · Cited by 168 · cited 9×
“The most recurrent themes within this construct revolved around ease of use and intuitive design [33,45,49,62].”
InfluentialHogg, H. D. J., Al-Zubaidy, M., Technology Enhanced Macular Services Study Reference Group, Talks, J., Denniston, A. K., Kelly, C. J., Malawana, J., Papoutsi, C., Teare, M. D., Keane, P. A., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/39742 · Cited by 163
Ginestra, J. C., Giannini, H. M., Schweickert, W. D., Meadows, L., Lynch, M. J., Pavan, K., Chivers, C. J., Draugelis, M., Donnelly, P. J., Fuchs, B. D., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003803 · Cited by 159 · cited 1×
“Similarly, recommendations that did not arrive in a timely or adequate manner negatively influenced trust in performance [37,44].”
McCoy, A., & Das, R. (2017) · BMJ Open Quality · doi:10.1136/bmjoq-2017-000158 · Cited by 148 · cited 3×
“Implementation teams were observed as well in the form of quality improvement teams [34], AI governance committees [40,46,63], or interdisciplinary teams of HCPs, software engineers, developers, IT specialists, and other figures [46,51,55,64].”
Romero-Brufau, S., Wyatt, K. D., Boyum, P., Mickelson, M., Moore, M., & Cognetta-Rieke, C. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.104072 · Cited by 143 · cited 4×
“Similarly, recommendations that did not arrive in a timely or adequate manner negatively influenced trust in performance [37,44].”
InfluentialChomutare, T., Tejedor, M., Svenning, T. O., Marco-Ruiz, L., Tayefi, M., Lind, K., Godtliebsen, F., Moen, A., Ismail, L., Makhlysheva, A., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192316359 · Cited by 142 · cited 6×
“On the one hand, tailoring and adaptation strategies, early end-user engagement, and appropriate framing of ML-based applications as decision-support tools might favor HCP trust in both the application’s performance and its purpose [21].”
InfluentialLee, T. C., Shah, N. U., Haack, A., & Baxter, S. L. (2020) · Informatics · doi:10.3390/informatics7030025 · Cited by 133
The Rise of the Empowered Physician in the Digital Health Era: Viewpoint
Mesko, B., & Győrffy, Z. (2019) · Journal of Medical Internet Research · doi:10.2196/12490 · Cited by 126 · cited 1×
“In the realm of digital health interventions, while there is frequent emphasis on patient engagement in the design of solutions, the empowerment of HCPs is often overlooked [82,83].”
Hassan, A. E., Ringheanu, V. M., Rabah, R. R., Preston, L., Tekle, W. G., & Qureshi, A. I. (2020) · Interventional Neuroradiology · doi:10.1177/1591019920953055 · Cited by 122 · cited 1×
“…benefits were related to the organizational dimension, in terms of optimization of the workflow resulting from the elimination of unnecessary steps [43], increased attention from end users to all cases managed by the application [45], and enhanced interactions among physicians and other HCPs [62].”
Systematic reviews of machine learning in healthcare: a literature review
Kolasa, K., Admassu, B. Y., Hołownia-Voloskova, M. E., Kędzior, K. J., Poirrier, J., & Perni, S. (2023) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2023.2279107 · Cited by 116 · cited 1×
“On a similar note, a recent review encompassing over 10,000 ML applications in health care settings corroborated the relevance of prognostic algorithms among those in use [75].”
Bridging the chasm between AI and clinical implementation
Aristidou, A., Jena, R., & Topol, E. J. (2022) · The Lancet · doi:10.1016/s0140-6736(22)00235-5 · Cited by 115 · cited 1×
“The road to implementing ML applications in patient care has several challenges, creating an inevitable chasm between ML and its clinical integration [7,8].”
The Last Mile: Where Artificial Intelligence Meets Reality
Coiera, E. (2019) · Journal of Medical Internet Research · doi:10.2196/16323 · Cited by 102 · cited 1×
“The road to implementing ML applications in patient care has several challenges, creating an inevitable chasm between ML and its clinical integration [7,8].”
Detecting Patient Deterioration Using Artificial Intelligence in a Rapid Response System
Cho, K.-J., Kwon, O., Kwon, J., Lee, Y., Park, H., Jeon, K., Kim, K., Park, J., & Oh, B. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004236 · Cited by 97 · cited 1×
“The absence of this aspect was perceived as a barrier to trust in the ML application’s performance [39,41,44,51].”
Transforming Hospital Emergency Department Workflow and Patient Care
Lee, E. K., Atallah, H., Wright, M. D., Post, E. T., Thomas, C., Wu, D., & Haley, L. L. (2015) · INFORMS Journal on Applied Analytics · doi:10.1287/inte.2014.0788 · Cited by 95 · cited 3×
“For HCPs, it was often important to maintain a sense of control over the application and not perceive it as an attempt at uncontrolled substitution and automation [32,33,56,65].”
Schwartz, J. M., George, M., Rossetti, S. C., Dykes, P. C., Minshall, S. R., Lucas, E., & Cato, K. D. (2022) · JMIR Human Factors · doi:10.2196/33960 · Cited by 88 · cited 7×
“Themes, such as data collection and quality [39,58,59], security [33], availability [36,38], and sharing [39,61], were highly described as challenges for the adoption of the application.”
InfluentialBarragán-Montero, A., Bibal, A., Dastarac, M. H., Draguet, C., Valdés, G., Nguyen, D., Willems, S., Vandewinckele, L., Holmström, M., Löfman, F., et al. (2022) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ac678a · Cited by 84 · cited 2×
“On the other hand, specific tailoring strategies should be adopted to increase the explainability of nontotally interpretable models [13].”
Berge, G. T., Granmo, O., Tveit, T. O., Munkvold, B. E., Ruthjersen, A. L., & Sharma, J. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02101-x · Cited by 83 · cited 5×
“…of reflecting and evaluating , feedback and feedback loops emerged as recurring topics, with many studies underscoring the importance for both ML developers and implementation teams to incorporate end users’ feedback on either technical issues, system design, or clinical needs [34,36,38,49,62,65].”
Influentialvan der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad075 · Cited by 72 · cited 2×
“Active involvement of HCPs and frequent communications to raise awareness have been unambiguously identified among the most common enablers of trust in previous reviews on the implementation of ML applications [21,22,74,84].”
Davis, M. A., Rao, B., Cedeno, P. A., Saha, A., & Zohrabian, V. M. (2022) · Current Problems in Diagnostic Radiology · doi:10.1067/j.cpradiol.2020.10.007 · Cited by 62 · cited 3×
“For instance, Jauk et al [49] enhanced clinical reasoning using a web application presenting relevant features from ML modeling, Davis et al [53] allowed radiologists to interact with the ML system by showing the types and locations of the abnormalities identified by the algorithm, and Henry et al…”
Making machine learning matter to clinicians: model actionability in medical decision-making
Ehrmann, D. E., Joshi, S., Goodfellow, S. D., Mazwi, M. L., & Eytan, D. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00753-7 · Cited by 60 · cited 1×
“In such cases, other contributions have emphasized the need to highlight the level of actionability of ML models, in terms of their ability to enhance medical decision-making compared to clinical judgment alone, to power trust [81].”
Jauk, S., Kramer, D., Avian, A., Berghold, A., Leodolter, W., & Schulz, S. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01727-6 · Cited by 59 · cited 6×
“…of reflecting and evaluating , feedback and feedback loops emerged as recurring topics, with many studies underscoring the importance for both ML developers and implementation teams to incorporate end users’ feedback on either technical issues, system design, or clinical needs [34,36,38,49,62,65].”
InfluentialImplementation of an Artificial Intelligence Algorithm for sepsis detection
Gonçalves, L. S., Amaro, M. L. D. M., Romero, A. D. L. M., Schamne, F. K., Fressatto, J. L., & Bezerra, C. W. (2020) · Revista Brasileira de Enfermagem · doi:10.1590/0034-7167-2018-0421 · Cited by 53 · cited 6×
“…of reflecting and evaluating , feedback and feedback loops emerged as recurring topics, with many studies underscoring the importance for both ML developers and implementation teams to incorporate end users’ feedback on either technical issues, system design, or clinical needs [34,36,38,49,62,65].”
InfluentialImproving the delivery of palliative care through predictive modeling and healthcare informatics
Murphree, D. H., Wilson, P. M., Asai, S. W., Quest, D. J., Lin, Y., Mukherjee, P., Chhugani, N., Strand, J. J., Demuth, G., Mead, D. W., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa211 · Cited by 52 · cited 3×
“The absence of this aspect was perceived as a barrier to trust in the ML application’s performance [39,41,44,51].”
Polhemus, A. M., Novák, J. M., Ferrão, J. C., Simblett, S. K., Radaelli, M., Locatelli, P., Matcham, F., Kerz, M., Weyer, J., Burke, P., et al. (2020) · JMIR mhealth and uhealth · doi:10.2196/16043 · Cited by 47 · cited 1×
“Some studies explicitly cited the theme of human-centered design as a development framework that starts with the assessment of end users’ needs and the environment in which the ML application will be used [54,64,68].”
Wijnhoven, F. (2022) · Information Systems Frontiers · doi:10.1007/s10796-021-10206-9 · Cited by 46 · cited 9×
“Regardless, feedback collection was described as an iterative activity [46,48,60,61], which also positively influenced professional buy-in [47].”
InfluentialJoshi, M., Mecklai, K., Rozenblum, R., & Samal, L. (2022) · JAMIA Open · doi:10.1093/jamiaopen/ooac022 · Cited by 46 · cited 5×
“Implementation teams were observed as well in the form of quality improvement teams [34], AI governance committees [40,46,63], or interdisciplinary teams of HCPs, software engineers, developers, IT specialists, and other figures [46,51,55,64].”
InfluentialXu, H., Li, P., Yang, Z., Liu, X., Wang, Z., Yan, W., He, M., Chu, W., She, Y., Li, Y., et al. (2020) · Journal of Medical Systems · doi:10.1007/s10916-020-01653-z · Cited by 43 · cited 4×
“Some studies noted that feedback collection extended beyond implementation, with structured feedback loop processes integrated into routine use [46,48].”
InfluentialA Framework to Guide Implementation of AI in Health Care: Protocol for a Cocreation Research Project
Nilsen, P., Svedberg, P., Neher, M., Nair, M., Larsson, I., Petersson, L., & Nygren, J. (2023) · JMIR Research Protocols · doi:10.2196/50216 · Cited by 37 · cited 1×
“Several implementation frameworks for health care technologies are on hand, but no widely recognized model addresses all the specific issues that are relevant to ML applications [15-17].”
Rushlow, D. R., Croghan, I. T., Inselman, J. W., Thacher, T. D., Friedman, P. A., Yao, X., Pellikka, P. A., Lopez-Jimenez, F., Bernard, M. E., Barry, B. A., et al. (2022) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2022.04.008 · Cited by 33 · cited 1×
“…of providing end users with access to training programs on both hard and soft skills before implementation [48,49,55,64], including computer and technical literacy linked with the complexity of the application’s functioning [57,63], and the medical domain that the application addresses [45].”
Frontoni, E., Romeo, L., Bernardini, M., Moccia, S., Migliorelli, L., Paolanti, M., Ferri, A., Misericordia, P., Mancini, A., & Zingaretti, P. (2020) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2020.3031107 · Cited by 32 · cited 1×
“Within the partnership and connections construct, building partnerships with scientific societies and professional communities was considered a facilitator for implementation, as these can act as knowledge platforms or hubs [42,47].”
Singer, S. J., Kellogg, K. C., Galper, A. B., & Viola, D. (2021) · Health Care Management Review · doi:10.1097/hmr.0000000000000324 · Cited by 32 · cited 2×
“…phase, stakeholder engagement was linked to evident benefits, such as improvement in the implementation climate [46], greater willingness to adopt the role of implementation leader [38], greater professional buy-in [51,55], and better iterative collection of information and feedback [60].”
Tricco, A. C., Hezam, A., Parker, A., Nincic, V., Harris, C., Fennelly, O., Thomas, S. M., Ghassemi, M., McGowan, J., Paprica, P. A., et al. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-065845 · Cited by 32 · cited 3×
“Active involvement of HCPs and frequent communications to raise awareness have been unambiguously identified among the most common enablers of trust in previous reviews on the implementation of ML applications [21,22,74,84].”
Mittermaier, M., Raza, M., & Kvedar, J. C. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00889-6 · Cited by 32 · cited 1×
“ML is expected to serve primarily as a decision support tool to enhance human work rather than replace it [4], thereby providing health care professionals (HCPs) with improved predictions and rendering their decision-making process more accurate [5].”
Barriers to Implementing an Artificial Intelligence Model for Unplanned Readmissions
Baxter, S. L., Bass, J. S., & Sitapati, A. M. (2020) · ACI Open · doi:10.1055/s-0040-1716748 · Cited by 30 · cited 3×
“On the one hand, faster decision-making could be considered a potential advantage [47], and on the other hand, human-machine interaction could lead to a loss of efficiency compared to human intervention only [40,56].”
Hinson, J. S., Klein, E., Smith, A., Toerper, M., Dungarani, T., Hager, D., Hill, P., Kelen, G., Niforatos, J. D., Stephens, R. S., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00646-1 · Cited by 30 · cited 3×
“…emphasized the importance of providing end users with access to training programs on both hard and soft skills before implementation [48,49,55,64], including computer and technical literacy linked with the complexity of the application’s functioning [57,63], and the medical domain that the…”
Bhattacharya, S., Sharma, N., & Singh, A. (2019) · Journal of Family Medicine and Primary Care · doi:10.4103/jfmpc.jfmpc_391_18 · Cited by 28 · cited 1×
“Cultural aspects, innovation attitudes, and public expectations could influence the acceptability of ML [35,39,50].”
Zhai, Y., Yu, Z., Zhang, Q., & Zhang, Y. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104841 · Cited by 28
From compute to care: Lessons learned from deploying an early warning system into clinical practice
Pou-Prom, C., Murray, J., Kuzulugil, S. S., Mamdani, M., & Verma, A. A. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.932123 · Cited by 26 · cited 4×
“Although it may theoretically involve figures that emerge from bottom-up processes, all works referring to this role mentioned a top-down identification [45,47,62,63].”
InfluentialNilsén, P., Reed, J. E., Nair, M., Savage, C., Macrae, C., Barlow, J., Svedberg, P., Larsson, I., Lundgren, L. E., & Nygren, J. (2022) · Frontiers in Health Services · doi:10.3389/frhs.2022.961475 · Cited by 25 · cited 1×
“…further divided trust in technology into 3 dimensions: trust in performance, focusing on the accuracy and consistency of the output; trust in the process, concerning the understanding of the reasoning behind a given output; and trust in the purpose of the innovation to be implemented [16,44].”
Morales, H. M. P., Guedes, M. H., Silva, J. S., & Massuda, A. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.648585 · Cited by 24 · cited 5×
“Moreover, forging public-private partnerships was deemed a useful step for implementation, to leverage expertise not always available within public health care organizations [50,61].”
InfluentialAI detection of mild COVID-19 pneumonia from chest CT scans
Yao, J.-C., Wang, T., Hou, G.-H., Ou, D., Li, W., Zhu, Q.-D., Chen, W.-C., Yang, C., Wang, L., Wang, L.-P., et al. (2021) · European Radiology · doi:10.1007/s00330-021-07797-x · Cited by 20 · cited 1×
“The former was often linked to minimizing manual interventions, such as data input [49,52], and was also associated with dimensions of trust in the applications, such as trust in the process and the cognitive burden for HCPs, in the form of fatigue from overalerting [33,62], which could be a…”
Cruz Noa, P., Lea, C., García, M. J., Bernardino, P., & Ignacio, A. (2019) · Studies in Health Technology and Informatics · doi:10.3233/shti190285 · Cited by 19 · cited 3×
“…the expected benefits, positively affecting trust in ML-based innovations [33,38,45-47, 54] and In terms of adapting , the first theme involved the need for collected feedback to be effectively incorporated into the application, adapting systems to the local context of implementation [36,58].”
Sibbald, M., Abdulla, B., Keuhl, A., Norman, G. R., Monteiro, S., & Sherbino, J. (2022) · JMIR Human Factors · doi:10.2196/39234 · Cited by 11 · cited 7×
“Themes, such as data collection and quality [39,58,59], security [33], availability [36,38], and sharing [39,61], were highly described as challenges for the adoption of the application.”
InfluentialArdito, V., Cappellaro, G., Compagni, A., Petracca, F., & Preti, L. M. (2023) · JMIR Research Protocols · doi:10.2196/47971 · Cited by 10 · cited 3×
“Data items were extracted based on established classifications or schemes, when applicable [24].”
InfluentialSelf-citation
Research works printed but not linked: 1
- Artificial intelligence in healthcare: Applications, risks, and ethical and societal impactsEuropean Parliament2024-11-04https://data.europa.eu/doi/10.2861/568473 · doi:10.2861/568473
Cited by
Ardito, V., Cappellaro, G., Compagni, A., Petracca, F., & Preti, L. M. (2025) · Digital Health · doi:10.1177/20552076251355680 · Cited by 16
“Survey development was informed by a prior literature review by Preti et al., 15,16 which examined the characteristics of ML applications and their implementation processes in clinical practice.”
InfluentialSelf-citationFarahani, S., Hejazi, M., Tabassum, M., Ieva, A. D., Mahdavifar, N., & Liu, S. (2025) · European Radiology · doi:10.1007/s00330-025-11898-2 · Cited by 26
Nasef, D., Nasef, D., Sher, M., & Toma, M. (2025) · Algorithms · doi:10.3390/a18060343 · Cited by 13
Farahani, S., Hejazi, M., Moradizeyveh, S., Ieva, A. D., Fatemizadeh, E., & Liu, S. (2025) · Diagnostics · doi:10.3390/diagnostics15070797 · Cited by 13
Transforming towards AI-augmented Healthcare: Experiences of physicians in Sweden
Ismail, M., Barth, H., Holmén, M., Petersson, L., & Irgang, L. (2025) · Technovation · doi:10.1016/j.technovation.2025.103333 · Cited by 16
Metadata
- Authors
- Preti, L. M., Ardito, V., Compagni, A., Petracca, F., & Cappellaro, G.
- Affiliations
- Bocconi University (IT)
- Corresponding author
- Cappellaro, Giulia
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-11-25
- Volume
- 26
- Pages
- e55897
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11629039
- OpenAlex
- W4403095177
- Cited by
- 62
- References
- 82
- Citations by year
- 2026: 42 · 2025: 20
- FWCI
- 1.40
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Archivio istituzionale della Ricerca - Bocconi (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- European Commission
- NextGenerationEU
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- machine learning implementation, artificial intelligence in healthcare, clinical implementation barriers, Consolidated Framework for Implementation Research, health care organization, clinical workflow, organizational culture, stakeholder engagement, implementation leadership, IT infrastructure, implementation strategies, hospitals, implementation success factors, implementation failure factors, implementation readiness, implementation evaluation, implementation monitoring, implementation sustainability
- MeSH terms
- Machine Learning, Information Technology, Digital Health, Artificial Intelligence, Delivery of Health Care, Hospitals, Humans, Practice Patterns, Physicians', Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.