Implementation of Machine Learning Applications in Health Care Organizations: Systematic Review of Empirical Studies
Open accessPublished version of a preprint
The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.
Abstract
Background: There is a growing enthusiasm for machine learning (ML) among academics and health care practitioners. Despite the transformative potential of ML-based applications for patient care, their uptake and implementation in health care organizations are sporadic. Numerous challenges currently impede or delay the widespread implementation of ML in clinical practice, and limited knowledge is available regarding how these challenges have been addressed. Objective: This work aimed to (1) examine the characteristics of ML-based applications and the implementation process in clinical practice, using the Consolidated Framework for Implementation Research (CFIR) for theoretical guidance and (2) synthesize the strategies adopted by health care organizations to foster successful implementation of ML. Methods: A systematic literature review was conducted based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. The search was conducted in PubMed, Scopus, and Web of Science over a 10-year period (2013-2023). The search strategy was built around 4 blocks of keywords (artificial intelligence, implementation, health care, and study type). Only empirical studies documenting the implementation of ML applications in clinical settings were considered. The implementation process was investigated using a thematic analysis and coding procedure. Results: Thirty-four studies were selected for data synthesis. Selected papers were relatively recent, with only 9% (3/34) of records published before 2019. ML-based applications were implemented mostly within hospitals (29/34, 85%). In terms of clinical workflow, ML-based applications supported mostly prognosis (20/34, 59%) and diagnosis (10/34, 29%). The implementation efforts were analyzed using CFIR domains. As for the inner setting domain, access to knowledge and information (12/34, 35%), information technology infrastructure (11/34, 32%), and organizational culture (9/34, 26%) were among the most observed dimensions influencing the success of implementation. As for the ML innovation itself, factors deemed relevant were its design (15/34, 44%), the relative advantage with respect to existing clinical practice (14/34, 41%), and perceived complexity (14/34, 41%). As for the other domains (ie, processes, roles, and outer setting), stakeholder engagement (12/34, 35%), reflecting and evaluating practices (11/34, 32%), and the presence of implementation leaders (9/34, 26%) were the main factors identified as important. Conclusions: This review sheds some light on the factors that are relevant and that should be accounted for in the implementation process of ML-based applications in health care. While the relevance of ML-specific dimensions, like trust, emerges clearly across several implementation domains, the evidence from this review highlighted that relevant implementation factors are not necessarily specific for ML but rather transversal for digital health technologies. More research is needed to further clarify the factors that are relevant to implementing ML-based applications at the organizational level and to support their uptake within health care organizations. Trial Registration: PROSPERO 403873; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=403873 International Registered Report Identifier (IRRID): RR2-10.2196/47971
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 1×
“The recently updated version of the Consolidated Framework for Implementation Research (CFIR), a commonly used model to assess factors influencing implementation and to explain barriers and facilitators to implementation effectiveness [27,28], was used as a criterion for inclusion.”
Valente, T. W. (2003) · Genetics in Medicine · doi:10.1097/01.gim.0000061743.67794.c4 · Cited by 13,723
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Methods for the thematic synthesis of qualitative research in systematic reviews
Thomas, J., & Harden, A. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-45 · Cited by 9,312
Rudd, K. E., Johnson, S. C., Agesa, K. M., Shackelford, K. A., Tsoi, D., Kievlan, D. R., Colombara, D. V., Ikuta, K. S., Kissoon, N., Finfer, S., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(19)32989-7 · Cited by 7,971 · cited 1×
“The only condition for which we observed a conspicuous number of studies was sepsis, a dysfunction accounting for around 20% of deaths worldwide [72], for which ML-based applications are proliferating [73], although no definitive causal link with reduced mortality has been demonstrated to date [74].”
Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations
GREENHALGH, T., ROBERT, G., MACFARLANE, F., BATE, P., & KYRIAKIDOU, O. (2004) · The Milbank Quarterly · doi:10.1111/j.0887-378x.2004.00325.x · Cited by 7,448 · cited 1×
“We adopted the definition of implementation as an “active and planned effort to mainstream innovation within an organization” [26], while health care organizations encompassed all entities delivering health services, including hospitals, outpatient centers, primary care facilities, and public…”
The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers
Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., Nicolau, B., O’Cathain, A., et al. (2018) · Education for Information · doi:10.3233/efi-180221 · Cited by 4,737 · cited 1×
“Critical appraisal of the studies selected for data synthesis was performed using the Mixed Method Appraisal Tool (MMAT [29]), which has been designed specifically for systematic reviews that include heterogeneous studies, as it allows to assess the methodological quality of 5 types of study…”
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
Research works printed but not linked: 1
- Artificial intelligence in healthcare: Applications, risks, and ethical and societal impactsEuropean Parliament2024-11-04https://data.europa.eu/doi/10.2861/568473 · doi:10.2861/568473
Cited by
Ardito, V., Cappellaro, G., Compagni, A., Petracca, F., & Preti, L. M. (2025) · Digital Health · doi:10.1177/20552076251355680 · Cited by 16
“Survey development was informed by a prior literature review by Preti et al., 15,16 which examined the characteristics of ML applications and their implementation processes in clinical practice.”
InfluentialSelf-citationFarahani, S., Hejazi, M., Tabassum, M., Ieva, A. D., Mahdavifar, N., & Liu, S. (2025) · European Radiology · doi:10.1007/s00330-025-11898-2 · Cited by 26
Nasef, D., Nasef, D., Sher, M., & Toma, M. (2025) · Algorithms · doi:10.3390/a18060343 · Cited by 13
Farahani, S., Hejazi, M., Moradizeyveh, S., Ieva, A. D., Fatemizadeh, E., & Liu, S. (2025) · Diagnostics · doi:10.3390/diagnostics15070797 · Cited by 13
Transforming towards AI-augmented Healthcare: Experiences of physicians in Sweden
Ismail, M., Barth, H., Holmén, M., Petersson, L., & Irgang, L. (2025) · Technovation · doi:10.1016/j.technovation.2025.103333 · Cited by 16
Metadata
- Authors
- Preti, L. M., Ardito, V., Compagni, A., Petracca, F., & Cappellaro, G.
- Affiliations
- Bocconi University (IT)
- Corresponding author
- Cappellaro, Giulia
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-11-25
- Volume
- 26
- Pages
- e55897
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11629039
- OpenAlex
- W4403095177
- Cited by
- 62
- References
- 82
- Citations by year
- 2026: 42 · 2025: 20
- FWCI
- 1.40
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Archivio istituzionale della Ricerca - Bocconi (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- European Commission
- NextGenerationEU
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- machine learning implementation, artificial intelligence in healthcare, clinical implementation barriers, Consolidated Framework for Implementation Research, health care organization, clinical workflow, organizational culture, stakeholder engagement, implementation leadership, IT infrastructure, implementation strategies, hospitals, implementation success factors, implementation failure factors, implementation readiness, implementation evaluation, implementation monitoring, implementation sustainability
- MeSH terms
- Machine Learning, Information Technology, Digital Health, Artificial Intelligence, Delivery of Health Care, Hospitals, Humans, Practice Patterns, Physicians', Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.