CoChat

Implementation of Machine Learning Applications in Health Care Organizations: Systematic Review of Empirical Studies

Open access
Preti, Luigi M, Ardito, Vittoria, Compagni, Amelia, Petracca, Francesco, Cappellaro, Giulia · 2024 · Journal of Medical Internet Research

Published version of a preprint

The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.

Abstract

Background: There is a growing enthusiasm for machine learning (ML) among academics and health care practitioners. Despite the transformative potential of ML-based applications for patient care, their uptake and implementation in health care organizations are sporadic. Numerous challenges currently impede or delay the widespread implementation of ML in clinical practice, and limited knowledge is available regarding how these challenges have been addressed. Objective: This work aimed to (1) examine the characteristics of ML-based applications and the implementation process in clinical practice, using the Consolidated Framework for Implementation Research (CFIR) for theoretical guidance and (2) synthesize the strategies adopted by health care organizations to foster successful implementation of ML. Methods: A systematic literature review was conducted based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. The search was conducted in PubMed, Scopus, and Web of Science over a 10-year period (2013-2023). The search strategy was built around 4 blocks of keywords (artificial intelligence, implementation, health care, and study type). Only empirical studies documenting the implementation of ML applications in clinical settings were considered. The implementation process was investigated using a thematic analysis and coding procedure. Results: Thirty-four studies were selected for data synthesis. Selected papers were relatively recent, with only 9% (3/34) of records published before 2019. ML-based applications were implemented mostly within hospitals (29/34, 85%). In terms of clinical workflow, ML-based applications supported mostly prognosis (20/34, 59%) and diagnosis (10/34, 29%). The implementation efforts were analyzed using CFIR domains. As for the inner setting domain, access to knowledge and information (12/34, 35%), information technology infrastructure (11/34, 32%), and organizational culture (9/34, 26%) were among the most observed dimensions influencing the success of implementation. As for the ML innovation itself, factors deemed relevant were its design (15/34, 44%), the relative advantage with respect to existing clinical practice (14/34, 41%), and perceived complexity (14/34, 41%). As for the other domains (ie, processes, roles, and outer setting), stakeholder engagement (12/34, 35%), reflecting and evaluating practices (11/34, 32%), and the presence of implementation leaders (9/34, 26%) were the main factors identified as important. Conclusions: This review sheds some light on the factors that are relevant and that should be accounted for in the implementation process of ML-based applications in health care. While the relevance of ML-specific dimensions, like trust, emerges clearly across several implementation domains, the evidence from this review highlighted that relevant implementation factors are not necessarily specific for ML but rather transversal for digital health technologies. More research is needed to further clarify the factors that are relevant to implementing ML-based applications at the organizational level and to support their uptake within health care organizations. Trial Registration: PROSPERO 403873; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=403873 International Registered Report Identifier (IRRID): RR2-10.2196/47971

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/55897

Cites

82 of 82 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram.
Figure 1 PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram.CC BY · PubMed Central
Figure 2 Relative importance of Consolidated Framework for Implementation Research (CFIR) constructs. The percentages represent the proportions of papers in which each construct and domain appears out of the 34 included in the review. The size of the bubbles corresponds to the frequency of occurrence of each construct.
Figure 2 Relative importance of Consolidated Framework for Implementation Research (CFIR) constructs. The percentages represent the proportions of papers in which each construct and domain appears out of the 34 included in the review. The size of the bubbles corresponds to the frequency of occurrence of each construct.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 82 references

Research works printed but not linked: 1

  1. Artificial intelligence in healthcare: Applications, risks, and ethical and societal impactsEuropean Parliament2024-11-04https://data.europa.eu/doi/10.2861/568473 · doi:10.2861/568473

Cited by

See all 62

Metadata

Authors
Preti, L. M., Ardito, V., Compagni, A., Petracca, F., & Cappellaro, G.
Affiliations
Bocconi University (IT)
Corresponding author
Cappellaro, Giulia
Venue
Journal of Medical Internet Research
Published
2024-11-25
Volume
26
Pages
e55897
ISSN
1438-8871 (ISSN-L)
OpenAlex
W4403095177
Cited by
62
References
82
Citations by year
2026: 42 · 2025: 20
FWCI
1.40
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Archivio istituzionale della Ricerca - Bocconi (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • European Commission
  • NextGenerationEU
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
machine learning implementation, artificial intelligence in healthcare, clinical implementation barriers, Consolidated Framework for Implementation Research, health care organization, clinical workflow, organizational culture, stakeholder engagement, implementation leadership, IT infrastructure, implementation strategies, hospitals, implementation success factors, implementation failure factors, implementation readiness, implementation evaluation, implementation monitoring, implementation sustainability
MeSH terms
Machine Learning, Information Technology, Digital Health, Artificial Intelligence, Delivery of Health Care, Hospitals, Humans, Practice Patterns, Physicians', Workflow

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.