CoChat

Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review

Open access

The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index; 14 of the 22 sources its claims cite were checked for support.

Abstract

BACKGROUND: The number of artificial intelligence (AI) studies in medicine has exponentially increased recently. However, there is no clear quantification of the clinical benefits of implementing AI-assisted tools in patient care. OBJECTIVE: This study aims to systematically review all published randomized controlled trials (RCTs) of AI-assisted tools to characterize their performance in clinical practice. METHODS: CINAHL, Cochrane Central, Embase, MEDLINE, and PubMed were searched to identify relevant RCTs published up to July 2021 and comparing the performance of AI-assisted tools with conventional clinical management without AI assistance. We evaluated the primary end points of each study to determine their clinical relevance. This systematic review was conducted following the updated PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines. RESULTS: Among the 11,839 articles retrieved, only 39 (0.33%) RCTs were included. These RCTs were conducted in an approximately equal distribution from North America, Europe, and Asia. AI-assisted tools were implemented in 13 different clinical specialties. Most RCTs were published in the field of gastroenterology, with 15 studies on AI-assisted endoscopy. Most RCTs studied biosignal-based AI-assisted tools, and a minority of RCTs studied AI-assisted tools drawn from clinical data. In 77% (30/39) of the RCTs, AI-assisted interventions outperformed usual clinical care, and clinically relevant outcomes improved with AI-assisted intervention in 70% (21/30) of the studies. Small sample size and single-center design limited the generalizability of these studies. CONCLUSIONS: There is growing evidence supporting the implementation of AI-assisted tools in daily clinical practice; however, the number of available RCTs is limited and heterogeneous. More RCTs of AI-assisted tools integrated into clinical practice are needed to advance the role of AI in medicine. TRIAL REGISTRATION: PROSPERO CRD42021286539; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=286539.

Overview

CoChat Summary

TL;DR

This systematic review identified only 39 RCTs (0.33% of 11,839 retrieved articles) of AI-assisted tools in clinical practice up to July 2021. Most (77%, 30/39) reported AI-assisted interventions outperforming usual care, but the evidence base was limited and heterogeneous.

Key findings

  • Among 11,839 articles retrieved, only 39 (0.33%) RCTs were included; 77% (30/39) reported positive outcomes for AI-assisted interventions, and clinically relevant outcomes improved in 70% (21/30) of those positive studies.
  • Gastroenterology dominated the field: 16/39 (41%) RCTs were related to gastroenterology, with 15 studies on AI-assisted endoscopy; biosignal-based AI tools were more common than clinical data–based tools, and endoscopic images were the most common biosignal (15/26, 58%).
  • Risk of bias was low in 20% (8/39), some concerns in 31% (12/39), and high in 49% (19/39); missing outcome data and outcome measurement were the most common risk factors.
  • Common study limitations were single-center design (22/39, 56%) and small sample size (n<1000; 33/39, 85%); 15/39 (38%) were open-label, and 7 studies were underpowered.

Methods

  • Systematic review following PRISMA 2020 guidelines; searched CINAHL, Cochrane Central, Embase, MEDLINE, and PubMed from inception to July 14, 2021, for RCTs comparing AI-assisted tools with conventional clinical management without AI assistance; registered on PROSPERO (CRD42021286539).
  • Two authors independently screened articles and extracted data, with disagreements resolved by consensus or senior author consultation; risk of bias was assessed using the Cochrane risk-of-bias tool for randomized trials (RoB 2), and primary end points were evaluated for clinical relevance.

Limitations

  • Paper-stated: small sample size and single-center design limited generalizability; the number of available RCTs was limited and heterogeneous; among included studies, common author-listed limitations were single-center design (22/39, 56%) and small sample size (n<1000; 33/39, 85%), with open-label design (15/39, 38%) and 7 underpowered studies; 49% had high risk of bias.
  • Evident from the text: the search ended July 2021, so later RCTs are not captured; 18% (7/39) were conference abstracts only; no meta-analysis or pooled effect estimates were reported.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/37188

Cites

64 of 64 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram.
Figure 1 PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram.CC BY · PubMed Central
Figure 2 Number of randomized controlled trials of artificial intelligence–assisted medicine per year.
Figure 2 Number of randomized controlled trials of artificial intelligence–assisted medicine per year.CC BY · PubMed Central
Figure 3 Distribution of original of publications and specialty.
Figure 3 Distribution of original of publications and specialty.CC BY · PubMed Central

Verification

Claims the paper makes

11 Valid · 1 Not valid · 5 Not checked · 99 Own results and interpretations

59 more claims

Create a free account to see all 116 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 38
  • It may state two separate claims as one. 18

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Research works printed but not linked: 1

  1. Pouska, J, 2019, Intensive Care Med Exp, 7, 001525

Cited by

See all 120

Metadata

Authors
Lam, T. Y. T., Cheung, M. F. K., Munro, Y. L., Lim, K. M., Shung, D., & Sung, J. J. Y.
Affiliations
  • Chinese University of Hong Kong (HK)
  • Nanyang Technological University (SG)
  • Yale University (US)
Corresponding author
Sung, Joseph J Y
Venue
Journal of Medical Internet Research
Published
2022-08-25
Volume
24
Issue
8
Pages
e37188
ISSN
1438-8871 (ISSN-L)
OpenAlex
W4288467884
Cited by
120
References
64
Citations by year
2026: 33 · 2025: 46 · 2024: 29 · 2023: 12
FWCI
1.49
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • DR-NTU (Nanyang Technological University) (submittedVersion, cc-by)
Funders
  • Yale University
  • National Institutes of Health (UL1TR001863)
  • Raymond and Beverly Sackler Institute for Biological, Physical and Engineering Sciences, Yale University
  • National Center for Advancing Translational Sciences
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
artificial intelligence, AI-assisted tools, clinical practice, randomized controlled trials, clinical outcomes, endoscopy, gastroenterology
MeSH terms
Artificial Intelligence, Europe, Humans, North America, Randomized Controlled Trials as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.