Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review
Open accessThe DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index; 14 of the 22 sources its claims cite were checked for support.
Abstract
BACKGROUND: The number of artificial intelligence (AI) studies in medicine has exponentially increased recently. However, there is no clear quantification of the clinical benefits of implementing AI-assisted tools in patient care. OBJECTIVE: This study aims to systematically review all published randomized controlled trials (RCTs) of AI-assisted tools to characterize their performance in clinical practice. METHODS: CINAHL, Cochrane Central, Embase, MEDLINE, and PubMed were searched to identify relevant RCTs published up to July 2021 and comparing the performance of AI-assisted tools with conventional clinical management without AI assistance. We evaluated the primary end points of each study to determine their clinical relevance. This systematic review was conducted following the updated PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines. RESULTS: Among the 11,839 articles retrieved, only 39 (0.33%) RCTs were included. These RCTs were conducted in an approximately equal distribution from North America, Europe, and Asia. AI-assisted tools were implemented in 13 different clinical specialties. Most RCTs were published in the field of gastroenterology, with 15 studies on AI-assisted endoscopy. Most RCTs studied biosignal-based AI-assisted tools, and a minority of RCTs studied AI-assisted tools drawn from clinical data. In 77% (30/39) of the RCTs, AI-assisted interventions outperformed usual clinical care, and clinically relevant outcomes improved with AI-assisted intervention in 70% (21/30) of the studies. Small sample size and single-center design limited the generalizability of these studies. CONCLUSIONS: There is growing evidence supporting the implementation of AI-assisted tools in daily clinical practice; however, the number of available RCTs is limited and heterogeneous. More RCTs of AI-assisted tools integrated into clinical practice are needed to advance the role of AI in medicine. TRIAL REGISTRATION: PROSPERO CRD42021286539; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=286539.
Overview
CoChat Summary
TL;DR
This systematic review identified only 39 RCTs (0.33% of 11,839 retrieved articles) of AI-assisted tools in clinical practice up to July 2021. Most (77%, 30/39) reported AI-assisted interventions outperforming usual care, but the evidence base was limited and heterogeneous.
Key findings
- Among 11,839 articles retrieved, only 39 (0.33%) RCTs were included; 77% (30/39) reported positive outcomes for AI-assisted interventions, and clinically relevant outcomes improved in 70% (21/30) of those positive studies.
- Gastroenterology dominated the field: 16/39 (41%) RCTs were related to gastroenterology, with 15 studies on AI-assisted endoscopy; biosignal-based AI tools were more common than clinical data–based tools, and endoscopic images were the most common biosignal (15/26, 58%).
- Risk of bias was low in 20% (8/39), some concerns in 31% (12/39), and high in 49% (19/39); missing outcome data and outcome measurement were the most common risk factors.
- Common study limitations were single-center design (22/39, 56%) and small sample size (n<1000; 33/39, 85%); 15/39 (38%) were open-label, and 7 studies were underpowered.
Methods
- Systematic review following PRISMA 2020 guidelines; searched CINAHL, Cochrane Central, Embase, MEDLINE, and PubMed from inception to July 14, 2021, for RCTs comparing AI-assisted tools with conventional clinical management without AI assistance; registered on PROSPERO (CRD42021286539).
- Two authors independently screened articles and extracted data, with disagreements resolved by consensus or senior author consultation; risk of bias was assessed using the Cochrane risk-of-bias tool for randomized trials (RoB 2), and primary end points were evaluated for clinical relevance.
Limitations
- Paper-stated: small sample size and single-center design limited generalizability; the number of available RCTs was limited and heterogeneous; among included studies, common author-listed limitations were single-center design (22/39, 56%) and small sample size (n<1000; 33/39, 85%), with open-label design (15/39, 38%) and 7 underpowered studies; 49% had high risk of bias.
- Evident from the text: the search ended July 2021, so later RCTs are not captured; 18% (7/39) were conference abstracts only; no meta-analysis or pooled effect estimates were reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
11 Valid · 1 Not valid · 5 Not checked · 99 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“The adenoma detection algorithm appears to be easier for cross-compatibility because of the distinct difference in appearance between adenoma (or polyp) and normal mucosa [56].”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Discussion > Principal Findings
What the sources say
- Supportedcites [56] source
The section “Pitfall No. 4: Cross-compatibility and external validation” says colon polyp detection appears easier to make cross-compatible because the visual differences between a polyp and normal colonic mucosa are distinct. This supports the citing sentence’s paraphrase. (LLM, conf=0.94)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Another obstacle to the implementation of AI or ML in daily clinical practice is the regulation of these technologies [8].”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Discussion > Conclusions and Recommendations
What the sources say
- Supportedcites [8] source
The Introduction identifies regulation as a crucial element shaping obstacles to implementing AI/ML in daily clinical practice. The Discussion likewise attributes challenges in adopting these technologies in medical practice partly to regulatory frameworks. (LLM, conf=0.95)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there have been well-designed studies where more “complex” ML approaches have outperformed regression-derived clinical risk scores on external validation [7,59,60].”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Discussion > Principal Findings
What the sources say
- Supportedcites [7] source
The Methods (Study Design) identify the final model as XGBoost, a gradient-boosted decision-tree algorithm. The Results (Performance of the ML Model, External Validation) report that it outperformed all three clinical risk scores on the composite endpoint. The Strengths section describes a large, prospective, multicenter cohort with external validation. (LLM, conf=0.92)
- Supportedcites [59] source
The Results > External Validation section reports that the support-vector-machine risk calculator, trained on MESA and tested on the external FLEMENGHO cohort, outperformed the ACC/AHA calculator in sensitivity, specificity, and accuracy. The Methods > External Validation section describes the separate validation cohort. Page numbers are not available in our copy. (LLM, conf=0.93)
- Partly supportedcites [60] sourceExperimental — accuracy not yet measured
The Abstract reports validation of a naïve Bayes sepsis model, and Overall model performance reports that it identified more additional cases than the triage-nurse process. That process was not a regression-derived clinical risk score. Practical observations during model development also identifies single-center validation as a limitation, so this paper does not establish the claimed outperformance of regression-derived scores on external validation. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most RCTs were published in the field of gastroenterology, with 15 studies on AI-assisted endoscopy.”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most RCTs were published in the field of gastroenterology, with 15 studies on AI-assisted endoscopy.”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
59 more claims
Create a free account to see all 116 claims and every source verdict.
- Probably not a substantial claim. 38
- It may state two separate claims as one. 18
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“PPx—AI: 17% (54/224) versus control: 8% (26/318)”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Results > Study Outcomes
Reported 17 · recomputed 24.107143
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“A total of 147 full manuscripts were individually assessed, of which 34 (23.1%) met the inclusion criteria. In addition, 4 more articles were identified by examining the references of the listed articles and manual searches (Figure 1). A total of 39 articles were included in this systematic review [6,17-54] as listed in Table 1.”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Results > Overview
Reported 39 · recomputed 38
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Results > Study Outcomes
“ENDOANGEL was an effective and robust system to improve the quality of EGD and has the potential to detect electrocardiogram in real time”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Results > Study End Points
“To verify the effectiveness of ENDOANGEL in improving endoscopy quality and pretest its performance in detecting EGCae in a multicenter randomized controlled trial”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Results > Study Outcomes
“El Solh et al [17], 2019”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Results > Overview
“El Solh et al [17], 2009”
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 34,787 · cited 1×
“Risk of bias was assessed using the Cochrane risk-of-bias tool for randomized trials [16].”
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/v7gm2 · Cited by 5,177
A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955
McCarthy, J., Minsky, M., Rochester, N., et al. (2006) · The AI Magazine · doi:10.1609/aimag.v27i4.1904 · Cited by 2,029 · cited 1×
“Artificial intelligence (AI) was first described in the 1950s as a theory of human intelligence being exhibited by machines, including but not limited to learning, reasoning, and problem-solving [1].”
Benjamens, S., Dhunnoo, P., & Meskó, B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00324-0 · Cited by 1,215
Wang, P., Berzin, T. M., Glissen Brown, J. R., Bharadwaj, S., Becq, A., Xiao, X., Liu, P., Li, L., Song, Y., Zhang, D., et al. (2019) · Gut · doi:10.1136/gutjnl-2018-317500 · Cited by 865 · cited 1×
“Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study Wang et al [28], 2019”
Trends in Chronic Kidney Disease in China
Zhang, L., Long, J., Jiang, W., Shi, Y., He, X., Zhou, Z., Li, Y., Yeung, R. O., Wang, J., Matsushita, K., et al. (2016) · New England Journal of Medicine · doi:10.1056/nejmc1602469 · Cited by 765 · cited 1×
“Another obstacle to the implementation of AI or ML in daily clinical practice is the regulation of these technologies [8].”
Brown, J. M., Campbell, J. P., Beers, A., Chang, K., Ostmo, S., Chan, R. V. P., Dy, J., Erdogmus, D., Ioannidis, S., Kalpathy-Cramer, J., et al. (2018) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2018.1934 · Cited by 657
Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial
Repici, A., Badalamenti, M., Maselli, R., Correale, L., Radaelli, F., Rondonotti, E., Ferrara, E., Spadaccini, M., Alkandari, A., Fugazza, A., et al. (2020) · Gastroenterology · doi:10.1053/j.gastro.2020.04.062 · Cited by 646 · cited 2×
“Original article Italy Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial Repici et al [38], 2020”
Wijnberge, M., Geerts, B. F., Hol, L., Lemmers, N., Mulder, M. P., Berge, P., Schenk, J., Terwindt, L. E., Hollmann, M. W., Vlaar, A. P., et al. (2020) · JAMA · doi:10.1001/jama.2020.0592 · Cited by 586 · cited 2×
“…care on depth and duration of intraoperative hypotension during elective noncardiac surgery: the HYPE randomized clinical trial Wijnberge et al [42], 2020 Neurology Abstract Israel Artificial intelligence-powered non-invasive and frequency-tuned electromagnetic field therapy improves upper…”
Research works printed but not linked: 1
- Pouska, J, 2019, Intensive Care Med Exp, 7, 001525
Cited by
Han, R., Acosta, J. N., Shakeri, Z., Ioannidis, J. P. A., Topol, E. J., & Rajpurkar, P. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00047-5 · Cited by 336
Deisenhofer, I., Albenque, J.-P., Busch, S., Gitenay, E., Mountantonakis, S., Roux, A., Horvilleur, J., Bakouboula, B., Oza, S. R., Abbey, S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03517-w · Cited by 162
El Arab, R. A., Abu-Mahfouz, M. S., Abuadas, F. H., Alzghoul, H., Almari, M., Ghannam, A., & Seweid, M. M. (2025) · Healthcare · doi:10.3390/healthcare13070701 · Cited by 108
Artificial intelligence in digital pathology — time for a reality check
Aggarwal, A., Bharadwaj, S., Corredor, G., Pathak, T. B., Badve, S., & Madabhushi, A. (2025) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-025-00991-6 · Cited by 96
“Despite the increasing efforts to incorporate AI into digital pathology, especially within prospective clinical trials, these initiatives remain limited in both scope and number 103 .”
Singareddy, S., SN, V. P., Jaramillo, A. P., Yasir, M., Iyer, N., Hussein, S., & Nath, T. S. (2023) · Cureus · doi:10.7759/cureus.46066 · Cited by 57
Metadata
- Authors
- Lam, T. Y. T., Cheung, M. F. K., Munro, Y. L., Lim, K. M., Shung, D., & Sung, J. J. Y.
- Affiliations
- Chinese University of Hong Kong (HK)
- Nanyang Technological University (SG)
- Yale University (US)
- Corresponding author
- Sung, Joseph J Y
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2022-08-25
- Volume
- 24
- Issue
- 8
- Pages
- e37188
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC9459941
- OpenAlex
- W4288467884
- Cited by
- 120
- References
- 64
- Citations by year
- 2026: 33 · 2025: 46 · 2024: 29 · 2023: 12
- FWCI
- 1.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion, cc-by)
- Funders
- Yale University
- National Institutes of Health (UL1TR001863)
- Raymond and Beverly Sackler Institute for Biological, Physical and Engineering Sciences, Yale University
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- artificial intelligence, AI-assisted tools, clinical practice, randomized controlled trials, clinical outcomes, endoscopy, gastroenterology
- MeSH terms
- Artificial Intelligence, Europe, Humans, North America, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.