Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review
Open accessThe DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index; 14 of the 22 sources its claims cite were checked for support.
Abstract
BACKGROUND: The number of artificial intelligence (AI) studies in medicine has exponentially increased recently. However, there is no clear quantification of the clinical benefits of implementing AI-assisted tools in patient care. OBJECTIVE: This study aims to systematically review all published randomized controlled trials (RCTs) of AI-assisted tools to characterize their performance in clinical practice. METHODS: CINAHL, Cochrane Central, Embase, MEDLINE, and PubMed were searched to identify relevant RCTs published up to July 2021 and comparing the performance of AI-assisted tools with conventional clinical management without AI assistance. We evaluated the primary end points of each study to determine their clinical relevance. This systematic review was conducted following the updated PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines. RESULTS: Among the 11,839 articles retrieved, only 39 (0.33%) RCTs were included. These RCTs were conducted in an approximately equal distribution from North America, Europe, and Asia. AI-assisted tools were implemented in 13 different clinical specialties. Most RCTs were published in the field of gastroenterology, with 15 studies on AI-assisted endoscopy. Most RCTs studied biosignal-based AI-assisted tools, and a minority of RCTs studied AI-assisted tools drawn from clinical data. In 77% (30/39) of the RCTs, AI-assisted interventions outperformed usual clinical care, and clinically relevant outcomes improved with AI-assisted intervention in 70% (21/30) of the studies. Small sample size and single-center design limited the generalizability of these studies. CONCLUSIONS: There is growing evidence supporting the implementation of AI-assisted tools in daily clinical practice; however, the number of available RCTs is limited and heterogeneous. More RCTs of AI-assisted tools integrated into clinical practice are needed to advance the role of AI in medicine. TRIAL REGISTRATION: PROSPERO CRD42021286539; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=286539.
Overview
CoChat Summary
TL;DR
This systematic review identified only 39 RCTs (0.33% of 11,839 retrieved articles) of AI-assisted tools in clinical practice up to July 2021. Most (77%, 30/39) reported AI-assisted interventions outperforming usual care, but the evidence base was limited and heterogeneous.
Key findings
- Among 11,839 articles retrieved, only 39 (0.33%) RCTs were included; 77% (30/39) reported positive outcomes for AI-assisted interventions, and clinically relevant outcomes improved in 70% (21/30) of those positive studies.
- Gastroenterology dominated the field: 16/39 (41%) RCTs were related to gastroenterology, with 15 studies on AI-assisted endoscopy; biosignal-based AI tools were more common than clinical data–based tools, and endoscopic images were the most common biosignal (15/26, 58%).
- Risk of bias was low in 20% (8/39), some concerns in 31% (12/39), and high in 49% (19/39); missing outcome data and outcome measurement were the most common risk factors.
- Common study limitations were single-center design (22/39, 56%) and small sample size (n<1000; 33/39, 85%); 15/39 (38%) were open-label, and 7 studies were underpowered.
Methods
- Systematic review following PRISMA 2020 guidelines; searched CINAHL, Cochrane Central, Embase, MEDLINE, and PubMed from inception to July 14, 2021, for RCTs comparing AI-assisted tools with conventional clinical management without AI assistance; registered on PROSPERO (CRD42021286539).
- Two authors independently screened articles and extracted data, with disagreements resolved by consensus or senior author consultation; risk of bias was assessed using the Cochrane risk-of-bias tool for randomized trials (RoB 2), and primary end points were evaluated for clinical relevance.
Limitations
- Paper-stated: small sample size and single-center design limited generalizability; the number of available RCTs was limited and heterogeneous; among included studies, common author-listed limitations were single-center design (22/39, 56%) and small sample size (n<1000; 33/39, 85%), with open-label design (15/39, 38%) and 7 underpowered studies; 49% had high risk of bias.
- Evident from the text: the search ended July 2021, so later RCTs are not captured; 18% (7/39) were conference abstracts only; no meta-analysis or pooled effect estimates were reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
11 Valid · 1 Not valid · 5 Not checked · 99 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“The adenoma detection algorithm appears to be easier for cross-compatibility because of the distinct difference in appearance between adenoma (or polyp) and normal mucosa [56].”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Discussion > Principal Findings
What the sources say
- Supportedcites [56] source
The section “Pitfall No. 4: Cross-compatibility and external validation” says colon polyp detection appears easier to make cross-compatible because the visual differences between a polyp and normal colonic mucosa are distinct. This supports the citing sentence’s paraphrase. (LLM, conf=0.94)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Another obstacle to the implementation of AI or ML in daily clinical practice is the regulation of these technologies [8].”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Discussion > Conclusions and Recommendations
What the sources say
- Supportedcites [8] source
The Introduction identifies regulation as a crucial element shaping obstacles to implementing AI/ML in daily clinical practice. The Discussion likewise attributes challenges in adopting these technologies in medical practice partly to regulatory frameworks. (LLM, conf=0.95)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there have been well-designed studies where more “complex” ML approaches have outperformed regression-derived clinical risk scores on external validation [7,59,60].”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Discussion > Principal Findings
What the sources say
- Supportedcites [7] source
The Methods (Study Design) identify the final model as XGBoost, a gradient-boosted decision-tree algorithm. The Results (Performance of the ML Model, External Validation) report that it outperformed all three clinical risk scores on the composite endpoint. The Strengths section describes a large, prospective, multicenter cohort with external validation. (LLM, conf=0.92)
- Supportedcites [59] source
The Results > External Validation section reports that the support-vector-machine risk calculator, trained on MESA and tested on the external FLEMENGHO cohort, outperformed the ACC/AHA calculator in sensitivity, specificity, and accuracy. The Methods > External Validation section describes the separate validation cohort. Page numbers are not available in our copy. (LLM, conf=0.93)
- Partly supportedcites [60] sourceExperimental — accuracy not yet measured
The Abstract reports validation of a naïve Bayes sepsis model, and Overall model performance reports that it identified more additional cases than the triage-nurse process. That process was not a regression-derived clinical risk score. Practical observations during model development also identifies single-center validation as a limitation, so this paper does not establish the claimed outperformance of regression-derived scores on external validation. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most RCTs were published in the field of gastroenterology, with 15 studies on AI-assisted endoscopy.”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most RCTs were published in the field of gastroenterology, with 15 studies on AI-assisted endoscopy.”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
59 more claims
Create a free account to see all 116 claims and every source verdict.
- Probably not a substantial claim. 38
- It may state two separate claims as one. 18
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“PPx—AI: 17% (54/224) versus control: 8% (26/318)”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Results > Study Outcomes
Reported 17 · recomputed 24.107143
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“A total of 147 full manuscripts were individually assessed, of which 34 (23.1%) met the inclusion criteria. In addition, 4 more articles were identified by examining the references of the listed articles and manual searches (Figure 1). A total of 39 articles were included in this systematic review [6,17-54] as listed in Table 1.”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Results > Overview
Reported 39 · recomputed 38
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Results > Study Outcomes
“ENDOANGEL was an effective and robust system to improve the quality of EGD and has the potential to detect electrocardiogram in real time”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Results > Study End Points
“To verify the effectiveness of ENDOANGEL in improving endoscopy quality and pretest its performance in detecting EGCae in a multicenter randomized controlled trial”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Results > Study Outcomes
“El Solh et al [17], 2019”
§ Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review > Results > Overview
“El Solh et al [17], 2009”
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 34,787 · cited 1×
“Risk of bias was assessed using the Cochrane risk-of-bias tool for randomized trials [16].”
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/v7gm2 · Cited by 5,177
A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955
McCarthy, J., Minsky, M., Rochester, N., et al. (2006) · The AI Magazine · doi:10.1609/aimag.v27i4.1904 · Cited by 2,029 · cited 1×
“Artificial intelligence (AI) was first described in the 1950s as a theory of human intelligence being exhibited by machines, including but not limited to learning, reasoning, and problem-solving [1].”
Benjamens, S., Dhunnoo, P., & Meskó, B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00324-0 · Cited by 1,215
Wang, P., Berzin, T. M., Glissen Brown, J. R., Bharadwaj, S., Becq, A., Xiao, X., Liu, P., Li, L., Song, Y., Zhang, D., et al. (2019) · Gut · doi:10.1136/gutjnl-2018-317500 · Cited by 865 · cited 1×
“Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study Wang et al [28], 2019”
Trends in Chronic Kidney Disease in China
Zhang, L., Long, J., Jiang, W., Shi, Y., He, X., Zhou, Z., Li, Y., Yeung, R. O., Wang, J., Matsushita, K., et al. (2016) · New England Journal of Medicine · doi:10.1056/nejmc1602469 · Cited by 765 · cited 1×
“Another obstacle to the implementation of AI or ML in daily clinical practice is the regulation of these technologies [8].”
Brown, J. M., Campbell, J. P., Beers, A., Chang, K., Ostmo, S., Chan, R. V. P., Dy, J., Erdogmus, D., Ioannidis, S., Kalpathy-Cramer, J., et al. (2018) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2018.1934 · Cited by 657
Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial
Repici, A., Badalamenti, M., Maselli, R., Correale, L., Radaelli, F., Rondonotti, E., Ferrara, E., Spadaccini, M., Alkandari, A., Fugazza, A., et al. (2020) · Gastroenterology · doi:10.1053/j.gastro.2020.04.062 · Cited by 646 · cited 2×
“Original article Italy Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial Repici et al [38], 2020”
Wijnberge, M., Geerts, B. F., Hol, L., Lemmers, N., Mulder, M. P., Berge, P., Schenk, J., Terwindt, L. E., Hollmann, M. W., Vlaar, A. P., et al. (2020) · JAMA · doi:10.1001/jama.2020.0592 · Cited by 586 · cited 2×
“…care on depth and duration of intraoperative hypotension during elective noncardiac surgery: the HYPE randomized clinical trial Wijnberge et al [42], 2020 Neurology Abstract Israel Artificial intelligence-powered non-invasive and frequency-tuned electromagnetic field therapy improves upper…”
Hassan, C., Spadaccini, M., Iannone, A., Maselli, R., Jovani, M., Chandrasekar, V. T., Antonelli, G., Yu, H., Areia, M., Dinis-Ribeiro, M., et al. (2021) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2020.06.059 · Cited by 529 · cited 1×
“…to summarize the performance of recent AI-assisted tools in specific clinical settings, such as AI-assisted adenoma detection during colonoscopy [9], AI-assisted mammography in detecting breast cancer [10], AI-assisted intracranial hemorrhage recognition on computed tomography head imaging…”
Cruz Rivera, S., Liu, X., Chan, A.-W., Denniston, A. K., Calvert, M. J., Ashrafian, H., Beam, A. L., Collins, G. S., Darzi, A., Deeks, J. J., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30219-3 · Cited by 486
Yao, X., Rushlow, D. R., Inselman, J. W., McCoy, R. G., Thacher, T. D., Behnken, E. M., Bernard, M. E., Rosas, S. L., Akfaly, A., Misra, A., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01335-4 · Cited by 424
Role of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review
Yin, J., Ngiam, K. Y., & Teo, H. H. (2021) · Journal of Medical Internet Research · doi:10.2196/25759 · Cited by 419 · cited 1×
“A recent systematic review examined all studies of AI application in clinical practice, but was limited by restriction to English language and only searching full manuscripts published between January 2010 and May 2020 [13].”
Wang, P., Liu, X., Berzin, T. M., Glissen Brown, J. R., Liu, P., Zhou, C., Lei, L., Li, L., Guo, Z., Lei, S., et al. (2020) · The Lancet Gastroenterology & Hepatology · doi:10.1016/s2468-1253(19)30411-x · Cited by 415 · cited 2×
“…number of hypotensive events per patient): total time with hypotension and percentage of time spent with hypotension during surgery; incidence of hypertension (the number of hypotensive events per patient): total time with hypertension and percentage of time spent with hypertension during surgery”
Gong, D., Wu, L., Zhang, J., Mu, G., Shen, L., Liu, J., Wang, Z., Zhou, W., An, P., Huang, X., et al. (2020) · The Lancet Gastroenterology & Hepatology · doi:10.1016/s2468-1253(19)30413-3 · Cited by 393 · cited 3×
“To evaluate the noninferiority of day 30 dosimetry between a machine learning–based treatment planning system for prostate low-dose-rate brachytherapy and the conventional manual planning technique Nicolae et al [37], 2020”
Shimabukuro, D. W., Barton, C. W., Feldman, M. D., Mataraso, S. J., & Das, R. (2017) · BMJ Open Respiratory Research · doi:10.1136/bmjresp-2017-000234 · Cited by 382 · cited 1×
“For example, still images of the intestinal lumen captured during colonoscopy for AI-assisted adenoma detection are static data, whereas hourly captured vital signs and selected available laboratory tests for AI-assisted prediction of severe sepsis are dynamic data [18].”
Gupta, S., Lieberman, D., Anderson, J. C., Burke, C. A., Dominitz, J. A., Kaltenbach, T., Robertson, D. J., Shaukat, A., Syngal, S., & Rex, D. K. (2020) · American Journal of Gastroenterology · doi:10.14309/ajg.0000000000000544 · Cited by 343
Wu, L., Zhang, J., Zhou, W., An, P., Shen, L., Liu, J., Jiang, X., Huang, X., Mu, G., Wan, X., et al. (2019) · Gut · doi:10.1136/gutjnl-2018-317366 · Cited by 316 · cited 1×
“…Randomised controlled trial of WISENSE, a real-time quality improving system for monitoring blind spots during esophagogastroduodenoscopy Wu et al [29], 2019 Neurology Original article Ireland, the Netherlands, A machine-learning algorithm for neonatal seizure recognition: a multicentre,…”
Labovitz, D. L., Shafner, L., Reyes Gil, M., Virmani, D., & Hanina, A. (2017) · Stroke · doi:10.1161/strokeaha.116.016281 · Cited by 311 · cited 2×
“AI-assisted tools have been successfully applied in various clinical settings to assist diagnosis [5], improve therapy [6], and predict risk of mortality [7].”
Su, J.-R., Li, Z., Shao, X.-J., Ji, C.-R., Ji, R., Zhou, R.-C., Li, G.-C., Liu, G.-Q., He, Y.-S., Zuo, X.-L., et al. (2020) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2019.08.026 · Cited by 286
Deep Learning for Diabetes: A Systematic Review
Zhu, T., Li, K., Herrero, P., & Georgiou, P. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3040225 · Cited by 264 · cited 1×
“…colonoscopy [9], AI-assisted mammography in detecting breast cancer [10], AI-assisted intracranial hemorrhage recognition on computed tomography head imaging [11], AI-assisted glycemic control for patients with diabetes, and AI-assisted diagnosis of diabetes and its related complications [12].”
Artificial intelligence and colonoscopy experience: lessons from two randomised trials
Repici, A., Spadaccini, M., Antonelli, G., Correale, L., Maselli, R., Galtieri, P. A., Pellegatta, G., Capogreco, A., Milluzzo, S. M., Lollo, G., et al. (2022) · Gut · doi:10.1136/gutjnl-2021-324471 · Cited by 248 · cited 1×
“Artificial intelligence and colonoscopy experience: lessons from two randomised trials Repici et al [50]”
Glissen Brown, J. R., Mansour, N. M., Wang, P., Chuchuca, M. A., Minchenberg, S. B., Chandnani, M., Liu, L., Gross, S. A., Sengupta, N., & Berzin, T. M. (2022) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2021.09.009 · Cited by 235
Wang, P., Liu, P., Glissen Brown, J. R., Berzin, T. M., Zhou, G., Lei, S., Liu, X., Li, L., & Xiao, X. (2020) · Gastroenterology · doi:10.1053/j.gastro.2020.06.023 · Cited by 231 · cited 3×
“Original article China Lower adenoma miss rate of computer-aided detection-as-sisted colonoscopy vs routine white-light colonoscopy in a prospective tandem study Wang et al [40], 2020”
Machine Learning Outperforms ACC/AHA CVD Risk Calculator in MESA
Kakadiaris, I. A., Vrigkas, M., Yen, A. A., Kuznetsova, T., Budoff, M., & Naghavi, M. (2018) · Journal of the American Heart Association · doi:10.1161/jaha.118.009476 · Cited by 227 · cited 1×
“We should also test how machine intelligence and”
Shung, D. L., Au, B., Taylor, R. A., Tay, J. K., Laursen, S. B., Stanley, A. J., Dalton, H. R., Ngu, J., Schultz, M., & Laine, L. (2020) · Gastroenterology · doi:10.1053/j.gastro.2019.09.009 · Cited by 227 · cited 2×
“AI-assisted tools have been successfully applied in various clinical settings to assist diagnosis [5], improve therapy [6], and predict risk of mortality [7].”
Self-citationJayakumar, P., Moore, M. G., Furlough, K. A., Uhler, L. M., Andrawis, J. P., Koenig, K. M., Aksan, N., Rathouz, P. J., & Bozic, K. J. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.37107 · Cited by 224
Voss, C., Schwartz, J., Daniels, J., Kline, A., Haber, N., Washington, P., Tariq, Q., Robinson, T. N., Desai, M., Phillips, J. M., et al. (2019) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2019.0285 · Cited by 206 · cited 1×
“The United States Effect of wearable digital intervention for improving socialization in children with autism spectrum disorder: a randomized clinical trial Voss et al [27], 2019”
Lin, H., Li, R., Liu, Z., Chen, J., Yang, Y., Chen, H., Lin, Z., Lai, W., Long, E., Wu, X., et al. (2019) · EClinicalMedicine · doi:10.1016/j.eclinm.2019.03.001 · Cited by 199 · cited 1×
“To compare the diagnostic efficacy and treatment decision-making capacity between CC-Cruiser and ophthalmologists in real-world clinical settings Lin et al [24], 2019”
Persell, S. D., Peprah, Y. A., Lipiszko, D., Lee, J. Y., Li, J. J., Ciolino, J. D., Karmali, K. N., & Sato, H. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.0255 · Cited by 185 · cited 2×
“The United States Effect of home blood pressure monitoring via a smartphone hypertension coaching application or tracking application Persell et al [26], 2020 on adults with uncontrolled hypertension: a randomized clinical trial”
Wu, L., He, X., Liu, M., Xie, H., An, P., Zhang, J., Zhang, H., Ai, Y., Tong, Q., Guo, M., et al. (2021) · Endoscopy · doi:10.1055/a-1350-5583 · Cited by 168
Sackett, D. L., Haynes, B., Marshall, T., & Morgan, W. K. C. (1995) · Wiley eBooks · doi:10.1002/0470011815.b2a08019 · Cited by 153
Pavel, A. M., Rennie, J. M., de Vries, L. S., Blennow, M., Foran, A., Shah, D. K., Pressler, R. M., Kapellou, O., Dempsey, E. M., Mathieson, S. R., et al. (2020) · The Lancet Child & Adolescent Health · doi:10.1016/s2352-4642(20)30239-x · Cited by 151 · cited 3×
“…Ireland, the Netherlands, A machine-learning algorithm for neonatal seizure recognition: a multicentre, randomised, controlled trial Pavel et al [30], 2020 Sweden, and the United Kingdom Endocrinology, diabetes, and metabolism Original article Canada Carbohydrate counting app using image…”
InfluentialSackett, D., Haynes, B., Marshall, T., & Morgan, W. K. C. (1995) · The Lancet · doi:10.1016/s0140-6736(95)91850-7 · Cited by 148 · cited 1×
“…evidence to examine the effects of an intervention while controlling for unmeasured confounding factors, a comprehensive search of all RCTs studying AI-assisted tools in clinical practice would provide information regarding areas of opportunity for AI to affect real-world patient care [14].”
Auloge, P., Cazzato, R. L., Ramamurthy, N. K., Marini, P. D., Rousseau, C., Garnon, J., Charles, Y. P., Steib, J.-P., & Gangi, A. (2019) · European Spine Journal · doi:10.1007/s00586-019-06054-6 · Cited by 142 · cited 3×
“To evaluate technical feasibility, accuracy, safety, and patient radiation exposure granted by a novel navigational tool integrating augmented reality and artificial intelligence during percutaneous vertebroplasty of patients with vertebral compression fractures Auloge et al [32], 2020”
Blomberg, S. N., Christensen, H. C., Lippert, F., Ersbøll, A. K., Torp-Petersen, C., Sayre, M. R., Kudenchuk, P. J., & Folke, F. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.32320 · Cited by 138 · cited 2×
“…algorithm based on behavioral tests of affective cognition and subjective symptoms and to guide antidepressant treatment Browning et al [45], 2021 Nil Level of shared decision-making (assessed using the CollaboRATE survey); patient satisfaction with the consultation (numerical rating scale);…”
Kamba, S., Tamai, N., Saitoh, I., Matsui, H., Horiuchi, H., Kobayashi, M., Sakamoto, T., Ego, M., Fukuda, A., Tonouchi, A., et al. (2021) · Journal of Gastroenterology · doi:10.1007/s00535-021-01808-w · Cited by 124
Higgins, D., & Madai, V. I. (2020) · Advanced Intelligent Systems · doi:10.1002/aisy.202000052 · Cited by 118
Vennalaganti, P. R., Kaul, V., Wang, K. K., Falk, G. W., Shaheen, N. J., Infantolino, A., Johnson, D. A., Eisen, G. M., Gerson, L. B., Smith, M. S., et al. (2017) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2017.07.039 · Cited by 106 · cited 1×
“To evaluate the use of WATS d as an adjunct to biopsy sampling for the detection of HGD e or EAC f in a referral population with BE g Vennalaganti et al [21], 2018”
Strömblad, C. T. B., Baxter-King, R. G., Meisami, A., Yee, S., Levine, M. R., Ostrovsky, A., Stein, D., Iasonos, A. E., Weiser, M. R., García‐Aguilar, J., et al. (2021) · JAMA Surgery · doi:10.1001/jamasurg.2020.6361 · Cited by 105
Chen, D., Wu, L., Li, Y., Zhang, J., Liu, J., Huang, L., Jiang, X., Huang, X., Mu, G., Hu, S., et al. (2019) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2019.09.016 · Cited by 98 · cited 1×
“…the assistance of ENDOANGEL; consistency between ENDOANGEL and endoscopists’review The blind spot of 3 types of EGD with the assistance of ENDOANGEL To compare blind spots of sedated C-EGD q , unsedated U-TOE r , and unsedated C-EGD with and without the assistance of EN-DOANG Chen et al [34], 2020”
Luo, Y., Zhang, Y., Liu, M., Lai, Y., Liu, P., Wang, Z., Xing, T., Huang, Y., Li, Y., Li, A., et al. (2021) · Journal of Gastrointestinal Surgery · doi:10.1007/s11605-020-04802-4 · Cited by 97 · cited 1×
“Original article China Artificial intelligence-assisted colonoscopy for detection of colon polyps: a prospective, randomized cohort study Luo et al [48], 2021”
Liu, P., Wang, P., Glissen Brown, J. R., Berzin, T. M., Zhou, G., Liu, W., Xiao, X., Chen, Z., Zhang, Z., Zhou, C., et al. (2020) · Therapeutic Advances in Gastroenterology · doi:10.1177/1756284820979165 · Cited by 93 · cited 2×
“The single-monitor trial: an embedded CADe system increased adenoma detection during colonoscopy: a prospective randomized study Liu et al [36], 2020 Clinical oncology Original article Canada Conventional vs machine learning-based treatment planning in prostate brachytherapy: results of a phase I…”
Alfonsi, J., Choi, E., Arshad, T., Sammott, S.-A. S., Pais, V., Nguyen, C., Maguire, B. R., Stinson, J., & Palmert, M. R. (2020) · JMIR mhealth and uhealth · doi:10.2196/22074 · Cited by 85 · cited 2×
“To test the app’s usability and potential impact on carbohydrate counting accuracy Alfonsi et al [31], 2020 Nil Comparison between groups A and B in terms of accuracy, procedural safety, time for trocar placement, and patient radiation exposure (dose area product and fluoroscopy time)”
Yeo, M., Tahayori, B., Kok, H. K., Maingard, J., Kutaiba, N., Russell, J., Thijs, V., Jhamb, A., Chandra, R. V., Brooks, M., et al. (2021) · Journal of NeuroInterventional Surgery · doi:10.1136/neurintsurg-2020-017099 · Cited by 65
Browning, M. D., Bilderbeck, A. C., Dias, R., Dourish, C. T., Kingslake, J., Deckert, J., Goodwin, G. M., Gorwood, P. A. P. M., Guo, B., Harmer, C. J., et al. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-021-00981-z · Cited by 56 · cited 2×
“The United Kingdom, Spain, Germany, France, and the Netherlands The clinical effectiveness of using a predictive algorithm to guide antidepressant treatment in primary care (PRe-DicT): an open-label, randomised controlled trial Browning et al [45], 2021”
Nicolae, A., Semple, M., Lu, L., Smith, M., Chung, H., Loblaw, A., Morton, G., Mendez, L. C., Tseng, C.-L., Davidson, M., et al. (2020) · Brachytherapy · doi:10.1016/j.brachy.2020.03.004 · Cited by 48 · cited 10×
“Wang et al [40], 2020 Further investigation PDR; number of polyps per colonoscopy; number of adenomas per colonoscopy; sensitivity; specificity of the 3 skilled endo-scopists ADR To perform a double-blinded study using a sham control to more rigorously assess the effectiveness of a CADe system for improving detection…”
InfluentialApplication of Artificial Intelligence for Medical Research
Hamamoto, R. (2021) · Biomolecules · doi:10.3390/biom11010090 · Cited by 47 · cited 1×
“With an exponential increase of computational power, reduced cost of data storage, improved algorithmic sophistication, and increased availability of health data from electronic health records, the era of AI has arrived in different specialties of medicine [2-4].”
Brown, S. M., Jones, J., Kuttler, K. G., Keddington, R. K., Allen, T. L., & Haug, P. (2016) · BMC Emergency Medicine · doi:10.1186/s12873-016-0095-0 · Cited by 45 · cited 1×
“We should also test how machine intelligence and”
A Review of Applications of Machine Learning in Mammography and Future Challenges
Batchu, S., Liu, F., Amireh, A., Waller, J., & Umair, M. (2021) · Oncology · doi:10.1159/000515698 · Cited by 41 · cited 1×
“…in specific clinical settings, such as AI-assisted adenoma detection during colonoscopy [9], AI-assisted mammography in detecting breast cancer [10], AI-assisted intracranial hemorrhage recognition on computed tomography head imaging [11], AI-assisted glycemic control for patients with…”
Rafferty, A., Hall, R., & Johnston, C. S. (2021) · Journal of Medical Internet Research · doi:10.2196/24134 · Cited by 39 · cited 1×
“The United States A novel mobile app (Heali) for disease treatment in participants with irritable bowel syndrome: randomized controlled pilot trial Rafferty et al [49], 2021”
Artificial intelligence in health care: preparing for the fifth Industrial Revolution
Sung, J. J. �., Stewart, C. L., & Freedman, B. (2020) · The Medical Journal of Australia · doi:10.5694/mja2.50755 · Cited by 38 · cited 1×
“With an exponential increase of computational power, reduced cost of data storage, improved algorithmic sophistication, and increased availability of health data from electronic health records, the era of AI has arrived in different specialties of medicine [2-4].”
Self-citationAvari, P., Leal, Y., Herrero, P., Wos, M., Jugnee, N., Arnoriaga-Rodríguez, M., Thomas, M., Liu, C., Massana, Q., Lopez, B., et al. (2021) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2020.0301 · Cited by 37 · cited 2×
“The United Kingdom and Spain Safety and feasibility of the PEPPER adaptive bolus advisor and safety system; a randomized control study Avari et al [33], 2020”
InfluentialBiester, T., Nir, J., Remus, K., Farfel, A., Muller, I., Biester, S., Atlas, E., Dovč, K., Bratina, N., Kordonouri, O., et al. (2018) · Diabetes Obesity and Metabolism · doi:10.1111/dom.13585 · Cited by 35 · cited 1×
“…using the MD-Logic system in individuals with type 1 diabetes at home for day and night use, particularly without remote monitoring Biester et al [22], 2019 mg/dL (10-13.3 mmol/L), average and SD of glucose sensor readings, and overnight percentage of readings (“overnight” defined as 11:00…”
InfluentialBradley, F., & FIELD, J. B. (1995) · PubMed · Cited by 34
Challenges of developing artificial intelligence‐assisted tools for clinical medicine
Shung, D. L., & Sung, J. J. �. (2021) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.15378 · Cited by 33
Self-citationKamdar, M. M., Centi, A. J., Agboola, S. O., Fischer, N. C., Rinaldi, S. P., Strand, J. J., Traeger, L. N., Temel, J. S., Greer, J. A., El-Jawahri, A. R., et al. (2019) · Journal of Clinical Oncology · doi:10.1200/jco.2019.37.15_suppl.11514 · Cited by 15 · cited 1×
“Attitudes toward cancer treatment (Barriers Questionnaire II); anxiety (General anxiety Disorder-7); pain-related hospital admissions Pain severity To examine the impact of ePAL on cancer pain severity, attitudes toward cancer pain, and health care use Kamdar et al [25], 2019”
Predicting optimal CPAP by neural network reduces titration failure: a randomized study
Solh, A. E., Akinnusi, M. E., Patel, A. N., Bhat, A., & TenBrock, R. (2009) · Sleep And Breathing · doi:10.1007/s11325-009-0247-5 · Cited by 8
Gracey, B., Jones, C., Cho, D., et al. (2018) · Value in Health · doi:10.1016/j.jval.2018.04.532 · Cited by 5
Kamba, S., Tamai, N., Horiuchi, H., Matsui, H., Kobayashi, M., Ego, M., Sakamoto, T., Fukuda, A., Tonouchi, A., Shimahara, Y., et al. (2021) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2021.03.439 · Cited by 5 · cited 1×
“Abstract Japan A multicentre randomized controlled trial to verify the re-ducibility of adenoma miss rate of colonoscopy assisted with artificial intelligence–based software Kamba et al [47], 2021”
Weisinger, B., Bornstein, N. M., Shohami, E., et al. (2021) · Stroke · doi:10.1161/str.52.suppl_1.p194 · Cited by 4 · cited 1×
“…field therapy improves upper extremity motor function in sub-acute stroke patients: a pilot randomized controlled trial Weisinger et al [43], 2021 Emergency medicine Original article Denmark Effect of machine learning on dispatcher recognition of out-of-hospital cardiac arrest during…”
Guns, germs, and steel…and artificial intelligence
Samarasena, J. B. (2021) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2020.07.021 · Cited by 2 · cited 1×
“We should also test how machine intelligence and”
Liu, X. (2018) · Journal of Hypertension · doi:10.1097/01.hjh.0000547994.70348.44 · Cited by 2 · cited 1×
“To assess the effects of clinical decision support system of graph-based machine learning algorithms on blood pressure management and economic burden of disease Liu et al [20], 2018”
Research works printed but not linked: 1
- Pouska, J, 2019, Intensive Care Med Exp, 7, 001525
Cited by
Han, R., Acosta, J. N., Shakeri, Z., Ioannidis, J. P. A., Topol, E. J., & Rajpurkar, P. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00047-5 · Cited by 336
Deisenhofer, I., Albenque, J.-P., Busch, S., Gitenay, E., Mountantonakis, S., Roux, A., Horvilleur, J., Bakouboula, B., Oza, S. R., Abbey, S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03517-w · Cited by 162
El Arab, R. A., Abu-Mahfouz, M. S., Abuadas, F. H., Alzghoul, H., Almari, M., Ghannam, A., & Seweid, M. M. (2025) · Healthcare · doi:10.3390/healthcare13070701 · Cited by 108
Artificial intelligence in digital pathology — time for a reality check
Aggarwal, A., Bharadwaj, S., Corredor, G., Pathak, T. B., Badve, S., & Madabhushi, A. (2025) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-025-00991-6 · Cited by 96
“Despite the increasing efforts to incorporate AI into digital pathology, especially within prospective clinical trials, these initiatives remain limited in both scope and number 103 .”
Singareddy, S., SN, V. P., Jaramillo, A. P., Yasir, M., Iyer, N., Hussein, S., & Nath, T. S. (2023) · Cureus · doi:10.7759/cureus.46066 · Cited by 57
Metadata
- Authors
- Lam, T. Y. T., Cheung, M. F. K., Munro, Y. L., Lim, K. M., Shung, D., & Sung, J. J. Y.
- Affiliations
- Chinese University of Hong Kong (HK)
- Nanyang Technological University (SG)
- Yale University (US)
- Corresponding author
- Sung, Joseph J Y
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2022-08-25
- Volume
- 24
- Issue
- 8
- Pages
- e37188
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC9459941
- OpenAlex
- W4288467884
- Cited by
- 120
- References
- 64
- Citations by year
- 2026: 33 · 2025: 46 · 2024: 29 · 2023: 12
- FWCI
- 1.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion, cc-by)
- Funders
- Yale University
- National Institutes of Health (UL1TR001863)
- Raymond and Beverly Sackler Institute for Biological, Physical and Engineering Sciences, Yale University
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- artificial intelligence, AI-assisted tools, clinical practice, randomized controlled trials, clinical outcomes, endoscopy, gastroenterology
- MeSH terms
- Artificial Intelligence, Europe, Humans, North America, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.