Role of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review
Open accessPublished version of a preprint
The DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index.
Abstract
Background: Artificial intelligence (AI) applications are growing at an unprecedented pace in health care, including disease diagnosis, triage or screening, risk analysis, surgical operations, and so forth. Despite a great deal of research in the development and validation of health care AI, only few applications have been actually implemented at the frontlines of clinical practice. Objective: The objective of this study was to systematically review AI applications that have been implemented in real-life clinical practice. Methods: We conducted a literature search in PubMed, Embase, Cochrane Central, and CINAHL to identify relevant articles published between January 2010 and May 2020. We also hand searched premier computer science journals and conferences as well as registered clinical trials. Studies were included if they reported AI applications that had been implemented in real-world clinical settings. Results: We identified 51 relevant studies that reported the implementation and evaluation of AI applications in clinical practice, of which 13 adopted a randomized controlled trial design and eight adopted an experimental design. The AI applications targeted various clinical tasks, such as screening or triage (n=16), disease diagnosis (n=16), risk analysis (n=14), and treatment (n=7). The most commonly addressed diseases and conditions were sepsis (n=6), breast cancer (n=5), diabetic retinopathy (n=4), and polyp and adenoma (n=4). Regarding the evaluation outcomes, we found that 26 studies examined the performance of AI applications in clinical settings, 33 studies examined the effect of AI applications on clinician outcomes, 14 studies examined the effect on patient outcomes, and one study examined the economic impact associated with AI implementation. Conclusions: This review indicates that research on the clinical implementation of AI applications is still at an early stage despite the great potential. More research needs to assess the benefits and challenges associated with clinical AI applications through a more rigorous methodology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A recent systematic review examined all studies of AI application in clinical practice, but was limited by restriction to English language and only searching full manuscripts published between January 2010 and May 2020 [13].”
From Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review (2022) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Of these, 6 were related to diabetes.”
From An Assessment of How Clinicians and Staff Members Use a Diabetes Artificial Intelligence Prediction Tool: Mixed Methods Study (2023) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These applications used computer vision to diagnose diabetic retinopathy from retinal images and EHR data to predict those at risk for hyperglycemia.”
From An Assessment of How Clinicians and Staff Members Use a Diabetes Artificial Intelligence Prediction Tool: Mixed Methods Study (2023) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unfortunately, the implementation of AI tools for diabetes has lagged, and few tools are used in practice, limiting their impact.”
From An Assessment of How Clinicians and Staff Members Use a Diabetes Artificial Intelligence Prediction Tool: Mixed Methods Study (2023) · cites this paper as [31]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Despite the ongoing development and testing of AI applications in healthcare, the implementation of medical AI systems in clinical care remains in its early stages [2, 3].”
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Cochrane Handbook for Systematic Reviews of Interventions
(2019) · doi:10.1002/9781119536604 · Cited by 14,828
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
Cumpston, M., Li, T., Page, M. J., Chandler, J., Welch, V. A., Higgins, J. P., & Thomas, J. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000142 · Cited by 7,218
Artificial intelligence in healthcare: past, present and future
Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., & Wang, Y. (2017) · Stroke and Vascular Neurology · doi:10.1136/svn-2017-000101 · Cited by 4,837
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817
The practical implementation of artificial intelligence technologies in medicine
He, J., Baxter, S. L., Xu, J., Xu, J., Zhou, X., & Zhang, K. (2019) · Nature Medicine · doi:10.1038/s41591-018-0307-0 · Cited by 2,246
A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955
McCarthy, J., Minsky, M., Rochester, N., et al. (2006) · The AI Magazine · doi:10.1609/aimag.v27i4.1904 · Cited by 2,029
Research works printed but not linked: 1
- 10.1136/bjophthalmol-2019-314729 · DOI
Cited by
Petersson, L., Larsson, I., Nygren, J. M., Nilsen, P., Neher, M., Reed, J. E., Tyskbo, D., & Svedberg, P. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08215-8 · Cited by 434
Esmaeilzadeh, P. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102861 · Cited by 377
Artificial Intelligence Applications in Health Care Practice: Scoping Review
Sharma, M., Savage, C., Nair, M., Larsson, I., Svedberg, P., & Nygren, J. M. (2022) · Journal of Medical Internet Research · doi:10.2196/40238 · Cited by 198
Next-generation agentic AI for transforming healthcare
Karunanayake, N. (2025) · Informatics and Health · doi:10.1016/j.infoh.2025.03.001 · Cited by 165
Hogg, H. D. J., Al-Zubaidy, M., Technology Enhanced Macular Services Study Reference Group, Talks, J., Denniston, A. K., Kelly, C. J., Malawana, J., Papoutsi, C., Teare, M. D., Keane, P. A., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/39742 · Cited by 163
Formosa, P., Rogers, W., Griep, Y., Bankins, S., & Richards, D. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107296 · Cited by 129
Co-design of human-centered, explainable AI for clinical decision support
Panigutti, C., Beretta, A., Fadda, D., Giannotti, F., Pedreschi, D., Perotti, A., & Rinzivillo, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3587271 · Cited by 127
Randomized Controlled Trials of Artificial Intelligence in Clinical Practice: Systematic Review
Lam, T. Y. T., Cheung, M. F. K., Munro, Y. L., Lim, K. M., Shung, D., & Sung, J. J. Y. (2022) · Journal of Medical Internet Research · doi:10.2196/37188 · Cited by 120
“A recent systematic review examined all studies of AI application in clinical practice, but was limited by restriction to English language and only searching full manuscripts published between January 2010 and May 2020 [13].”
Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework
van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad088 · Cited by 109
Clinical and Surgical Applications of Large Language Models: A Systematic Review
Pressman, S. M., Borna, S., Gomez-Cabello, C. A., Haider, S. A., Haider, C. R., & Forte, A. J. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13113041 · Cited by 107
Voets, M. M., Veltman, J., Slump, C. H., Siesling, S., & Koffijberg, H. (2022) · Value in Health · doi:10.1016/j.jval.2021.11.1362 · Cited by 105
Allen, M. R., Webb, S., Mandvi, A., Frieden, M., Tai-Seale, M., & Kallenberg, G. (2024) · BMC Primary Care · doi:10.1186/s12875-024-02282-y · Cited by 100
van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad075 · Cited by 72
Strategies to improve the impact of artificial intelligence on health equity: scoping review
Berdahl, C. T., Baker, L., Mann, S., Osoba, O., & Girosi, F. (2023) · JMIR AI · doi:10.2196/42936 · Cited by 68
Achieving large-scale clinician adoption of AI-enabled decision support
Scott, I. A., van der Vegt, A., Lane, P., McPhail, S., & Magrabi, F. (2024) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100971 · Cited by 65
The use of AI in mental health services to support decision-making: scoping review
Auf, H., Svedberg, P., Nygren, J., Nair, M., & Lundgren, L. E. (2025) · Journal of Medical Internet Research · doi:10.2196/63548 · Cited by 56
Wang, D.-Y., Ding, J., Sun, A.-L., Liu, S.-G., Jiang, D., Li, N., & Yu, J.-K. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad118 · Cited by 54
Preti, L. M., Ardito, V., Compagni, A., Petracca, F., & Cappellaro, G. (2024) · Journal of Medical Internet Research · doi:10.2196/55897 · Cited by 62
Impact of Artificial Intelligence–Based Technology on Nurse Management: A Systematic Review
Gonzalez-Garcia, A., Pérez-González, S., Benavides, C., Pinto-Carral, A., Quiroga-Sánchez, E., & Marqués-Sánchez, P. (2024) · Journal of Nursing Management · doi:10.1155/2024/3537964 · Cited by 53
Henzler, D., Schmidt, S., Koçar, A., Herdegen, S., Lindinger, G. L., Maris, M. T., Bak, M. A. R., Willems, D. L., Tan, H. L., Lauerer, M., et al. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-12664-2 · Cited by 54
“Despite the ongoing development and testing of AI applications in healthcare, the implementation of medical AI systems in clinical care remains in its early stages [2, 3].”
The aspects of running artificial intelligence in emergency care; a scoping review
(2023) · Archives of Academic Emergency Medicine · doi:10.22037/aaem.v11i1.1974 · Cited by 45
The AI Future of Emergency Medicine
Petrella, R. J. (2024) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2024.01.031 · Cited by 43
Artificial intelligence tool development: what clinicians need to know?
Chew, B.-H., & Ngiam, K. Y. (2025) · BMC Medicine · doi:10.1186/s12916-025-04076-0 · Cited by 41
“The integration of AI into routine clinical care is accelerating, which is now reviewing patient histories, drafting physician notes, offering patient instructions, and not just reading X-rays and histopathological images [11–14].”
Self-citationSvedberg, P., Reed, J., Nilsen, P., Barlow, J., Macrae, C., & Nygren, J. (2022) · JMIR Research Protocols · doi:10.2196/34920 · Cited by 39
Weinert, L., Müller, J., Svensson, L., & Heinze, O. (2022) · JMIR Medical Informatics · doi:10.2196/34678 · Cited by 33
Wenderott, K., Krups, J., Luetkens, J. A., & Weigl, M. (2024) · Applied Ergonomics · doi:10.1016/j.apergo.2024.104243 · Cited by 33
Bentwich, I. (2023) · Drug Discovery Today · doi:10.1016/j.drudis.2023.103515 · Cited by 22
Zhang, M., Tang, E., Ding, H., & Zhang, Y. (2024) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2024_jslhr-24-00157 · Cited by 22
Liaw, W. R., Ramos Silva, Y., Soltero, E. G., Krist, A., & Stotts, A. L. (2023) · JMIR AI · doi:10.2196/45032 · Cited by 14
“A systematic review identified only 51 studies involving AI implementation [31].”
BackgroundHarnessing AI in critical care: opportunities, challenges and key steps for success
Kalimouttou, A., Stevens, R. D., & Pirracchio, R. (2026) · Thorax · doi:10.1136/thorax-2024-222125 · Cited by 12
Alam, M. S. B., Lameesa, A., Sharmin, S., Afrin, S., Ahmed, S. F., Nikoo, M. R., & Gandomi, A. H. (2025) · Digital Communications and Networks · doi:10.1016/j.dcan.2025.04.001 · Cited by 9
Alayed, H. M., & Awamleh, F. T. (2025) · ECONOMICS · doi:10.2478/eoik-2025-0094 · Cited by 3
Metadata
- Authors
- Yin, J., Ngiam, K. Y., & Teo, H. H.
- Affiliations
- National University of Singapore (SG)
- National University Hospital (SG)
- Corresponding author
- Teo, Hock Hai
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2021-04-22
- Volume
- 23
- Issue
- 4
- Pages
- e25759
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC8103304
- OpenAlex
- W3134138028
- Cited by
- 419
- References
- 86
- Citations by year
- 2026: 75 · 2025: 126 · 2024: 106 · 2023: 69 · 2022: 34 · 2021: 6
- FWCI
- 4.50
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- National University of Singapore (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National University of Singapore
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- artificial intelligence in healthcare, clinical implementation of AI, clinical practice, disease diagnosis, risk analysis, sepsis, breast cancer, diabetic retinopathy, patient outcomes, economic impact of AI
- MeSH terms
- Artificial Intelligence, Humans, Randomized Controlled Trials as Topic, Risk Assessment, Sepsis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.