Prospective real-world implementation of deep learning systems in healthcare: a systematic review guided by implementation science
Open accessAbstract
Deep learning (DL) applications in healthcare are expanding beyond proof-of-concept studies. Yet, the extent of its real-world implementation and impact on patient care and clinical workflows remains unclear due to the limited prospective real-world findings. Understanding how DL tools perform in real clinical environments is critical for guiding successful and sustainable deployment. Using a layered methodology with established implementation science frameworks, this systematic review aimed to systematically map the implementation strategies and outcomes of prospective DL implementation studies, proposing recommendations based on identified gaps of relevant studies to serve as a guide for the future implementation of DL systems. 20 articles were included: 3 from radiology, 1 from otolaryngology, 3 from dermatology, and 13 from ophthalmology. All studies assessed clinical outcomes, demonstrating the effectiveness and feasibility of integrating DL systems into existing clinical workflows. Adoption and appropriateness were the most frequently evaluated implementation outcomes; only one study evaluated implementation costs, and none evaluated sustainability. Stakeholder acceptability was only evaluated in 8 studies. Given the paucity of real-world DL implementation research, continued research into the clinical deployment of DL systems using hybrid effectiveness-implementation study designs as a framework is essential to facilitate its seamless and effective adoption into clinical practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Artificial intelligence is the most frequently implemented deep learning application in prospective real-world studies.
“Multiple artificial intelligence (AI) algorithms have been developed and validated for DR5–8 screening and it is the most frequently implemented deep learning application in prospective real-world studies9.”
From Referral uptake after diabetic retinopathy screening with artificial intelligence-assisted care pathways: a systematic review and meta-analysis (2026) · cites this paper as [9]
The Results table (page not available) includes several prospective real-world studies of deep-learning systems for diabetic retinopathy (DR) screening. The cited sentence’s “it” refers to DR screening, not artificial intelligence generally. The passages establish that DR screening is a recurring application, but do not establish that it is the most frequently implemented application. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · see citing works
Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012) · Medical Care · doi:10.1097/mlr.0b013e3182408812 · Cited by 4,058 · see citing works
Abràmoff, M. D., Lavin, P. T., Birch, M., Shah, N., & Folk, J. C. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0040-6 · Cited by 1,638 · see citing works
Standards for Reporting Implementation Studies (StaRI) Statement
Pinnock, H., Barwick, M., Carpenter, C. R., Eldridge, S., Grandes, G., Griffiths, C. J., Rycroft-Malone, J., Meissner, P., Murray, E., Patel, A., et al. (2017) · BMJ · doi:10.1136/bmj.i6795 · Cited by 1,446 · see citing works
Fleuren, L. M., Klausch, T. L. T., Zwager, C. L., Schoonmade, L. J., Guo, T., Roggeveen, L. F., Swart, E. L., Girbes, A. R. J., Thoral, P., Ercole, A., et al. (2020) · Intensive Care Medicine · doi:10.1007/s00134-019-05872-y · Cited by 755 · cited 1× · see citing works
“speci fi c medical specialties 4,10 – 13 .”
Arbabshirani, M. R., Fornwalt, B. K., Mongelluzzo, G. J., Suever, J. D., Geise, B. D., Patel, A. A., & Moore, G. J. (2018) · npj Digital Medicine · doi:10.1038/s41746-017-0015-z · Cited by 443 · see citing works
Young, A. T., Amara, D., Bhattacharya, A., & Wei, M. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00132-1 · Cited by 400 · see citing works
Sustainability in quality improvement: redefining value
Mortimer, F., Isherwood, J., Wilkinson, A., & Vaux, E. (2018) · Future Healthcare Journal · doi:10.7861/futurehosp.5-2-88 · Cited by 373 · see citing works
A deep learning system for predicting time to progression of diabetic retinopathy
Dai, L., Sheng, B., Chen, T., Wu, Q., Liu, R., Cai, C., Wu, L., Yang, D., Hamzah, H., Liu, Y., et al. (2024) · Nature Medicine · doi:10.1038/s41591-023-02702-z · Cited by 339 · see citing works
Self-citationEusebi, P. (2013) · Cerebrovascular Diseases · doi:10.1159/000353863 · Cited by 316 · see citing works
Ipp, E., Liljenquist, D., Bode, B., Shah, V. N., Silverstein, S., Regillo, C. D., Lim, J. I., Sadda, S., Domalpally, A., Gray, G., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.34254 · Cited by 256 · see citing works
Ruamviboonsuk, P., Tiwari, R., Sayres, R., Nganthavee, V., Hemarat, K., Kongprayoon, A., Raman, R., Levinstein, B., Liu, Y., Schaekermann, M., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00017-6 · Cited by 250 · see citing works
Integrated image-based deep learning and language models for primary diabetes care
Li, J., Guan, Z., Wang, J., Cheung, C. Y., Zheng, Y., Lim, L.-L., Lim, C. C., Ruamviboonsuk, P., Raman, R., Corsino, L., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03139-8 · Cited by 250 · see citing works
Self-citationBuchlak, Q. D., Esmaili, N., Leveque, J., Farrokhi, F. R., Bennett, C. C., Piccardi, M., & Sethi, R. K. (2019) · Neurosurgical Review · doi:10.1007/s10143-019-01163-8 · Cited by 226 · see citing works
Keel, S., Lee, P. Y., Scheetz, J., Li, Z., Kotowicz, M. A., MacIsaac, R. J., & He, M. (2018) · Scientific Reports · doi:10.1038/s41598-018-22612-2 · Cited by 220 · see citing works
The impact of machine learning on patient care: A systematic review
Ben-Israel, D., Jacobs, W. B., Casha, S., Lang, S., Ryu, W. H. A., de Lotbiniere-Bassett, M., & Cadotte, D. W. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2019.101785 · Cited by 207 · see citing works
Deep Learning Intervention for Health Care Challenges: Some Biomedical Domain Considerations
Tobore, I., Li, J., Yuhang, L., Al-Handarish, Y., Kandwal, A., Nie, Z., & Wang, L. (2019) · JMIR mHealth and uHealth · doi:10.2196/11966 · Cited by 180 · see citing works
Cost data in implementation science: categories and approaches to costing
Gold, H. T., McDermott, C., Hoomans, T., & Wagner, T. H. (2022) · Implementation Science · doi:10.1186/s13012-021-01172-6 · Cited by 152 · see citing works
Nwanosike, E. M., Conway, B. R., Merchant, H. A., & Hasan, S. S. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104679 · Cited by 140 · see citing works
Sustainability in Healthcare: Perspectives and Reflections Regarding Laboratory Medicine
Molero, A., Calabrò, M., Vignes, M., Gouget, B., & Gruson, D. (2020) · Annals of Laboratory Medicine · doi:10.3343/alm.2021.41.2.139 · Cited by 129 · see citing works
Nusinovici, S., Rim, T. H., Yu, M., Lee, G., Tham, Y. C., Cheung, N., Chong, C. C. Y., Soh, Z. D., Thakur, S., Lee, C. J., et al. (2022) · Age and Ageing · doi:10.1093/ageing/afac065 · Cited by 112 · see citing works
Self-citationArtificial Intelligence Applications in Pediatric Brain Tumor Imaging: A Systematic Review
Huang, J., Shlobin, N. A., Lam, S. K., & DeCuypere, M. G. (2021) · World Neurosurgery · doi:10.1016/j.wneu.2021.10.068 · Cited by 86 · see citing works
Wu, L., Xu, M., Jiang, X., He, X., Zhang, H., Ai, Y., Tong, Q.-Y., Lv, P., Lu, B., Guo, M., et al. (2021) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2021.09.017 · Cited by 80 · see citing works
Artificial intelligence-based psoriasis severity assessment: real-world study and application
Huang, K., Wu, X., Li, Y., Lv, C., Yan, Y., Wu, Z., Zhang, M., Huang, W., Jiang, Z., Hu, K., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/44932 · Cited by 70 · see citing works
Muñoz‐López, C., Ramírez‐Cornejo, C., Marchetti, M. A., Han, S. S., Del Barrio‐Díaz, P., Jaque, A., Uribe, P., Majerson, D., Curi, M., Del Puerto, C., et al. (2021) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.16979 · Cited by 70 · see citing works
Detecting visually significant cataract using retinal photograph-based deep learning
Tham, Y. C., Goh, J. H. L., Anees, A., Lei, X., Rim, T. H., Chee, M.-L., Wang, Y. X., Jonas, J. B., Thakur, S., Teo, Z. L., et al. (2022) · Nature Aging · doi:10.1038/s43587-022-00171-6 · Cited by 59 · see citing works
Self-citationHu, W., Joseph, S., Li, R., Woods, E., Sun, J., Shen, M., Jan, C. L., Zhu, Z., He, M., & Zhang, L. (2024) · eClinicalMedicine · doi:10.1016/j.eclinm.2023.102387 · Cited by 47 · see citing works
Srisubat, A., Kittrongsiri, K., Sangroongruangsri, S., Khemvaranan, C., Shreibati, J. B., Ching, J., Hernandez, J., Tiwari, R., Hersch, F. B., Liu, Y., et al. (2023) · Ophthalmology and Therapy · doi:10.1007/s40123-023-00688-y · Cited by 42 · see citing works
Gu, C., Wang, Y., Jiang, Y., Xu, F., Wang, S., Liu, R., Yuan, W., Abudureyimu, N., Wang, Y., Lu, Y., et al. (2023) · British Journal of Ophthalmology · doi:10.1136/bjo-2022-322940 · Cited by 39 · see citing works
Self-citationTseng, R. M. W. W., Gunasekeran, D. V., Tan, S. S. H., Rim, T. H., Lum, E., Tan, G. S. W., Wong, T. Y., & Tham, Y. C. (2021) · Asia-Pacific Journal of Ophthalmology · doi:10.1097/apo.0000000000000400 · Cited by 34 · see citing works
Self-citationGoessinger, E., Niederfeilner, J.-C., Cerminara, S., Maul, J., Kostner, L., Kunz, M., Huber, S., Koral, E., Habermacher, L., Sabato, G., et al. (2024) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.19905 · Cited by 29 · see citing works
Skevas, C., de Olaguer, N. P., Lleó, A., Thiwa, D., Schroeter, U., Lopes, I. V., Mautone, L., Linke, S. J., Spitzer, M. S., Yap, D., et al. (2024) · BMC Ophthalmology · doi:10.1186/s12886-024-03306-y · Cited by 29 · see citing works
Acceptability of artificial intelligence-based retina screening in general population
Shah, P. N., Mishra, D. K. C., Shanmugam, M. P., Vighnesh, M. J., & Jayaraj, H. (2022) · Indian Journal of Ophthalmology · doi:10.4103/ijo.ijo_1840_21 · Cited by 28 · see citing works
Chen, B., Li, Y., Sun, Y., Sun, H., Wang, Y., Lyu, J., Guo, J., Bao, S., Cheng, Y., Niu, X., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/51706 · Cited by 21 · see citing works
National Use of Artificial Intelligence for Eye Screening in Singapore
Gunasekeran, D. V., Miller, S., Hsu, W., Lee, M. L., Wong, H. T., Lee, M. T., Lamoureux, E. L., Ting, D. S. W., Tan, G. S. W., & Wong, T. Y. (2024) · NEJM AI · doi:10.1056/aics2400404 · Cited by 20 · see citing works
Tseng, R. M. W. W., Tham, Y. C., Rim, T. H., & Cheng, C. (2021) · Clinical and Experimental Ophthalmology · doi:10.1111/ceo.13971 · Cited by 20 · see citing works
Self-citationWang, J., Sui, X., Zhao, R., Du, H., Wang, J., Wang, Y., Qin, R., Lü, X., Ma, Z., Xu, Y., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10087-3 · Cited by 19 · see citing works
Bora, A., Tiwari, R., Bavishi, P., Virmani, S., Huang, R., Traynis, I., Corrado, G. S., Peng, L. H., Webster, D. R., Varadarajan, A. V., et al. (2023) · Translational Vision Science & Technology · doi:10.1167/tvst.12.12.11 · Cited by 15 · see citing works
Zhu, A., Tailor, P., Verma, R., Zhang, I., Schott, B., Ye, C., Szirth, B., Habiel, M. M., & Khouri, A. S. (2023) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x231158832 · Cited by 14 · see citing works
Antaki, F., Hammana, I., Tessier, M.-C., Boucher, A., Jetté, M. L. D., Beauchemin, C., Hammamji, K., Ong, A. Y., Rhéaume, M., Gauthier, D., et al. (2024) · JMIR Diabetes · doi:10.2196/59867 · Cited by 11 · see citing works
Factors Associated With Patient Ratings of Timeliness of Primary Care Appointments
Mayo‐Smith, M. F., Radwin, L. E., Abdulkerim, H., & Mohr, D. C. (2020) · Journal of Patient Experience · doi:10.1177/2374373520968979 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- [1]Takahashi, T., Nozaki, K., Gonda, T., Mameno, T. & Ikebe, K. Deep learning-based detection of dental prostheses and restorations. Sci. Rep. 11, 1960 (2021). 33479303 PMC7820223
- [16]Liu, X. et al. Evaluation of an OCT-AI-based telemedicine platform for retinal disease screening and referral in a primary care setting. Transl. Vis. Sci. Technol. 11, 4 (2022). 35254422 PMC8914565
- [17]Liu, B. et al. Assisting scalable diagnosis automatically via CT images in the combat against COVID-19. Sci. Rep. 11, 4145 (2021). 33603047 PMC7892869
- [26]Liu, R. et al. Application of artificial intelligence-based dual-modality analysis combining fundus photography and optical coherence tomography in diabetic retinopathy screening in a community hospital. Biomed. Eng. Online 21, 47 (2022). 35859144 PMC9301845
- [30]Scheetz, J. et al. Real-world artificial intelligence-based opportunistic screening for diabetic retinopathy in endocrinology and indigenous healthcare settings in Australia. Sci. Rep. 11, 15808 (2021). 34349130 PMC8339059
- [46]Li, R. et al. Cost-effectiveness and cost-utility of traditional and telemedicine combined population-based age-related macular degeneration and diabetic retinopathy screening in rural and urban China. Lancet Reg. Health West Pac. 23, 100435 (2022). 35355615 PMC8958534
Cited by
Sorted by citations
Leigh, J. A., Sherrington, A., Barber, A. R. J., Turner, A. W., Kidd, M., Powell, J., & Pope, C. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02616-3 · Cited by 2 · see citing works
“Multiple arti fi cial intelligence (AI) algorithms have been developed and validatedforDR 5 – 8 screeninganditisthemostfrequentlyimplemented deep learning application in prospective real-world studies 9 .”
Translating AI Into the Eye Clinic—From Models to Clinical Workflow
Zhang, D., Wang, Y. X., Tham, Y. C., & Wong, T. Y. (2026) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2026.2360
Self-citationZhang, S., Lang, X., & Liu, L. (2026) · Human Mutation · doi:10.1155/humu/9236120
任栋慧, 曲玉芳, & Sun, H. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1891645
State-of-the-art artificial intelligence frameworks and models for diagnostic veterinary anatomy
Sahu, S. K., & Mishra, U. K. (2026) · Annals of Anatomy - Anatomischer Anzeiger · doi:10.1016/j.aanat.2026.153377
Low-power approximate multiplier design based on fast approximate compressors for AI applications
Tang, X., Shang, D., Liu, W., & Liu, X. (2026) · Integration · doi:10.1016/j.vlsi.2026.102913
Eusuff, S., Kim, J., Fatima, Z., Yalcinkaya, D., Sahashi, Y., Sandhu, A., Nguyen, L. H., Ambrosy, A. P., Cheng, P. P., Chang, S. T., et al. (2026) · American Heart Journal · doi:10.1016/j.ahj.2026.107628
Saranya, K. R., & Suganya, Y. (2026) · Diagnostics · doi:10.3390/diagnostics16203407
Risks and Ethical Concerns of Artificial Intelligence in Plastic Surgery
Özmen, B. B., & Schwarz, G. S. (2026) · Clinics in Plastic Surgery · doi:10.1016/j.cps.2026.09.004
- Authors
- Tseng, R. M. W. W., Ong, L. C., Goh, J. H. L., Chen, Y., Chen, T., Lum, E., & Tham, Y.-C.
- Affiliations
- National University of Singapore (SG)
- SingHealth (SG)
- Corresponding author
- Tham, Yih-Chung
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2026-01-23
- Volume
- 9
- Issue
- 1
- Pages
- 172-172
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC12913662
- OpenAlex
- W7125529244
- Cited by
- 9
- References
- 41
- Citations by year
- 2026: 9
- FWCI
- 5.52
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- npj Digital Medicine (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- National University of Singapore (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Medical Research Council of Singapore (NMRC/MOH/HCSAINV21nov-0001)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- deep learning, clinical implementation, implementation science, field deployment, clinical workflow, implementation outcomes, technology adoption, stakeholder acceptance, implementation costs, sustainability, radiology, ophthalmology, dermatology, otolaryngology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
