CoChat
Sign Up
Cited by 59Retinal Imaging and AnalysisArtificial Intelligence in Healthcare and EducationRetinal Diseases and Treatments

Health Economic and Safety Considerations for Artificial Intelligence Applications in Diabetic Retinopathy Screening

Open access
Xie, Yuchen, Gunasekeran, Dinesh V., Balaskas, Konstantinos, Keane, Pearse A., Sim, Dawn A., Bachmann, Lucas M., Macrae, Carl, Ting, Daniel S. W.·2020·Translational Vision Science & Technology
DOI 10.1167/tvst.9.2.22PMC PMC7396187
Open this paper in CoChat — free account

Abstract

Systematic screening for diabetic retinopathy (DR) has been widely recommended for early detection in patients with diabetes to address preventable vision loss. However, substantial manpower and financial resources are required to deploy opportunistic screening and transition to systematic DR screening programs. The advent of artificial intelligence (AI) technologies may improve access and reduce the financial burden for DR screening while maintaining comparable or enhanced clinical effectiveness. To deploy an AI-based DR screening program in a real-world setting, it is imperative that health economic assessment (HEA) and patient safety analyses are conducted to guide appropriate allocation of resources and design safe, reliable systems. Few studies published to date include these considerations when integrating AI-based solutions into DR screening programs. In this article, we provide an overview of the current state-of-the-art of AI technology (focusing on deep learning systems), followed by an appraisal of existing literature on the applications of AI in ophthalmology. We also discuss practical considerations that drive the development of a successful DR screening program, such as the implications of false-positive or false-negative results and image gradeability. Finally, we examine different plausible methods for HEA and safety analyses that can be used to assess concerns regarding AI-based screening.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1167/tvst.9.2.22

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The trade-off between diagnostic performance and cost-effectiveness of AI in health screening has not been adequately addressed.

    “However, this aspect is commonly underestimated, and the trade-off between diagnostic performance and cost-effectiveness has not been adequately addressed10.”

    From Economic evaluation for medical artificial intelligence: accuracy vs. cost-effectiveness in a diabetic retinopathy screening case (2024) · cites this paper as [10]

    The Methods of HEA section says few studies have assessed the health economics of AI-assisted diabetic retinopathy screening, while the section on false positives explains how screening performance can affect costs. The paper supports a gap in evaluating performance alongside costs for diabetic retinopathy screening, but does not establish the broader claim about AI in health screening generally. Page numbers are unavailable in our copy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 93 · Cited by 59

Research works printed but not linked: 73

  1. [2]GBD 2015. Mortality and Causes of Death Collaborators. Global, regional, and national life expectancy, all-cause mortality, and cause-specific mortality for 249 causes of death, 1980-2015: a systematic analysis for the Global Burden of Disease Study 2015. Lancet. 2016; 388: 1459– 1544. 27733281 PMC5388903
  2. [3]Ting DS, Cheung GC, Wong TY Diabetic retinopathy: global prevalence, major risk factors, screening practices and public health challenges: a review. Clin Exp Ophthalmol. 2016; 44: 260– 277. 26716602
  3. [4]Nangia V, Jonas JB, George R, et al. Prevalence and causes of blindness and vision impairment: magnitude, temporal trends and projections in South and Central Asia. Br J Ophthalmol. 2019; 103: 871– 877. 30409914
  4. [5]Wild S, Roglic G, Green A, Sicree R, King H Global prevalence of diabetes: estimates for the year 2000 and projections for 2030. Diabetes Care. 2004; 27: 1047– 1053. 15111519
  5. [6]Mohamed Q, Gillies MC, Wong TY Management of diabetic retinopathy: a systematic review. JAMA. 2007; 298: 902– 916. 17712074
  6. [7]Klein R, Klein BE, Moss SE, Cruickshanks KJ The Wisconsin Epidemiologic Study of Diabetic Retinopathy. XV. The long-term incidence of macular edema. Ophthalmology. 1995; 102: 7– 16. 7831044
  7. [8]Ferris FL, Patz A Macular edema. A complication of diabetic retinopathy. Surv Ophthalmol. 1984; 28(suppl): 452– 461. 6379946
  8. [9]Wong TY, Bressler NM Artificial intelligence with deep learning technology looks into diabetic retinopathy screening. JAMA. 2016; 316: 2366– 2367. 27898977
  9. [10]Ting DSW, Pasquale LR, Peng L, et al. Artificial intelligence and deep learning in ophthalmology. Br J Ophthalmol. 2019; 103: 167– 175. 30361278 PMC6362807
  10. [11]Liu X, Faes L, Kale AU, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health, 2019; 1: 271– 297. 33323251
  11. [12]Challen R, Denny J, Pitt M, Gompels L, Edwards T, Tsaneva-Atanasova K Artificial intelligence, bias and clinical safety. BMJ Qual Saf. 2019; 28: 231– 237. PMC6560460 30636200
  12. [13]Ting DS, Gunasekeran DV, Wickham L, Wong TY Next generation telemedicine platforms to screen and triage. Br J Ophthalmol. 2020; 104: 299– 300. 31796427
  13. [14]Wilkinson CP, Ferris FL, Klein RE, et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology. 2003; 110: 1677– 1682. 13129861
  14. [15]Abràmoff MD, Folk JC, Han DP, et al. Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol. 2013; 131: 351– 357. 23494039
  15. [16]Ting DSW, Cheung CY, Lim G, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA. 2017; 318: 2211– 2223. 29234807 PMC5820739
  16. [17]Bellemo V, Lim G, Rim TH, et al. Artificial intelligence screening for diabetic retinopathy: the real-world emerging application. Curr Diab Rep. 2019; 19: 72. 31367962
  17. [18]Abràmoff MD, Lavin PT, Birch M, Shah N, Folk JC Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med. 2018; 1: 39. 31304320 PMC6550188
  18. [19]De Fauw J, Ledsam JR, Romera-Paredes B, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med. 2018; 24: 1342– 1350. 30104768
  19. [20]Tufail A, Kapetanakis VV, Salas-Vega S, et al. An observational study to assess if automated diabetic retinopathy image assessment software can replace one or more steps of manual imaging grading and to determine their cost-effectiveness. Health Technol Assess. 2016; 20: 1– 72. PMC5204130 27981917
  20. [21]Raumviboonsuk P, Krause J, Chotcomwongse P, et al. Deep learning versus human graders for classifying diabetic retinopathy severity in a nationwide screening program. NPJ Digit Med. 2019; 2: 25. 31304372 PMC6550283
  21. [23]Bellemo V, Lim ZW, Lim G, et al. Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study. Lancet Digit Health. 2019; 1: e35– e44. 33323239
  22. [24]Wahl B, Cossy-Gantner A, Germann S, Schwalbe NR Artificial intelligence (AI) and global health: how can AI contribute to health in resource-poor settings? BMJ Glob Health. 2018; 3: e000798. PMC6135465 30233828
  23. [25]Bjørvig S, Johansen MA, Fossen K An economic analysis of screening for diabetic retinopathy. J Telemed Telecare. 2002; 8: 32– 35. 11809082
  24. [26]Aoki N, Dunn K, Fukui T, Beck JR, Schull WJ, Li HK Cost-effectiveness analysis of telemedicine to evaluate diabetic retinopathy in a prison population. Diabetes Care. 2004; 27: 1095– 1101. 15111527
  25. [27]Kirkizlar E, Serban N, Sisson JA, Swann JL, Barnes CS, Williams MD Evaluation of telemedicine for screening of diabetic retinopathy in the Veterans Health Administration. Ophthalmology. 2013; 120: 2604– 2610. 24084501
  26. [28]Whited JD, Datta SK, Aiello LM, et al. A modeled economic analysis of a digital tele-ophthalmology system as used by three federal health care agencies for detecting proliferative diabetic retinopathy. Telemed J E Health. 2005; 11: 641– 651. 16430383
  27. [29]Li Z, Wu C, Olayiwola JN, Hilaire DS, Huang JJ Telemedicine-based digital retinal imaging vs standard ophthalmologic evaluation for the assessment of diabetic retinopathy. Conn Med. 2012; 76: 85– 90. 22670358
  28. [30]Kumar S, Tay-Kearney ML, Chaves F, Constable IJ, Yogesan K Remote ophthalmology services: cost comparison of telemedicine and alternative service delivery options. J Telemed Telecare. 2006; 12: 19– 22. 16438774
  29. [31]Nguyen HV, Tan GS, Tapp RJ, et al. Cost-effectiveness of a National Telemedicine Diabetic Retinopathy Screening Program in Singapore. Ophthalmology. 2016; 123: 2571– 2580. 27726962
  30. [32]Peng J, Zou H, Wang W, et al. Implementation and first-year screening results of an ocular telehealth system for diabetic retinopathy in China. BMC Health Serv Res. 2011; 11: 250. 21970365 PMC3200176
  31. [33]Rachapelle S, Legood R, Alavi Y, et al. The cost-utility of telemedicine to screen for diabetic retinopathy in India. Ophthalmology. 2013; 120: 566– 573. 23211635
  32. [34]Scotland GS, McNamee P, Philip S, et al. Cost-effectiveness of implementing automated grading within the national screening programme for diabetic retinopathy in Scotland. Br J Ophthalmol. 2007; 91: 1518– 1523. 17585001 PMC2095413
  33. [35]Philip S, Fleming AD, Goatman KA, et al. The efficacy of automated “disease/no disease” grading for diabetic retinopathy in a systematic screening programme. Br J Ophthalmol. 2007; 91: 1512– 1517. 17504851 PMC2095421
  34. [36]Scotland GS, McNamee P, Fleming AD, et al. Costs and consequences of automated algorithms versus manual grading for the detection of referable diabetic retinopathy. Br J Ophthalmol. 2010; 94: 712– 719. 19965826
  35. [37]Prescott G, Sharp P, Goatman K, et al. Improving the cost-effectiveness of photographic screening for diabetic macular oedema: a prospective, multi-centre, UK study. Br J Ophthalmol. 2014; 98: 1042– 1049. 24682180
  36. [38]Brown MM, Brown GC, Sharma S, Landy J Health care economic analyses and value-based medicine. Surv Ophthalmol. 2003; 48: 204– 223. 12686305
  37. [39]Drummond MF, Jefferson TO Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. BMJ. 1996; 313: 275– 283. 8704542 PMC2351717
  38. [40]Jakubiak-Lasocka J, Jakubczyk M Cost-effectiveness versus cost-utility analyses: what are the motives behind using each and how do their results differ?-A Polish example. Value Health Reg Issues. 2014; 4: 66– 74. 29702810
  39. [41]Gold MR, Siegel JE, Russell LB, Weinstein MC Cost-Effectiveness in Health and Medicine. New York, NY: Oxford University Press; 1996 Available at: https://books.google.com.sg/books?id=dazBueIX9L8C. Accessed January 19, 2020.
  40. [42]National Institute for Health and Care Excellence. Guide to the methods of technology appraisal 2013. Available at: https://www.ncbi.nlm.nih.gov/books/NBK395867/pdf/Bookshelf_NBK395867.pdf. Accessed 20 Dec 2019.. 27905712
  41. [44]Robinson R Costs and cost-minimisation analysis. BMJ. 1993; 307: 726– 728. 8401098 PMC1678700
  42. [45]Briggs AH, O'Brien BJ The death of cost-minimization analysis? Health Econ. 2001; 10: 179– 184. 11252048
  43. [47]Dakin H, Wordsworth S Cost-minimisation analysis versus cost-effectiveness analysis, revisited. Health Econ. 2013; 22: 22– 34. 22109960
  44. [48]Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW Methods for the Economic Evaluation of Health Care Programmes. 4th ed Oxford: Oxford: Oxford University Press; 2015: 445 Available at: https://pure.york.ac.uk/portal/en/publications/methods-for-the-economic-evaluation-of-health-careprogrammes(8f69bcee-cdac-44fa-871c-f821470df60a).html. Accessed January 19, 2020.
  45. [49]Cartwright WS Cost-benefit analysis of drug treatment services: review of the literature. J Ment Health Policy Econ. 2000; 3: 11– 26. 11967433
  46. [50]Xie Y, Nguyen DQ, Bellemo V, et al. Cost-effectiveness analysis of an artificial intelligence-assisted deep learning system implemented in the national tele-medicine diabetic retinopathy screening in Singapore. Invest Ophthalmol Vis Sci. 2019; 60: 5471.
  47. [51]Petticrew MP, Sowden AJ, Lister-Sharp D, Wright K False-negative results in screening programmes: systematic review of impact and implications. Health Technol Assess. 2000; 4: 1– 120. 10859208
  48. [53]Focal photocoagulation treatment of diabetic macular edema. Relationship of treatment effect to fluorescein angiographic and other retinal characteristics at baseline: ETDRS report no. 19. Early Treatment Diabetic Retinopathy Study Research Group. Arch Ophthalmol. 1995; 113: 1144– 1155. 7661748
  49. [54]Wong RL, Tsang CW, Wong DS, et al. Are we making good use of our public resources? The false-positive rate of screening by fundus photography for diabetic macular oedema. Hong Kong Med J. 2017; 23: 356– 364. 28684650
  50. [55]Petticrew M, Sowden A, Lister-Sharp D False-negative results in screening programs. Medical, psychological, and other implications. Int J Technol Assess Health Care. 2001; 17: 164– 170. 11446128
  51. [56]Kapetanakis VV, Rudnicka AR, Liew G, et al. A study of whether automated diabetic retinopathy image assessment could replace manual grading steps in the English National Screening Programme. J Med Screen. 2015; 22: 112– 118. 25742804
  52. [58]Shiell A, Hawe P, Gold L Complex interventions or complex systems? Implications for health economic evaluation. BMJ. 2008; 336: 1281– 1283. 18535071 PMC2413333
  53. [59]Byford S, Sefton T Economic evaluation of complex health and social care interventions. Natl Inst Econ Rev. 2003; 186: 98– 108.
  54. [61]Deidda M, Geue C, Kreif N, Dundas R, McIntosh E A framework for conducting economic evaluations alongside natural experiments. Soc Sci Med. 2019; 220: 353– 361. 30513485 PMC6323352
  55. [64]Tarpey K, Schaaf E, Lakhani U, Balcitis J A proactive risk avoidance system using failure mode and effects analysis for “same-name” physician orders. Jt Comm J Qual Patient Saf. 2010; 36: 461– 467. 21548507
  56. [65]Robinson DL, Heigham M, Clark J Using failure mode and effects analysis for safe administration of chemotherapy to hospitalized children with cancer. Jt Comm J Qual Patient Saf. 2006; 32: 161– 166. 16617947
  57. [66]Coles G, Fuller B, Nordquist K, Weissenberger S, Anderson L, DuBois B Three kinds of proactive risk analyses for health care. Jt Comm J Qual Patient Saf. 2010; 36: 365– 375. 20860243
  58. [67]Ashley L, Armitage G, Neary M, Hollingsworth G A practical guide to failure mode and effects analysis in health care: making the most of the team and its meetings. Jt Comm J Qual Patient Saf. 2010; 36: 351– 358. 20860241
  59. [68]Faiella G, Parand A, Franklin BD, et al. Expanding healthcare failure mode and effect analysis: a composite proactive risk analysis approach. Reliab Eng Syst Saf. 2018; 169: 117– 126.
  60. [70]Salmon PM, Cornelissen M, Trotter MJ Systems-based accident analysis methods: a comparison of Accimap, HFACS, and STAMP. Saf Sci. 2012; 50: 1158– 1170.
  61. [71]Chatzimichailidou MM, Ward J, Horberry T, Clarkson PJ A comparison of the bow-tie and STAMP approaches to reduce the risk of surgical instrument retention. Risk Anal. 2018; 38: 978– 990. 29105803
  62. [72]Krause J, Gulshan V, Rahimy E, et al. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology. 2018; 125: 1264– 1272. 29548646
  63. [73]McLeod RW, Bowie P Bowtie analysis as a prospective risk assessment technique in primary healthcare. Policy Pract Heal Saf. 2018; 16: 177– 193.
  64. [74]Kerckhoffs MC, van der Sluijs AF, Binnekade JM, Dongelmans DA Improving patient safety in the ICU by prospective identification of missing safety barriers using the bow-tie prospective risk analysis model. J Patient Saf. 2013; 9: 154– 159. 23965838
  65. [76]Cheung CY, Tang F, Ting DSW, Tan GSW, Wong TY Artificial intelligence in diabetic eye disease screening. Asia Pac J Ophthalmol (Phila). 2019; 8: 158– 164. 31016915
  66. [78]Yoo TK, Choi JY, Seo JG, Ramasubramanian B, Selvaperumal S, Kim DW The possibility of the combination of OCT and fundus images for improving the diagnostic accuracy of deep learning for age-related macular degeneration: a preliminary experiment. Med Biol Eng Comput. 2019; 57: 677– 687. 30349958
  67. [79]Medeiros FA, Jammal AA, Thompson AC From machine to machine: an OCT-trained deep learning algorithm for objective quantification of glaucomatous damage in fundus photographs. Ophthalmology. 2019; 126: 513– 521. 30578810 PMC6884092
  68. [80]Zhen Y, Wang L, Liu H, Zhang J, Pu J Performance assessment of the deep learning technologies in grading glaucoma severity. 2018:arXiv:1810.13376. · arXiv
  69. [81]Liu H, Li L, Wormstone IM, et al. Development and validation of a deep learning system to detect glaucomatous optic neuropathy using fundus photographs. JAMA Ophthalmol. 2019; 137: 1353– 1360. PMC6743057 31513266
  70. [82]Burlina PM, Joshi N, Pekala M, Pacheco KD, Freund DE, Bressler NM Automated grading of age-related macular degeneration from color fundus images using deep convolutional neural networks. JAMA Ophthalmol. 2017; 135: 1170– 1176. 28973096 PMC5710387
  71. [83]Burr JM, Mowatt G, Hernández R, et al. The clinical effectiveness and cost-effectiveness of screening for open angle glaucoma: a systematic review and economic evaluation. Health Technol Assess. 2007; 11:iii–iv, ix–x, 1– 190. 17927922
  72. [84]Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016; 316: 2402– 2410. 27898976
  73. [85]Keel S, Lee PY, Scheetz J, et al. Feasibility and patient acceptability of a novel artificial intelligence-based screening model for diabetic retinopathy at endocrinology outpatient services: a pilot study. Sci Rep. 2018; 8: 4330. 29531299 PMC5847544

Authors
Xie, Y., Gunasekeran, D. V., Balaskas, K., Keane, P. A., Sim, D. A., Bachmann, L. M., Macrae, C., & Ting, D. S. W.
Affiliations
  • Singapore National Eye Center (SG)
  • National University of Singapore (SG)
  • National Health Service (GB)
  • University of Zurich (CH)
  • University of Nottingham (GB)
  • Sun Yat-sen University (CN)
Corresponding author
Ting, Daniel S. W.
ORCID
Venue
Translational Vision Science & Technology
Publisher
Association for Research in Vision and Ophthalmology (ARVO)
Published
2020-04-13
Volume
9
Issue
2
Pages
22
ISSN
2164-2591 (ISSN-L)
DOI
10.1167/tvst.9.2.22
PMC
PMC7396187
OpenAlex
W3016951780
Cited by
59
References
93
Citations by year
2026: 3 · 2025: 8 · 2024: 7 · 2023: 7 · 2022: 14 · 2021: 15 · 2020: 4
FWCI
4.87
Type
journal article
Language
en
Open access
Licence
Locations
  • Translational Vision Science & Technology (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • UCL Discovery (University College London) (submittedVersion)
Funders
  • Wellcome Trust (213632/Z/18/Z)
  • National Institute for Health and Care Research (CS-2014-14-023, CL-2010-18-004)
  • Academy of Medical Sciences (AMS-SGCL6-Keane)
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Ophthalmology › Retinal Diseases and Treatments
Keywords
diabetic retinopathy screening, artificial intelligence, health economic evaluation, patient safety, deep learning, ophthalmology, false positives and false negatives, blindness prevention, resource allocation
MeSH terms
Artificial Intelligence, Cost-Benefit Analysis, Diabetes Mellitus, Diabetic Retinopathy, Humans, Mass Screening, Ophthalmology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review