CoChat
Sign Up
Cited by 270Retinal Imaging and AnalysisRetinal Diseases and TreatmentsOphthalmology and Visual Health Research

Predicting the risk of developing diabetic retinopathy using deep learning

Open access
Bora, Ashish, Balasubramanian, Siva, Babenko, Boris, Virmani, Sunny, Venugopalan, Subhashini, Mitani, Akinori, de Oliveira Marinho, Guilherme, Cuadros, Jorge +8 more·2021·The Lancet Digital Health
DOI 10.1016/s2589-7500(20)30250-8
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Diabetic retinopathy screening is instrumental to preventing blindness, but scaling up screening is challenging because of the increasing number of patients with all forms of diabetes. We aimed to create a deep-learning system to predict the risk of patients with diabetes developing diabetic retinopathy within 2 years. METHODS: We created and validated two versions of a deep-learning system to predict the development of diabetic retinopathy in patients with diabetes who had had teleretinal diabetic retinopathy screening in a primary care setting. The input for the two versions was either a set of three-field or one-field colour fundus photographs. Of the 575 431 eyes in the development set 28 899 had known outcomes, with the remaining 546 532 eyes used to augment the training process via multitask learning. Validation was done on one eye (selected at random) per patient from two datasets: an internal validation (from EyePACS, a teleretinal screening service in the USA) set of 3678 eyes with known outcomes and an external validation (from Thailand) set of 2345 eyes with known outcomes. FINDINGS: The three-field deep-learning system had an area under the receiver operating characteristic curve (AUC) of 0·79 (95% CI 0·77-0·81) in the internal validation set. Assessment of the external validation set-which contained only one-field colour fundus photographs-with the one-field deep-learning system gave an AUC of 0·70 (0·67-0·74). In the internal validation set, the AUC of available risk factors was 0·72 (0·68-0·76), which improved to 0·81 (0·77-0·84) after combining the deep-learning system with these risk factors (p<0·0001). In the external validation set, the corresponding AUC improved from 0·62 (0·58-0·66) to 0·71 (0·68-0·75; p<0·0001) following the addition of the deep-learning system to available risk factors. INTERPRETATION: The deep-learning systems predicted diabetic retinopathy development using colour fundus photographs, and the systems were independent of and more informative than available risk factors. Such a risk stratification tool might help to optimise screening intervals to reduce costs while improving vision-related outcomes. FUNDING: Google.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/s2589-7500(20)30250-8

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Retinal images were used to predict the incidence and progression of chronic kidney disease and diabetic retinopathy.

    “Recently, retinal images were used to predict the incidence and progression of chronic kidney disease (CKD)39 and diabetic retinopathy (DR).101,102”

    From Artificial intelligence in diabetes management: Advancements, opportunities, and challenges (2023) · cites this paper as [102]

    The Introduction (p. 5) says the system used color fundus photographs to predict whether eyes would develop diabetic retinopathy within two years, supporting the incidence part of the DR claim. It does not report predicting progression of existing DR; the Discussion (p. 11) describes progression prediction by another study. CKD is attached to a different citation in the citing sentence and is not attributed to this source. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Bora et al. reported that a CNN using retinal images and risk factors to predict risk of developing DR within 2 years achieved an AUC of 0.79 in the internal validation set and 0.70 in the external validation set with one-field color fundus photographs.

    “the three-field DL system had an AUC of 0.79 in the internal validation set; assessment of the external validation set, which contained only one-field color fundus photographs, with the one-field DL system gave an AUC of 0.70”

    From Artificial intelligence in diabetes management: Advancements, opportunities, and challenges (2023) · cites this paper as Bora et al.102

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. [1]Cheung N, Mitchell P, Wong TY. Diabetic retinopathy. The Lancet. 2010;376(9735):124-36. http://paperpile.com/b/km6Kx3/I7ep
  2. [2]Ogurtsova K, da Rocha Fernandes JD, Huang Y, Linnenkamp U, Guariguata L, Cho NH, et al. IDF Diabetes Atlas: Global estimates for the prevalence of diabetes for 2015 and 2040. Diabetes Res Clin Pract. 2017 Jun;128:40-50. http://paperpile.com/b/km6Kx3/Dq7F
  3. [3]Scanlon PH. Screening Intervals for Diabetic Retinopathy and Implications for Care. Curr Diab Rep. 2017 Sep 5;17(10):96. http://paperpile.com/b/km6Kx3/2RfE
  4. [5]Ting DSW, Cheung GCM, Wong TY. Diabetic retinopathy: global prevalence, major risk factors, screening practices and public health challenges: a review. Clin Experiment Ophthalmol. 2016 May;44(4):260-77. http://paperpile.com/b/km6Kx3/2cMT
  5. [6]Solomon SD, Chew E, Duh EJ, Sobrin L, Sun JK, VanderBeek BL, et al. Diabetic Retinopathy: A Position Statement by the American Diabetes Association. Diabetes Care. 2017;40(3):412-8. http://paperpile.com/b/km6Kx3/KrfS
  6. [9]Jenkins AJ, Joglekar MV, Hardikar AA, Keech AC, O'Neal DN, Januszewski AS. Biomarkers in Diabetic Retinopathy. Rev Diabet Stud. 2015 Aug 10;12(1-2):159-95. http://paperpile.com/b/km6Kx3/OX1v
  7. 50: risk factors for incidence and progression of retinopathy in Type II diabetes over 6 years from diagnosis. Diabetologia. 2001 Feb;44(2):156-63. http://paperpile.com/b/km6Kx3/9jyQ
  8. [12]Wilkinson CP, Ferris FL 3rd, Klein RE, Lee PP, Agardh CD, Davis M, et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology. 2003 Sep;110(9):1677-82. http://paperpile.com/b/km6Kx3/kzBY
  9. [14]Modjtahedi BS, Theophanous C, Chiu S, Luong TQ, Nguyen N, Fong DS. Two-Year Incidence of Retinal Intervention in Patients With Minimal or No Diabetic Retinopathy on Telemedicine Screening. JAMA Ophthalmology. 2019;137(4):445. http://paperpile.com/b/km6Kx3/82rX
  10. [15]Ruamviboonsuk P, Krause J, Chotcomwongse P, Sayres R, Raman R, Widner K, et al. Deep learning versus human graders for classifying diabetic retinopathy severity in a nationwide screening program. npj Digital Medicine. 2019;2(1). Available from: http://dx.doi.org/10.1038/s41746-019-0099-8 http://paperpile.com/b/km6Kx3/WyLf · DOI · Paper page
  11. [16]Grading Diabetic Retinopathy from Stereoscopic Color Fundus Photographs - An Extension of the Modified Airlie House Classification. Ophthalmology. 2020;127(4):S99-119. http://paperpile.com/b/km6Kx3/pDqR
  12. [17]Varadarajan AV, Bavishi P, Ruamviboonsuk P, Chotcomwongse P, Venugopalan S, Narayanaswamy A, et al. Predicting optical coherence tomography-derived diabetic macular edema grades from fundus photographs using deep learning. Nat Commun. 2020 Jan 8;11(1):130. http://paperpile.com/b/km6Kx3/Au39
  13. [18]Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the Inception Architecture for Computer Vision. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016. http://paperpile.com/b/km6Kx3/34ka google-research. google-research/tf-slim. GitHub. [cited 2020 Jun 17]. Available from: https://github.com/google-research/tf-slim
  14. [20][Sundararajan M, Taly A, Yan Q. Axiomatic Attribution for Deep Networks. 2017 [cited 2020 Jun 17]. Available from: http://arxiv.org/abs/1703.01365](http://paperpile.com/b/km6Kx3/4qJW) · arXiv
  15. [21][Springenberg JT, Dosovitskiy A, Brox T, Riedmiller M. Striving for Simplicity: The All Convolutional Net. 2014 [cited 2020 Jun 17]. Available from: http://arxiv.org/abs/1412.6806](http://paperpile.com/b/km6Kx3/N50i) · arXiv
  16. [23]McKinney SM, Sieniek M, Godbole V, Godwin J, Antropova N, Ashrafian H, et al. International evaluation of an AI system for breast cancer screening. Nature. 2020 Jan;577(7788):89-94. http://paperpile.com/b/km6Kx3/QJQp
  17. [24]Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017 Feb 2;542(7639):115-8. http://paperpile.com/b/km6Kx3/yi9p
  18. [25]Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M, Fenyö D, et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nature Medicine. 2018;24(10):1559-67. http://paperpile.com/b/km6Kx3/IEWY
  19. [26]Krause J, Gulshan V, Rahimy E, Karth P, Widner K, Corrado GS, et al. Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy. Ophthalmology. 2018 Aug;125(8):1264-72. http://paperpile.com/b/km6Kx3/Yvtg
  20. [27]Ting DSW, Cheung CY-L, Lim G, Tan GSW, Quang ND, Gan A, et al. Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With Diabetes. JAMA. 2017 Dec 12;318(22):2211-23. http://paperpile.com/b/km6Kx3/Z8CE
  21. [28]Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA. 2016 Dec 13;316(22):2402-10. http://paperpile.com/b/km6Kx3/qLpk
  22. [29]Natarajan S, Jain A, Krishnan R, Rogye A, Sivaprasad S. Diagnostic Accuracy of Community-Based Diabetic Retinopathy Screening With an Offline Artificial Intelligence System on a Smartphone. JAMA Ophthalmol. 2019 Aug 8; Available from: http://dx.doi.org/10.1001/jamaophthalmol.2019.2923 http://paperpile.com/b/km6Kx3/xINc · DOI
  23. [30]Gargeya R, Leng T. Automated Identification of Diabetic Retinopathy Using Deep Learning. Ophthalmology. 2017 Jul;124(7):962-9. http://paperpile.com/b/km6Kx3/vuVi
  24. [31]Abràmoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, et al. Improved Automated Detection of Diabetic Retinopathy on a Publicly Available Dataset Through Integration of Deep Learning. Invest Ophthalmol Vis Sci. 2016 Oct 1;57(13):5200-6. http://paperpile.com/b/km6Kx3/kXJl
  25. [32]Abràmoff MD, Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med. 2018 Aug 28;1:39. http://paperpile.com/b/km6Kx3/PaH1
  26. [33]Mitani A, Huang A, Venugopalan S, Corrado GS, Peng L, Webster DR, et al. Detection of anaemia from retinal fundus images via deep learning. Nat Biomed Eng. 2020 Jan;4(1):18-27. http://paperpile.com/b/km6Kx3/lqol
  27. Eng. 2018 Mar;2(3):158-64. http://paperpile.com/b/km6Kx3/wH3k
  28. [35]Medeiros FA, Jammal AA, Thompson AC. From Machine to Machine: An OCT-Trained Deep Learning Algorithm for Objective Quantification of Glaucomatous Damage in Fundus Photographs. Ophthalmology. 2019 Apr;126(4):513-21. http://paperpile.com/b/km6Kx3/BRXh
  29. [36]Phene S, Dunn RC, Hammel N, Liu Y, Krause J, Kitade N, et al. Deep Learning and Glaucoma Specialists: The Relative Importance of Optic Disc Features to Predict Glaucoma Referral in Fundus Photographs. Ophthalmology. 2019 Dec;126(12):1627-39. http://paperpile.com/b/km6Kx3/XNc3
  30. [37]Arcadu F, Benmansour F, Maunz A, Willis J, Haskova Z, Prunotto M. Deep learning algorithm predicts diabetic retinopathy progression in individual patients. NPJ Digit Med. 2019 Sep 20;2:92. http://paperpile.com/b/km6Kx3/utjY
  31. [38]Yan Q, Weeks DE, Xin H, Swaroop A, Chew EY, Huang H, et al. Deep-learning-based Prediction of Late Age-Related Macular Degeneration Progression. Nat Mach Intell. 2020 Feb;2(2):141-50. http://paperpile.com/b/km6Kx3/UcOT
  32. [39]Bhuiyan A, Wong TY, Ting DSW, Govindaiah A, Souied EH, Theodore Smith R. Artificial Intelligence to Stratify Severity of Age-Related Macular Degeneration (AMD) and Predict Risk of Progression to Late AMD. Translational Vision Science & Technology. 2020;9(2):25. http://paperpile.com/b/km6Kx3/M2wq
  33. [40]Yim J, Chopra R, Spitz T, Winkens J, Obika A, Kelly C, et al. Predicting conversion to wet age-related macular degeneration using deep learning. Nat Med. 2020 May 18; Available from: http://dx.doi.org/10.1038/s41591-020-0867-7 http://paperpile.com/b/km6Kx3/09gb · DOI · Paper page
  34. [42]Estil S, Steinarsson AÞ, Einarsson S, Aspelund T, Stefánsson E. Diabetic eye screening with variable screening intervals based on individual risk factors is safe and effective in ophthalmic practice. Acta Ophthalmol. 2020 Mar 25; Available from: https://doi.org/10.1111/aos.14425 http://paperpile.com/b/km6Kx3/pjlB · DOI
  35. [44]Bearse MA Jr, Adams AJ, Han Y, Schneck ME, Ng J, Bronson-Castain K, et al. A multifocal electroretinogram model predicting the development of diabetic retinopathy. Prog Retin Eye Res. 2006 Sep;25(5):425-48. http://paperpile.com/b/km6Kx3/1TMY
  36. [45]Rogers SL, Tikellis G, Cheung N, Tapp R, Shaw J, Zimmet PZ, et al. Retinal arteriolar caliber predicts incident retinopathy: the Australian Diabetes, Obesity and Lifestyle (AusDiab) study. Diabetes Care. 2008 Apr;31(4):761-3. http://paperpile.com/b/km6Kx3/LKF6
  37. [46]Rohan TE, Frost CD, Wald NJ. Prevention of blindness by screening for diabetic retinopathy: a quantitative assessment. BMJ. 1989 Nov 11;299(6709):1198-201. http://paperpile.com/b/km6Kx3/wQ6V
  38. [48]Zhao Y, Singh RP. The role of anti-vascular endothelial growth factor (anti-VEGF) in the management of proliferative diabetic retinopathy. Drugs Context. 2018 Aug 13;7:212532. http://paperpile.com/b/km6Kx3/uF3z
  39. [49]Lin DY, Blumenkranz MS, Brothers RJ, Grosvenor DM. The sensitivity and specificity of single-field nonmydriatic monochromatic digital fundus photography with remote image interpretation for diabetic retinopathy screening: a comparison with ophthalmoscopy and standardized mydriatic color photography. Am J Ophthalmol. 2002 Aug;134(2):204-13. http://paperpile.com/b/km6Kx3/25tH
  40. [52]Downing J, Bollyky J, Schneider J. Use of a Connected Glucose Meter and Certified Diabetes Educator Coaching to Decrease the Likelihood of Abnormal Blood Glucose Excursions: The Livongo for Diabetes Program. J Med Internet Res. 2017 Jul 11;19(7):e234. http://paperpile.com/b/km6Kx3/ixb2 tensorflow. tensorflow/models. GitHub. [cited 2020 Jun 17]. Available from: https://github.com/tensorflow/models
  41. [54][Kingma DP, Ba J. Adam: A Method for Stochastic Optimization. 2014 [cited 2020 Jun 17]. Available from: http://arxiv.org/abs/1412.6980](http://paperpile.com/b/km6Kx3/wXcC) · arXiv

Cited by

Sorted by citations

Authors
Bora, A., Balasubramanian, S., Babenko, B., Virmani, S., Venugopalan, S., Mitani, A., de Oliveira Marinho, G., Cuadros, J., Ruamviboonsuk, P., Corrado, G. S., Peng, L., Webster, D. R., Varadarajan, A. V., Hammel, N., Liu, Y., & Bavishi, P.
Affiliations
  • Google (United States) (US)
  • Rajavithi Hospital (TH)
Corresponding author
Hammel, Naama
ORCID
Venue
The Lancet Digital Health
Publisher
Elsevier BV
Published
2021-01
Volume
3
Issue
1
Pages
e10-e19
ISSN
2589-7500 (ISSN-L)
DOI
10.1016/s2589-7500(20)30250-8
OpenAlex
W3048737435
Cited by
270
References
1
Citations by year
2026: 35 · 2025: 52 · 2024: 58 · 2023: 60 · 2022: 45 · 2021: 17 · 2020: 2 · 2018: 1
FWCI
15.10
Type
journal article
Language
en
Open access
Licence
Locations
  • The Lancet Digital Health (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Google
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
  • Health Sciences › Medicine › Ophthalmology › Retinal Diseases and Treatments
  • Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
Keywords
diabetic retinopathy, risk prediction, deep learning, primary care, risk stratification, screening frequency, external validation, EyePACS, blindness prevention, Thailand, United States, multi-task learning, transfer learning, diabetic retinopathy screening, cost reduction, visual outcome, risk prediction model validation
MeSH terms
Deep Learning, Aged, Diabetic Retinopathy, Diagnostic Techniques, Ophthalmological, Female, Humans, Male, Middle Aged, Photography, Prognosis, ROC Curve, Reproducibility of Results, Risk Assessment, Area Under Curve, Kaplan-Meier Estimate

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review