Validation of automated screening for referable diabetic retinopathy with the IDx‐DR device in the Hoorn Diabetes Care System
Open accessAbstract
Purpose: To increase the efficiency of retinal image grading, algorithms for automated grading have been developed, such as the IDx‐DR 2.0 device. We aimed to determine the ability of this device, incorporated in clinical work flow, to detect retinopathy in persons with type 2 diabetes. Methods: Retinal images of persons treated by the Hoorn Diabetes Care System (DCS) were graded by the IDx‐DR device and independently by three retinal specialists using the International Clinical Diabetic Retinopathy severity scale (ICDR) and EURODIAB criteria. Agreement between specialists was calculated. Results of the IDx‐DR device and experts were compared using sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV), distinguishing between referable diabetic retinopathy (RDR) and vision‐threatening retinopathy (VTDR). Area under the receiver operating characteristic curve (AUC) was calculated. Results: Of the included 1415 persons, 898 (63.5%) had images of sufficient quality according to the experts and the IDx‐DR device. Referable diabetic retinopathy (RDR) was diagnosed in 22 persons (2.4%) using EURODIAB and 73 persons (8.1%) using ICDR classification. Specific intergrader agreement ranged from 40% to 61%. Sensitivity, specificity, PPV and NPV of IDx‐DR to detect RDR were 91% (95% CI: 0.69–0.98), 84% (95% CI: 0.81–0.86), 12% (95% CI: 0.08–0.18) and 100% (95% CI: 0.99–1.00; EURODIAB) and 68% (95% CI: 0.56–0.79), 86% (95% CI: 0.84–0.88), 30% (95% CI: 0.24–0.38) and 97% (95% CI: 0.95–0.98; ICDR). The AUC was 0.94 (95% CI: 0.88–1.00; EURODIAB) and 0.87 (95% CI: 0.83–0.92; ICDR). For detection of VTDR, sensitivity was lower and specificity was higher compared to RDR. AUC's were comparable. Conclusion: Automated grading using the IDx‐DR device for RDR detection is a valid method and can be used in primary care, decreasing the demand on ophthalmologists.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multiple artificial intelligence algorithms have been developed and validated for diabetic retinopathy screening.
“Multiple artificial intelligence (AI) algorithms have been developed and validated for DR5–8 screening and it is the most frequently implemented deep learning application in prospective real-world studies9.”
From Referral uptake after diabetic retinopathy screening with artificial intelligence-assisted care pathways: a systematic review and meta-analysis (2026) · cites this paper as [7]
Artificial intelligence is an accurate screening tool for diabetic retinopathy.
“Artificial intelligence (AI) is an accurate screening tool for diabetic retinopathy (DR), the leading cause of blindness among working-aged adults.”
From Referral uptake after diabetic retinopathy screening with artificial intelligence-assisted care pathways: a systematic review and meta-analysis (2026) · cites this paper as [7]
Subsequent studies have further validated the accuracy, sensitivity, and specificity of this autonomous AI technology using ophthalmologists' reads as the reference standard.
“Subsequent studies have further validated the accuracy, sensitivity, and specificity of this autonomous AI technology, using ophthalmologists’ reads as the reference standard9–11.”
From Autonomous artificial intelligence for diabetic eye disease increases access and health equity in underserved populations (2024) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Global estimates of diabetes prevalence for 2013 and projections for 2035
Guariguata, L., Whiting, D. R., Hambleton, I., Beagley, J., Linnenkamp, U., & Shaw, J. E. (2014) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2013.11.002 · Cited by 4,642 · see citing works
Abràmoff, M. D., Lou, Y., Erginay, A., Clarida, W., Amelon, R., Folk, J. C., & Niemeijer, M. (2016) · Investigative Opthalmology & Visual Science · doi:10.1167/iovs.16-19964 · Cited by 1,101 · see citing works
Self-citationHansen, M. B., Abràmoff, M. D., Folk, J. C., Mathenge, W., Bastawrous, A., & Peto, T. (2015) · PLOS ONE · doi:10.1371/journal.pone.0139148 · Cited by 93 · see citing works
Self-citationChalk, D., Pitt, M., Vaidya, B., & Stein, K. (2012) · Diabetes Care · doi:10.2337/dc11-2282 · Cited by 48 · see citing works
Flagged first, then by citations
Research works printed but not linked: 25
- Abramoff MD & Suttorp‐Schulten MS (2005): Web‐based screening for diabetic retinopathy in a primary care population: the EyeCheck project. Telemed J E Health 11: 668– 674. 16430386
- Abramoff MD, Reinhardt JM, Russell SR, Folk JC, Mahajan VB, Niemeijer M & Quellec G (2010): Automated early detection of diabetic retinopathy. Ophthalmology 117: 1147– 1154. 20399502 PMC2881172
- Abramoff MD, Folk JC, Han DP et al. (2013): Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol 131: 351– 357. 23494039
- Aldington SJ, Kohner EM, Meuer S, Klein R & Sjolie AK (1995): Methodology for retinal photography and assessment of diabetic retinopathy: the EURODIAB IDDM complications study. Diabetologia 38: 437– 444. 7796984
- Christopher M, Moga DC, Russell SR, Folk JC, Scheetz T & Abramoff MD (2012): Validation of tablet‐based evaluation of color fundus images. Retina 32: 1629– 1635. 22495326 PMC3396716
- Friedenwald JS (1950): Diabetic retinopathy. Am J Ophthalmol 33: 1187– 1199. 15432555
- Hansen AB, Hartvig NV, Jensen MS, Borch‐Johnsen K, Lund‐Andersen H & Larsen M (2004): Diabetic retinopathy screening using digital non‐mydriatic fundus photography and automated image analysis. Acta Ophthalmol Scand 82: 666– 672. 15606461
- Hazin R, Colyer M, Lum F & Barazi MK (2011): Revisiting Diabetes 2000: challenges in establishing nationwide diabetic retinopathy prevention programs. Am J Ophthalmol 152: 723– 729. 21917235
- Helmchen LA, Lehmann HP & Abramoff MD (2014): Automated detection of retinal disease. Am J Manag Care 20: eSP48– eSP52. 25811819
- Hutchinson A, McIntosh A, Peters J, O'Keeffe C, Khunti K, Baker R & Booth A (2000): Effectiveness of screening and monitoring tests for diabetic retinopathy–a systematic review. Diabet Med 17: 495– 506. 10972578
- Lin DY, Blumenkranz MS, Brothers RJ & Grosvenor DM (2002): The sensitivity and specificity of single‐field nonmydriatic monochromatic digital fundus photography with remote image interpretation for diabetic retinopathy screening: a comparison with ophthalmoscopy and standardized mydriatic color photography. Am J Ophthalmol 134: 204– 213. 12140027
- Oke JL, Stratton IM, Aldington SJ, Stevens RJ & Scanlon PH (2016): The use of statistical methodology to determine the accuracy of grading within a diabetic retinopathy screening programme. Diabet Med 33: 896– 903. 26666463 PMC5019246
- Olafsdottir E & Stefansson E (2007): Biennial eye screening in patients with diabetes without retinopathy: 10‐year experience. Br J Ophthalmol 91: 1599– 1601. 17627978 PMC2095544
- Oliveira CM, Cristovao LM, Ribeiro ML & Abreu JR (2011): Improved automated screening of diabetic retinopathy. Ophthalmologica 226: 191– 197. 21865671
- Philip S, Fleming AD, Goatman KA et al. (2007): The efficacy of automated “disease/no disease” grading for diabetic retinopathy in a systematic screening programme. Br J Ophthalmol 91: 1512– 1517. 17504851 PMC2095421
- Quellec G & Abramoff MD (2014): Estimating maximal measurable performance for automated decision systems from the characteristics of the reference standard. Application to diabetic retinopathy screening. Conf Proc IEEE Eng Med Biol Soc 2014: 154– 157. 25569920
- Rodbard HW, Blonde L, Braithwaite SS et al. (2007): American Association of Clinical Endocrinologists medical guidelines for clinical practice for the management of diabetes mellitus. Endocr Pract 13(Suppl 1): 1– 68. 17613449
- Sabanayagam C, Yip W, Ting DS, Tan G & Wong TY (2016): Ten emerging trends in the epidemiology of diabetic retinopathy. Ophthalmic Epidemiol 23: 209– 222. 27355693
- Sallam A, Scanlon PH, Stratton IM, Jones V, Martin CN, Brelen M & Johnston RL (2011): Agreement and reasons for disagreement between photographic and hospital biomicroscopy grading of diabetic retinopathy. Diabet Med 28: 741– 746. 21342245
- Scanlon PH (2008): The English national screening programme for sight‐threatening diabetic retinopathy. J Med Screen 15: 1– 4. 18416946
- Valverde C, Garcia M, Hornero R & Lopez‐Galvez MI (2016): Automated detection of diabetic retinopathy in retinal images. Indian J Ophthalmol 64: 26– 32. 26953020 PMC4821117
- de Vet HC, Mokkink LB, Terwee CB, Hoekstra OS & Knol DL (2013): Clinicians are right not to like Cohen's kappa. BMJ 346: f2125. 23585065
- Wilkinson CP, Ferris FL III, Klein RE et al. (2003): Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology 110: 1677– 1682. 13129861
- Yau JW, Rogers SL, Kawasaki R et al. (2012): Global prevalence and major risk factors of diabetic retinopathy. Diabetes Care 35: 556– 564. 22301125 PMC3322721
- Zavrelova H, Hoekstra T, Alssema M et al. (2011): Progression and regression: distinct developmental patterns of diabetic retinopathy in patients with type 2 diabetes treated in the diabetes care system west‐friesland, the Netherlands. Diabetes Care 34: 867– 872. 21447662 PMC3064043
Cited by
Sorted by citations
Benjamens, S., Dhunnoo, P., & Meskó, B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00324-0 · Cited by 1,215 · see citing works
Sayres, R., Taly, A., Rahimy, E., Blumer, K., Coz, D., Hammel, N., Krause, J., Narayanaswamy, A., Rastegar, Z., Wu, D., et al. (2019) · Ophthalmology · doi:10.1016/j.ophtha.2018.11.016 · Cited by 418 · see citing works
Artificial intelligence for diabetic retinopathy screening: a review
Grzybowski, A., Brona, P., Lim, G., Ruamviboonsuk, P., Tan, G. S. W., Abramoff, M., & Ting, D. S. W. (2020) · Eye · doi:10.1038/s41433-019-0566-0 · Cited by 376 · see citing works
Bhaskaranand, M., Ramachandra, C., Bhat, S., Cuadros, J., Nittala, M. G., Sadda, S. R., & Solanki, K. (2019) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2019.0164 · Cited by 211 · see citing works
Verbraak, F. D., Abramoff, M. D., Bausch, G. C. F., Klaver, C., Nijpels, G., Schlingemann, R. O., & van der Heijden, A. A. (2019) · Diabetes Care · doi:10.2337/dc18-0148 · Cited by 115 · see citing works
Self-citationLim, G., Bellemo, V., Xie, Y., Lee, X. Q., Yip, M. Y. T., & Ting, D. S. W. (2020) · Eye and Vision · doi:10.1186/s40662-020-00182-7 · Cited by 96 · see citing works
KHAN, Z., GAIDHANE, A. M., SINGH, M., GANESAN, S., KAUR, M., SHARMA, G. C., RANI, P., SHARMA, R., THAPLIYAL, S., KUSHWAHA, M., et al. (2025) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2025.02.022 · Cited by 49 · see citing works
Dow, E. R., Khan, N. C., Chen, K. M., Mishra, K., Perera, C., Narala, R., Basina, M., Dang, J., Kim, M., Levine, M., et al. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100330 · Cited by 40 · see citing works
Huang, J. J., Channa, R., Wolf, R. M., Dong, Y., Liang, M., Wang, J., Abramoff, M. D., & Liu, T. Y. A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01197-3 · Cited by 39 · see citing works
Leigh, J. A., Sherrington, A., Barber, A. R. J., Turner, A. W., Kidd, M., Powell, J., & Pope, C. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02616-3 · Cited by 2 · see citing works
- Authors
- van der Heijden, A. A., Abramoff, M. D., Verbraak, F., van Hecke, M. V., Liem, A., & Nijpels, G.
- Affiliations
- Department of General Practice and Elderly Care Medicine VU University Medical Centre Amsterdam the Netherlands
- Department of Ophthalmology and Visual Sciences University of Iowa Hospital and Clinics Iowa City IA USA
- Amsterdam UMC Location Vrije Universiteit Amsterdam (NL)
- Department of Ophthalmology Elisabeth‐Tweestedenziekenhuis Tilburg the Netherlands
- University Medical Center Utrecht (NL)
- Amsterdam Public Health (NL)
- Venue
- Acta Ophthalmologica
- Publisher
- Wiley
- Published
- 2017-11-27
- Volume
- 96
- Issue
- 1
- Pages
- 63-68
- ISSN
- 0001-639X (ISSN-L)
- 1755-375X
- 1755-3768
- PMC
- PMC5814834
- OpenAlex
- W2768095311
- Cited by
- 296
- References
- 4
- Citations by year
- 2026: 27 · 2025: 32 · 2024: 42 · 2023: 42 · 2022: 49 · 2021: 39 · 2020: 32 · 2019: 26 · 2018: 7
- FWCI
- 13.64
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Acta Ophthalmologica (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Pure Amsterdam UMC (publishedVersion, cc-by)
- Amsterdam UMC (VU Amsterdam) - Institutional Repository (publishedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- IDx LLC, Iowa City, IA, USA.
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Health Sciences › Medicine › Ophthalmology › Retinal Diseases and Treatments
- Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
- Keywords
- referable diabetic retinopathy, vision-threatening diabetic retinopathy, type 2 diabetes, retinal image quality, intergrader agreement, diagnostic accuracy, clinical workflow integration
- MeSH terms
- Aged, Algorithms, Diabetes Mellitus, Type 2, Diabetic Retinopathy, Diagnostic Techniques, Ophthalmological, Equipment Design, Female, Humans, Image Interpretation, Computer-Assisted, Male, Mass Screening, Netherlands, ROC Curve, Incidence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
