Automated Early Detection of Diabetic Retinopathy

Abstract

Purpose To compare the performance of automated diabetic retinopathy (DR) detection, using the algorithm that won the 2009 Retinopathy Online Challenge Competition in 2009, (‘Challenge2009’) against that of the one currently used in EyeCheck, a large computer-aided early DR detection project. Design Evaluation of diagnostic test or technology. Participants Fundus photographic sets, consisting of two fundus images from each eye, were evaluated from 16,670 patient visits of 16,670 people with diabetes who had not previously been diagnosed with DR. Methods The fundus photographic set from each visit was analyzed by a single retinal expert; 793 of the 16,770 sets were classified as containing more than minimal DR (threshold for referral). The outcomes of the two algorithmic detectors were applied separately to the dataset and compared by standard statistical measures. Main Outcome Measures The area under the Receiver Operating Characteristic curve (AUC), a measure of the sensitivity and specificity of DR detection. Results Agreement was high, and exams containing more than minimal DR were detected with an AUC of 0.839 by the ‘Eyecheck’ algorithm and an AUC of 0.821 for ‘Challenge2009’, a statistically non-significant difference (z-score 1.91). If either of the algorithms detected DR in combination, AUC for detection was 0.86, the same as the theoretically expected maximum. At 90% sensitivity, the specificity of the ‘EyeCheck’ algorithm was 47.7% and the ‘Challenge2009’ algorithm, 43.6%. Conclusions DR detection algorithms appear to be maturing, and further improvements in detection performance cannot be differentiated from best clinical practices, because the performance of competitive algorithm development has now reached the human intra-reader variability limit. Additional validation studies on larger, well-defined, but more diverse populations of patients with diabetes are urgently needed, anticipating cost-effective early detection of DR in millions of people with diabetes to triage those patients who need further care at a time when they have early rather than advanced DR.