Cited by 40Retinal Imaging and AnalysisRetinal Diseases and TreatmentsOphthalmology and Visual Health Research

Artificial Intelligence-Human Hybrid Workflow Enhances Teleophthalmology for the Detection of Diabetic Retinopathy

Open access
Dow, Eliot R., Khan, Nergis C., Chen, Karen M., Mishra, Kapil, Perera, Chandrashan, Narala, Ramsudha, Basina, Marina, Dang, Jimmy +11 more·2023·Ophthalmology Science

Abstract

Objective: Detection of diabetic retinopathy (DR) outside of specialized eye care settings is an important means of access to vision-preserving health maintenance. Remote interpretation of fundus photographs acquired in a primary care or other nonophthalmic setting in a store-and-forward manner is a predominant paradigm of teleophthalmology screening programs. Artificial intelligence (AI)-based image interpretation offers an alternative means of DR detection. IDx-DR (Digital Diagnostics Inc) is a Food and Drug Administration-authorized autonomous testing device for DR. We evaluated the diagnostic performance of IDx-DR compared with human-based teleophthalmology over 2 and a half years. Additionally, we evaluated an AI-human hybrid workflow that combines AI-system evaluation with human expert-based assessment for referable cases. Design: Prospective cohort study and retrospective analysis. Participants: Diabetic patients ≥ 18 years old without a prior DR diagnosis or DR examination in the past year presenting for routine DR screening in a primary care clinic. Methods: Macula-centered and optic nerve-centered fundus photographs were evaluated by an AI algorithm followed by consensus-based overreading by retina specialists at the Stanford Ophthalmic Reading Center. Detection of more-than-mild diabetic retinopathy (MTMDR) was compared with in-person examination by a retina specialist. Main Outcome Measures: Sensitivity, specificity, accuracy, positive predictive value, and gradability achieved by the AI algorithm and retina specialists. Results: The AI algorithm had higher sensitivity (95.5% sensitivity; 95% confidence interval [CI], 86.7%-100%) but lower specificity (60.3% specificity; 95% CI, 47.7%-72.9%) for detection of MTMDR compared with remote image interpretation by retina specialists (69.5% sensitivity; 95% CI, 50.7%-88.3%; 96.9% specificity; 95% CI, 93.5%-100%). Gradability of encounters was also lower for the AI algorithm (62.5%) compared with retina specialists (93.1%). A 2-step AI-human hybrid workflow in which the AI algorithm initially rendered an assessment followed by overread by a retina specialist of MTMDR-positive encounters resulted in a sensitivity of 95.5% (95% CI, 86.7%-100%) and a specificity of 98.2% (95% CI, 94.6%-100%). Similarly, a 2-step overread by retina specialists of AI-ungradable encounters improved gradability from 63.5% to 95.6% of encounters. Conclusions: Implementation of an AI-human hybrid teleophthalmology workflow may both decrease reliance on human specialist effort and improve diagnostic accuracy. Financial Disclosures: Proprietary or commercial disclosure may be found after the references.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.xops.2023.100330

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In 2018, Health Canada approved EyeArt, allowing its sale in Canada.

    “In 2018, Health Canada approved EyeArt, allowing its sale in Canada.”

    From Autonomous Artificial Intelligence in Diabetic Retinopathy Testing—Lessons Learned on Successful Health System Adoption (2026) · cites this paper as [32]

    Our copy discusses U.S. FDA authorization of IDx-DR in the introductory passage following Financial Disclosure(s), but does not address EyeArt’s Canadian regulatory status. Searches for EyeArt, Health Canada, and Canadian approval found no passage stating that Health Canada approved EyeArt in 2018 or permitted its sale. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Dow, E. R., Khan, N. C., Chen, K. M., Mishra, K., Perera, C., Narala, R., Basina, M., Dang, J., Kim, M., Levine, M., Phadke, A., Tan, M., Weng, K., Do, D. V., Moshfeghi, D. M., Mahajan, V. B., Mruthyunjaya, P., Leng, T., & Myung, D.
Affiliations
  • VA Palo Alto Health Care System (US)
  • Stanford Medicine (US)
  • Stanford Health Care (US)
Corresponding author
Myung, David
Venue
Ophthalmology Science
Publisher
Elsevier BV
Published
2023-12
Volume
3
Issue
4
Pages
100330
ISSN
2666-9145 (ISSN-L)
OpenAlex
W4381707706
Cited by
40
References
29
Citations by year
2026: 17 · 2025: 14 · 2024: 7 · 2023: 2
FWCI
6.62
Type
journal article
Language
en
Open access
Locations
  • Ophthalmology Science (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Research to Prevent Blindness (NEI P30-EY026877)
  • University of Miami
  • Stanford Diabetes Research Center
  • National Eye Institute (P30-EY026877)
  • National Institutes of Health (R01 EY030151, R01 EY031952, R01EY031360, R01NS98950)
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
  • Health Sciences › Medicine › Ophthalmology › Retinal Diseases and Treatments
  • Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
Keywords
diabetic retinopathy, teleophthalmology, fundus photography, primary care screening, overreading, diagnostic accuracy, gradability, referable diabetic retinopathy, vision preservation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review