Impact of Artificial Intelligence Assessment of Diabetic Retinopathy on Referral Service Uptake in a Low-Resource Setting

Studies comparing AI to routine referral approaches with lower baseline rates (18.7-22.0%) showed the largest effect sizes (+36.7% to +42.0%), while studies comparing AI to existing manual screening workflows with higher baseline rates (39.6-77.3%) showed more modest improvements (+11.8% to +19.4%).

“Studies comparing AI to routine referral approaches with typically lower baseline rates of uptake (18.7–22.0%) showed the largest effect sizes (+36.7% to +42.0%)13,17, while studies comparing AI to existing manual screening workflows with higher baseline rates of uptake (39.6–77.3%) showed more modest improvements (+11.8% to +19.4%)14,16,18.”

Two studies that replaced routine referrals with targeted, AI-assisted referrals showed greater effect sizes than studies with a manual grading comparator.

“Firstly, comparator types whereby two studies replaced routine referrals with targeted, AI-assisted referrals13,17 showed greater effect sizes than those studies with a manual grading comparator14–16,18.”

The International Clinical Diabetic Retinopathy severity scale was the most commonly utilized severity scale, used in 15 out of 30 studies.

“Various severity scales are used to classify diabetic retinopathy (DR), with the International Clinical Diabetic Retinopathy (ICDR) severity scale being the most commonly utilized in 15 out of 30 studies [9, 11–13, 15, 18, 21–23, 32, 34–36].”

After AI interpretation, delayed result communication, unclear referral pathways, insufficient patient-facing communication, referral loss, or inadequate treatment capacity can prevent automated detection from becoming completed care.

“After AI interpretation, delayed result communication, unclear referral pathways, insufficient patientfacing communication, referral loss, or inadequate treatment capacity can prevent automated detection from becoming completed care (17)(18)(19)(20)(21)(22)(23).”

Real-world implementation of diabetic retinopathy screening with artificial intelligence-assisted care pathways has accelerated globally since 2020.

“Real-world implementation of DR screening with AI-assisted care pathways has accelerated globally since 20208,13–22, yet evidence on clinical and health system outcomes such as referral uptake remains limited.”

Studies comparing AI-assisted screening to existing manual grading workflows showed modest gains ranging from +11.8 to +23.5%.

“Conversely, studies comparing AI-assisted screening to existing manual grading workflows showed modest gains ranging from +11.8 to +23.5%14–16,18.”

In higher-performing systems (baseline referral uptake 39.6-77.3%), the gains from AI-assisted screening compared to manual grading were modest.

“This was most pronounced in in higher-performing systems (baseline referral uptake 39.6–77.3%)14,16,18 which indicates diminishing returns in settings with already functional screening infrastructure.”

In settings with pre-existing manual grading workflows, AI-assisted DR screening provided incremental optimisation rather than transformation.

“Conversely, in settings with pre-existing manual grading workflows, AI-assisted DR screening provided incremental optimisation rather than transformation”

Result timeliness alone modestly improves referral uptake through enhanced capability.

“demonstrates that result timeliness alone modestly improves uptake through enhanced capability14”

Baseline health system performance influenced the marginal gains, with settings with higher baseline rates of uptake experiencing relatively lower effect size, in contrast to lower-performing systems with greater margins for improvement.

“Thirdly, baseline health system performance appeared to influence the marginal gains, following a law of diminishing returns whereby settings with higher baseline rates of uptake experienced relatively lower effect size14,16,18, in contrast to lower-performing systems with greater margins for improvement13,15,17.”

Most included studies employed cross-sectional designs (n = 27), while three adopted clinical trials.

“Most employed cross-sectional designs (n = 27), while three adopted clinical trials [24, 29, 32].”

Among 14 studies conducted in tertiary hospitals or specialized clinics, most adopted a cross-sectional design; two employed randomized controlled trials and one was a validation study.

“Among the 14 studies conducted in tertiary hospitals or specialized clinics, most adopted a cross-sectional design. Two studies (Joseph et al. [24] and Mathenge et al. [32]) employed randomized controlled trials (RCTs), while Begum et al. [31] conducted a validation study.”

Recent advancements include CNN architectures like VGGNet and ResNet, achieving high AUCs for referable DR up to 0.97.

“Recent advancements include the use of CNN architectures like VGGNet [23] and ResNet [32], achieving high AUCs for referable DR up to 0.97 [21, 23].”

The RAIDERS randomized trial in Rwanda showed that AI assessment could improve referral service uptake in a low-resource setting when results and referral processes were embedded in an intervention rather than treated as a stand-alone test.

“The RAIDERS randomized trial in Rwanda showed that AI assessment could improve referral service uptake in a low-resource setting when results and referral processes were embedded in an intervention rather than treated as a stand-alone test (18).”

The included studies had two distinct status quo care pathways: routine referrals for people with diabetes without prior screening and manual grading workflows including asynchronous teleophthalmology.

“It should be noted there were two distinct, status quo care pathways: studies providing routine referrals for people with diabetes without prior screening13,17 and manual grading workflows including asynchronous teleophthalmology14–16,18.”

Both RCTs by Mathenge et al. and Wolf et al. were deemed low risk of bias as per RoB-2.

“Both RCTs by Mathenge et al.14 and Wolf et al.13 were deemed low risk of bias as per the RoB-2.”

Mathenge et al.'s RAIDERS RCT (n=275) found 51.5% versus 39.6% referral uptake for immediate AI result acquisition with colour-coded report versus delayed manual grading (3-5 days) with SMS and phone calls (+11.9%, OR 1.62, 95% CI 1.00-2.61, p=0.048).

“Mathenge et al.‘s RAIDERS RCT (n = 275) uniquely isolated the effect of immediate versus delayed feedback by randomising patients to immediate AI result acquisition with a colour-coded report based upon severity, or delayed manual grading (3–5 days) with SMS and phone calls to flag results, finding 51.5% versus 39.6% referral uptake respectively ( + 11.9%, OR 1.62, 95% CI: 1.00–2.61, p = 0.048)14.”

All studies consistently demonstrate improved referral uptake after AI-assisted DR screening.

“While all studies consistently demonstrate improved referral uptake after AI-assisted DR screening13–18, magnitude of benefit varies substantially from moderate improvements ( + 11.8%)16 to large effects ( + 42.0%)13.”

Findings from the review suggest the status quo for DR screening involves asynchronous teleophthalmology with delayed feedback of results or no prior screening as was the case in two North American studies by Wolf et al. and Liu et al.

“However, findings from our review suggest the status quo for DR screening involves asynchronous teleophthalmology with delayed feedback of results14–16,18 or no prior screening as was the case in two North American studies by Wolf et al. and Liu et al.13,17.”

In settings with pre-existing manual grading workflows, AI-assisted DR screening yielded modest improvements (+11.8-23.5%) despite higher baseline performance (39-77%), except Dow et al. (baseline 12%).

“yielding modest improvements ( + 11.8–23.5%)14–16,18 despite higher baseline performance (39–77%)14,16,18 save for Dow et al. (baseline of 12%)15.”

Mathenge et al. found that immediate versus delayed feedback after AI-assisted screening increased referral uptake by 11.9%.

“Mathenge et al.’s isolation of immediate versus delayed feedback after AI-assisted screening ( + 11.9% referral uptake)”

Two studies excluded ungradable images from analysis entirely, prior to allocation to AI or comparator arms.

“Two studies excluded ungradable images from analysis entirely, prior to allocation to AI or comparator arms14,18.”

Two studies compared AI-assisted targeted referrals to routine referral pathways for all people with diabetes, in contrast to the remaining four studies whose comparators included manual grading via asynchronous teleophthalmology.

“First, two studies compared AI-assisted targeted referrals to routine referral pathways for all people with diabetes13,17, in contrast to the remaining four studies whose comparators included manual grading via asynchronous teleophthalmology14–16,18.”