In-Context Learning for Data-Efficient Diabetic Retinopathy Detection via Multimodal Foundation Models
Open accessAbstract
Objective: This study aims to evaluate whether in-context learning (ICL), a prompt-based learning mechanism enabling multimodal foundation models to rapidly adapt to new tasks without retraining or large annotated datasets, can achieve comparable diagnostic performance to domain-specific foundation models. Specifically, we use diabetic retinopathy (DR) detection as an exemplar task to probe if a multimodal foundation model (Google Gemini 1.5 Pro), employing ICL, can match the performance of a domain-specific model (RETFound) fine-tuned explicitly for DR detection from color fundus photographs (CFPs). Design: A cross-sectional study. Subjects: A retrospective, publicly available dataset (Indian Diabetic Retinopathy Image Dataset) comprising 516 CFPs collected at an eye clinic in India, featuring both healthy individuals and patients with DR. Methods: The images were dichotomized into 2 groups based on the presence or absence of any signs of DR. RETFound was fine-tuned for this binary classification task, while Gemini 1.5 Pro was assessed for it under zero-shot and few-shot prompting scenarios, with the latter incorporating random or k-nearest-neighbors-based sampling of a varying number of example images. For experiments, data were partitioned into training, validation, and test sets in a stratified manner, with the process repeated for 10-fold cross-validation. Main Outcome Measures: Performance was assessed via accuracy, F1 score, and expected calibration error of predictive probabilities. Statistical significance was evaluated using Wilcoxon tests. Results: = 0.004). Conclusions: Gemini 1.5 Pro with ICL demonstrated performance comparable to RETFound for binary DR detection, illustrating how future medical artificial intelligence systems may build upon such frontier models rather than being bespoke solutions. Financial Disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Ayhan et al. employed the multimodal Gemini 1.5 Pro model with contextual learning to perform binary DR classification, achieving an accuracy of 0.841 (95% CI: 0.803-0.879) and an F1 score of 0.88 (95% CI: 0.844-0.909), comparable to RETFound, without requiring retraining or large-scale labeled datasets.
“Ayhan et al. [130] employed the multimodal Gemini 1.5 Pro model with contextual learning to perform binary DR classification, achieving an accuracy of 0.841 (95% CI: 0.803–0.879) and an F1 score of 0.88 (95% CI: 0.844–0.909), comparable to RETFound, without requiring retraining or large-scale labeled datasets.”
From Artificial intelligence-powered prediction of diabetic complications: from clinical data to molecular omics (2026) · cites this paper as [130]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · see citing works
Wilkinson, C. P., Ferris, F. L., Klein, R., Lee, P. P., Agardh, C., Davis, M., Dills, D. G., Kampik, A., Pararajasegaram, R., & Verdaguer, J. (2003) · Ophthalmology · doi:10.1016/s0161-6420(03)00475-5 · Cited by 3,786 · see citing works
Ting, D. S. W., Cheung, C. Y.-L., Lim, G., Tan, G. S. W., Quang, N. D., Gan, A., Hamzah, H., Garcia-Franco, R., San Yeo, I. Y., Lee, S. Y., et al. (2017) · JAMA · doi:10.1001/jama.2017.18152 · Cited by 2,402 · see citing works
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · cited 1× · see citing works
“6 However, their deployment in specialized domains like medical imaging often requires resource-intensive fine-tuning on domain-specific datasets, limiting accessibility and scalability in all settings, not just in resource-constrained ones.”
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560 · see citing works
A foundation model for generalizable disease detection from retinal images
Zhou, Y., Chia, M. A., Wagner, S. K., Ayhan, M. S., Williamson, D. J., Struyven, R. R., Liu, T., Xu, M., Lozano, M. G., Woodward-Court, P., et al. (2023) · Nature · doi:10.1038/s41586-023-06555-x · Cited by 1,014 · see citing works
Self-citationPorwal, P., Pachade, S., Kamble, R. M., Kokare, M., Deshmukh, G., Sahasrabuddhe, V., & Mériaudeau, F. (2018) · Data · doi:10.3390/data3030025 · Cited by 932 · see citing works
On the ethics of algorithmic decision-making in healthcare
Grote, T., & Berens, P. (2020) · Journal of Medical Ethics · doi:10.1136/medethics-2019-105586 · Cited by 551 · see citing works
A multimodal generative AI copilot for human pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Zhao, M., Chow, A. K., Ikemura, K., Kim, A., Pouli, D., Patel, A., et al. (2024) · Nature · doi:10.1038/s41586-024-07618-3 · Cited by 461 · see citing works
IDRiD: Diabetic Retinopathy – Segmentation and Grading Challenge
Porwal, P., Pachade, S., Kokare, M., Deshmukh, G., Son, J., Bae, W., Liu, L., Wang, J., Liu, X., Gao, L., et al. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101561 · Cited by 377 · see citing works
In-context learning enables multimodal large language models to classify cancer pathology images
Ferber, D., Wölflein, G., Wiest, I. C., Ligero, M., Sainath, S., Laleh, N. G., Nahhas, O. S. M. E., Müller‐Franzes, G. A., Jäger, D., Truhn, D., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-51465-9 · Cited by 167 · see citing works
What does CLIP know about a red circle? Visual prompt engineering for VLMs
Shtedritski, A., Rupprecht, C., & Vedaldi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01101 · Cited by 142 · see citing works
Foundation models for fast, label-free detection of glioma infiltration
Kondepudi, A., Pekmezci, M., Hou, X., Scotford, K., Jiang, C., Rao, A. T., Harake, E. S., Chowdury, A., Al‐Holou, W. N., Wang, L., et al. (2024) · Nature · doi:10.1038/s41586-024-08169-3 · Cited by 119 · see citing works
Ayhan, M. S., Kuehlewein, L., Aliyeva, G., INHOFFEN, W., Ziemssen, F., & Berens, P. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101724 · Cited by 85 · see citing works
Self-citationFoundation models in ophthalmology: opportunities and challenges
Sevgi, M., Ruffell, E., Antaki, F., Chia, M. A., & Keane, P. A. (2024) · Current Opinion in Ophthalmology · doi:10.1097/icu.0000000000001091 · Cited by 26 · see citing works
Self-citationMultitask Learning for Activity Detection in Neovascular Age-Related Macular Degeneration
Ayhan, M. S., Faber, H., Kuehlewein, L., INHOFFEN, W., Aliyeva, G., Ziemssen, F., & Berens, P. (2023) · Translational Vision Science & Technology · doi:10.1167/tvst.12.4.12 · Cited by 13 · see citing works
Self-citationAyhan, M. S., Neubauer, J., Uzel, M. M., Gelişken, F., & Berens, P. (2024) · Scientific Reports · doi:10.1038/s41598-024-57798-1 · Cited by 13 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 16
- [5]Fallahpour A. Ma J. Munim A. MedRAX: Medical Reasoning Agent for Chest X-ray 2025 arXiv
- [8]Brown T.B. Mann B. Ryder N. Language Models are Few-Shot Learners 2020 arXiv
- [10]Ravi N. Gabeur V. Hu Y.T. SAM 2: Segment Anything in Images and Videos 2024 arXiv
- [11]Zhang Y. Zhou K. Liu Z. What makes good examples for visual in-context learning? Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS ’23 2023 NeurIPS Foundation 17773 17794
- [12]Foster T. Croitoru I. Dorfman R. Edlund C. Varsavsky T. Almazán J. Flexible Visual Prompts for In-Context Learning in Computer Vision 2023 arXiv
- [13]Bommasani R. Hudson D.A. Adeli E. On the Opportunities and Risks of Foundation Models 2022 arXiv
- [14]Lampinen A.K. Chan S.C.Y. Singh A.K. Shanahan M. The broader spectrum of in-context learning 2024 arXiv
- [15]DeepSeek A.I. Guo D. Yang D. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning 2025 arXiv
- [27]Guo C. Pleiss G. Sun Y. Weinberger K.Q. On Calibration of Modern Neural Networks 2017 arXiv
- [29]Loshchilov I. Hutter F. Decoupled Weight Decay Regularization 2019 arXiv
- [30]Loshchilov I. Hutter F. SGDR: Stochastic Gradient Descent with Warm Restarts 2017 arXiv
- [31]Holtzman A. Buys J. Du L. Forbes M. Choi Y. The Curious Case of Neural Text Degeneration 2020 arXiv
- [33]Shtedritski A. Rupprecht C. Vedaldi A. What does CLIP know about a red circle? Visual prompt engineering for VLMs 2023 arXiv
- [34]Foster T. Croitoru I. Dorfman R. Edlund C. Varsavsky T. Almazán J. Flexible visual prompts for in-context learning in computer vision 2023 arXiv
- [35]Zhang Y. Zhou K. Liu Z. What Makes Good Examples for Visual In-Context Learning? 2023 arXiv
- [36]Błasiok J. Nakkiran P. Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing 2023 arXiv
Cited by
Sorted by citations
Choi, J. Y., & Yoo, T. K. (2025) · Informatics and Health · doi:10.1016/j.infoh.2025.07.002 · Cited by 12 · see citing works
Xie, X., Wu, C., Huang, Z., Zhou, Y., Huang, J., Dao, F., Yan, D., Deng, K., Lyu, H., Ma, C., et al. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag083 · Cited by 4 · see citing works
“…adaptation to multiple downstream tasks, including the diagnosis and prognosis of vision-threatening eye diseases such as DR. Ayhan et al. [130] employed the multimodal Gemini 1.5 Pro model with contextual learning to perform binary DR classification, achieving an accuracy of 0.841 (95%…”
InfluentialNouyed, M. I., Al-Mamun, M., Adjeroh, D., & Hu, G. (2026) · Diagnostics · doi:10.3390/diagnostics16101504 · Cited by 2 · see citing works
Gallardo, J. A. A., Rodríguez, E. A., Rueda-Gómez, L., Forero, O. L. T., De-La-Hoz-Franco, E., Gamarra, M., Sierra-Carrillo, J., & Escorcia-Gutierrez, J. (2026) · iScience · doi:10.1016/j.isci.2026.116565 · Cited by 1 · see citing works
Ong, A. Y., Nguyen, Q., Barai, I., Engelmann, J., Antaki, F., Sevgi, M., Merle, D. A., Ju, L., Dow, E. R., Zhou, Y., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-03136-w
Self-citationExplicit Inclusion of Diabetes Mellitus Without Retinopathy Within Diabetic Retinopathy Prediction
Rashidisabet, H., Lim, J. I., Kazlauskas, A., & Yi, D. (2026) · Translational Vision Science & Technology · doi:10.1167/tvst.15.7.3
Artificial intelligence in diabetic retinopathy: from automated screening to risk-stratified care
Zhang, F., Shao, P., Wei, D., & Wang, C. (2026) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-026-07390-2
Mohammad, A., & Ramakrishnaiah, N. (2026) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2026.111892
Jun, H. W., Nam, D., Choi, J. Y., & Yoo, T. K. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.116609
Djulbegovic, M. B., Marshall, R., Antonietti, M., Gonzalez, D. J. T., Bair, H., Wibbelsman, T. D., Gutterman, Y. J., Zambrano, R., Raimondo, C. D., Wollstein, G., et al. (2026) · Ophthalmology Science · doi:10.1016/j.xops.2026.101446
Self-citation
- Authors
- Ayhan, M. S., Ong, A. Y., Ruffell, E., Wagner, S. K., Merle, D. A., & Keane, P. A.
- Affiliations
- Moorfields Eye Hospital NHS Foundation Trust (GB)
- Corresponding author
- Keane, Pearse A.
- Venue
- Ophthalmology Science
- Publisher
- Elsevier BV
- Published
- 2026-01
- Volume
- 6
- Issue
- 1
- Pages
- 100934
- ISSN
- 2666-9145 (ISSN-L)
- PMC
- PMC12548082
- OpenAlex
- W4413949768
- Cited by
- 7
- References
- 19
- Citations by year
- 2026: 7
- FWCI
- 4.03
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Ophthalmology Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- University College London (EP/S021612/1)
- Moorfields Eye Charity
- Engineering and Physical Sciences Research Council
- UK Research and Innovation (MR/T019050/1)
- Ministério da Educação e Ciência
- European Society of Retina Specialists
- EPSRC Centre for Doctoral Training in Smart Medical Imaging (EP/Y017803/1)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- in-context learning, diabetic retinopathy detection, multimodal foundation models, RETFound, color fundus images, few-shot prompting, zero-shot prompting, expected calibration error
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
