ICGA-GPT: report generation and question answering for indocyanine green angiography images
The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Background: Indocyanine green angiography (ICGA) is vital for diagnosing chorioretinal diseases, but its interpretation and patient communication require extensive expertise and time-consuming efforts. We aim to develop a bilingual ICGA report generation and question-answering (QA) system. Methods: Our dataset comprised 213 129 ICGA images from 2919 participants. The system comprised two stages: image–text alignment for report generation by a multimodal transformer architecture, and large language model (LLM)-based QA with ICGA text reports and human-input questions. Performance was assessed using both qualitative metrics (including Bilingual Evaluation Understudy (BLEU), Consensus-based Image Description Evaluation (CIDEr), Recall-Oriented Understudy for Gisting Evaluation-Longest Common Subsequence (ROUGE-L), Semantic Propositional Image Caption Evaluation (SPICE), accuracy, sensitivity, specificity, precision and F1 score) and subjective evaluation by three experienced ophthalmologists using 5-point scales (5 refers to high quality). Results: We produced 8757 ICGA reports covering 39 disease-related conditions after bilingual translation (66.7% English, 33.3% Chinese). The ICGA-GPT model’s report generation performance was evaluated with BLEU scores (1–4) of 0.48, 0.44, 0.40 and 0.37; CIDEr of 0.82; ROUGE of 0.41 and SPICE of 0.18. For disease-based metrics, the average specificity, accuracy, precision, sensitivity and F1 score were 0.98, 0.94, 0.70, 0.68 and 0.64, respectively. Assessing the quality of 50 images (100 reports), three ophthalmologists achieved substantial agreement (kappa=0.723 for completeness, kappa=0.738 for accuracy), yielding scores from 3.20 to 3.55. In an interactive QA scenario involving 100 generated answers, the ophthalmologists provided scores of 4.24, 4.22 and 4.10, displaying good consistency (kappa=0.779). Conclusion: This pioneering study introduces the ICGA-GPT model for report generation and interactive QA for the first time, underscoring the potential of LLMs in assisting with automated ICGA image interpretation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“It consists of two main components, visual transformer 17 as the image encoder and Bidi-rectional Encoder Representations from Transformers 18 as the language encoder and decoder.”
MethodAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 1×
“It consists of two main components, visual transformer 17 as the image encoder and Bidi-rectional Encoder Representations from Transformers 18 as the language encoder and decoder.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 1×
“…using the general quantitative natural language generation (NLG) metrics, including BLEU, 20 Consensus-based Image Description Evaluation (CIDEr), 21 Recall-Oriented Under-study for Gisting Evaluation-Longest Common Subsequence (ROUGE-L) 22 and Semantic Propositional Image Caption Evaluation…”
MethodLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 2×
“25 To fully leverage the in-context learning and reasoning capabilities of LLMs without fine-tuning, we have integrated prompts to create an ICGA report-specific medical dialogue pipeline.”
BackgroundLarge Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · cited 2×
“25 To fully leverage the in-context learning and reasoning capabilities of LLMs without fine-tuning, we have integrated prompts to create an ICGA report-specific medical dialogue pipeline.”
BackgroundCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293 · cited 1×
“15 Specifically, we divided the dataset into the training set (data prior to June 2019), the validation set (data from June to September 2019) and the testing set (data from September to December 2019, serving as a form of external validation).”
MethodRajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 2×
“We further introduced the state- of- the-art open-source LLM Llama 2-7b 12 in the second step to explore its capability to read generated reports.”
BackgroundMethodSelf-citationA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · cited 1×
“11 These models possess powerful natural language processing and generation capabilities, enabling them to generate context-aware responses during interactions with users, potentially better meeting the needs of both patients and doctors.”
BackgroundA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090 · cited 1×
“11 These models possess powerful natural language processing and generation capabilities, enabling them to generate context-aware responses during interactions with users, potentially better meeting the needs of both patients and doctors.”
BackgroundSPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986
Transformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,296
Dave, T., Athaluri, S. A., & Singh, S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1169595 · Cited by 1,204 · cited 1×
“33 First, our ICGA-GPT model can serve as a vital auxiliary tool for ophthalmologists.”
MethodTowards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2024) · NEJM AI · doi:10.1056/aioa2300138 · Cited by 483
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.14334 · Cited by 483
Cross-modal Memory Networks for Radiology Report Generation
Chen, Z., Shen, Y., Song, Y., & Wan, X. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.459 · Cited by 369 · cited 1×
“10 27 In the realm of ophthalmic imaging, transformer-based models have been used in image-text conversion tasks.”
MethodIndocyanine Green Angiography: A Perspective on Use in the Clinical Setting
Yannuzzi, L. A. (2011) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2011.01.043 · Cited by 235
Measures of Interrater Agreement
Mandrekar, J. N. (2011) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e318200f983 · Cited by 221
Momenaei, B., Wakabayashi, T., Shahlaee, A., Durrani, A. F., Pandit, S. A., Wang, K., Mansour, H. A., Abishek, R. M., Xu, D., Sridhar, J., et al. (2023) · Ophthalmology Retina · doi:10.1016/j.oret.2023.05.022 · Cited by 178 · cited 1×
“19 For each report, we paired it with the prepared questions and inputted them into the Llama 2 model for testing.”
MethodInteractive computer-aided diagnosis on medical image using large language models
Wang, S., Zhao, Z., Ouyang, X., Liu, T., Wang, Q., & Shen, D. (2024) · Communications Engineering · doi:10.1038/s44172-024-00271-8 · Cited by 147
Self-citationShi, D., Lin, Z., Wang, W., Tan, Z., Shang, X., Zhang, X., Meng, W., Ge, Z., & He, M. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.823436 · Cited by 87 · cited 1×
“We excluded low-quality ICGA images by extracting vessels, 14 where images with detectable vessel ratios less than 0.01 were excluded.”
MethodInfluentialSelf-citationFew-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation
Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., & Elazar, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.16938 · Cited by 81 · cited 1×
“for using LLMs to solve medical issues but comes with certain limitations, including potential performance reduction in other unfine-tuned domains, 30 the need for significant computational resources 31 and the lack of high-quality medical dialogue data.”
MethodFew-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation
Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., & Elazar, Y. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.779 · Cited by 79
Xu, P., Chen, X., Zhao, Z., & Shi, D. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-325054 · Cited by 63
Self-citationCross-modal Clinical Graph Transformer for Ophthalmic Report Generation
Li, M., Cai, W., Verspoor, K., Pan, S., Liang, X., & Chang, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02000 · Cited by 60
Chen, R., Zhang, W., Song, F., Yu, H., Cao, D., Zheng, Y., He, M., & Shi, D. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01018-7 · Cited by 57
Self-citationEffect of Human-AI Interaction on Detection of Malignant Lung Nodules on Chest Radiographs
Lee, J. H., Hong, H., Nam, G., Hwang, E. J., & Park, C. M. (2023) · Radiology · doi:10.1148/radiol.222976 · Cited by 53
Tong, W.-J., Wu, S.-H., Cheng, M.-Q., Huang, H., Liang, J.-Y., Li, C.-Q., Guo, H.-L., He, D.-N., Liu, Y.-H., Xiao, H., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.13674 · Cited by 52
Polypoidal Choroidal Vasculopathy: An Update on Diagnosis and Treatment
Sen, P., Manayath, G. J., Shroff, D., Salloju, V., & Dhar, P. (2023) · Clinical ophthalmology · doi:10.2147/opth.s385827 · Cited by 45
Expert-defined Keywords Improve Interpretability of Retinal Image Captioning
Wu, T.-W., Huang, J.-H., Lin, J. L., & Worring, M. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00190 · Cited by 41 · cited 1×
“Wu et al 28 developed a hybrid CNN–transformer system for retinal images.”
BackgroundLin, Z., Zhang, D., Shi, D., Xu, R., Tao, Q., Wu, L., He, M., & Ge, Z. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104281 · Cited by 39 · cited 1×
“29 In our study, we adopted the BLIP, 16 a pure trans-former architecture pretrained on a large image–text dataset.”
MethodSelf-citationYang, J., Zhang, C., Wang, E., Chen, Y., & Yu, W. (2019) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-019-04493-x · Cited by 31 · cited 2×
“We each selected a representative high-quality English report of CNV, drusen, PCV, AMD and lacquer crack, respectively, and input a series of prepared questions.”
InfluentialCui, T., Lin, D., Yu, S., Zhao, X., Lin, Z., Zhao, L., Xu, F., Yun, D., Pang, J., Li, R., et al. (2023) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2023.4650 · Cited by 27
Ma, L., Wang, Y., Wang, Y., Li, N., Fung, S.-F., Zhang, L., & Zheng, Q. (2021) · Complexity · doi:10.1155/2021/9981202 · Cited by 17
Xu, P., Chen, X., Zhao, Z., et al. (2023) · medRxiv · doi:10.1101/2023.11.27.23299056 · Cited by 11 · cited 1×
“13 The potential of extending LLMs to interpret ophthalmic images and address patient’s concerns remains unexplored.”
BackgroundSelf-citation
Research works printed but not linked: 1
- Lin C-Y . n.d. ROUGE: a package for automatic evaluation of summaries. Text Summarization Branches Out;2004:74–81.
Cited by
Large language models in patient education: a scoping review of applications in medicine
Aydin, S., Karabacak, M., Vlachos, V., & Margetis, K. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1477898 · Cited by 288
Large language models for disease diagnosis: a scoping review
Zhou, S., Xu, Z., Zhang, M., Xu, C., Guo, Y., Zhan, Z., Fang, Y., Ding, S., Wang, J., Xu, K., et al. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00011-z · Cited by 122
Evaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 68
Self-citationA multimodal visual–language foundation model for computational ophthalmology
Shi, D., Zhang, W., Yang, J., Huang, S., Chen, X., Xu, P., Jin, K., Lin, S. C., Wei, J., Yusufu, M., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01772-2 · Cited by 163
“The medical reports were thereforeconvertedintoa set of keywords covering various aspects such as ophthalmic diseases, anatomical structures, and diagnostic indicators 8,36 .”
Self-citationXu, P., Chen, X., Zhao, Z., & Shi, D. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-325054 · Cited by 63
Self-citation
Metadata
- Authors
- Chen, X., Zhang, W., Zhao, Z., Xu, P., Zheng, Y., Shi, D., & He, M.
- Affiliations
- Hong Kong Polytechnic University (HK)
- State Key Laboratory of Ophthalmology (CN)
- Corresponding author
- Shi, Danli
- Venue
- British Journal of Ophthalmology
- Publisher
- BMJ
- Published
- 2024-03-20
- Volume
- 108
- Issue
- 10
- Pages
- 1450-1456
- ISSN
- 0007-1161 (ISSN-L)
- 1468-2079
- OpenAlex
- W4392984569
- Cited by
- 36
- References
- 32
- Citations by year
- 2026: 3 · 2025: 24 · 2024: 9
- FWCI
- 5.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- British Journal of Ophthalmology (publishedVersion)
- PubMed (publishedVersion)
- PolyU Institutional Research Archive (Hong Kong Polytechnic University) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion)
- Funders
- Start-up Fund for RAPs under the Strategic Hiring Scheme (P0048623)
- Global STEM Professorship Scheme from HKSAR (P0046113)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Keywords
- report generation, question answering, indocyanine green angiography, large language models, multimodal transformer, bilingual translation, chorioretinal diseases, automated medical report generation, patient communication
- MeSH terms
- Aged, Coloring Agents, Female, Fluorescein Angiography, Fundus Oculi, Humans, Indocyanine Green, Male, Middle Aged, Retinal Diseases, Reproducibility of Results, Choroid Diseases
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.