Potential of multimodal large language models for data mining of medical images and free-text reports
Open accessAbstract
Medical images and radiology reports are essential for physicians to diagnose medical conditions . However, the vast diversity and cross-source heterogeneity inherent in these data have posed significant challenges to the generalizability of current data-mining methods for clinical decision-making. Recently, multimodal large language models (MLLMs), especially Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models, have revolutionized numerous domains, significantly impacting the medical field. In this study, we conducted a detailed evaluation of the performance of the Gemini series models (including Gemini-1.0-Pro-Vision, Gemini-1.5-Pro, and Gemini-1.5-Flash) and GPT series models (including GPT-4o, GPT-4-Turbo, and GPT-3.5-Turbo) across 14 medical datasets, covering 5 medical imaging categories (dermatology, radiology, dentistry , ophthalmology , and endoscopy) and 3 radiology report datasets. The investigated tasks encompass disease classification , lesion segmentation , anatomical localization, disease diagnosis, report generation, and lesion detection. Moreover, we also validated the performance of the Claude-3-Opus, Yi-Large, Yi-Large-Turbo, and LLaMA 3 models to gain a comprehensive understanding of the MLLM models in the medical field. Our experimental results demonstrated that Gemini-series models excelled in report generation and lesion detection but faces challenges in disease classification and anatomical localization. Conversely, GPT-series models exhibited proficiency in lesion segmentation and anatomical localization but encountered difficulties in disease diagnosis and lesion detection. Additionally, both the Gemini series and GPT series contain models that have demonstrated commendable generation efficiency. While both models hold promise in reducing physician workload, alleviating pressure on limited healthcare resources, and fostering collaboration between clinical practitioners and artificial intelligence technologies, substantial enhancements and comprehensive validations remain imperative before clinical deployment.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Zhang et al.'s Gemini models produced R-1 scores of 0.2814, R-2 scores of 0.1334, and R-L scores of 0.2259 on the MIMIC-CXR dataset.
“Gemini models that the authors used in their work produced R-1 scores of 0.2814, R-2 scores of 0.1334, and R-L scores of 0.2259 on the MIMIC-CXR dataset.”
From Revolutionizing dermatopathology using AI in skin diagnostics: scoping review (2026) · cites this paper as [34]
Zhang et al. incorporated LLMs that gave R-1 0.2814, R-2 0.1334, and R-L 0.2259 on the MIMIC-CXR dataset.
“To enhance this system, Zhang et al. (34) incorporated large language models (LLMs) that gave R-1 scores of 0.2814, R-2 scores of 0.1334 and R-L scores of 0.2259 on the MIMIC-CXR dataset.”
From Revolutionizing dermatopathology using AI in skin diagnostics: scoping review (2026) · cites this paper as [34]
This version outperformed the previous one in lesion detection and reporting but requires more practice in different clinical environments.
“This version outperformed the previous one in lesion detection and reporting but requires more practice in different clinical environments.”
From Revolutionizing dermatopathology using AI in skin diagnostics: scoping review (2026) · cites this paper as [34]
Cites
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
Self-citationHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Self-citationLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Self-citationTschandl, P., Rosendahl, C., & Kittler, H. (2018) · Scientific Data · doi:10.1038/sdata.2018.161 · Cited by 3,628 · see citing works
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Self-citationLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Self-citationPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
GPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519 · see citing works
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428 · see citing works
Bernal, J., Sánchez, F. J., Fernández-Esparrach, G., Gil, D., Rodríguez, C., & Vilariño, F. (2015) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2015.02.007 · Cited by 2,191 · see citing works
Kvasir-SEG: A Segmented Polyp Dataset
Jha, D., Smedsrud, P., Riegler, M., Halvorsen, P., de Lange, T., Johansen, D., & Johansen, H. (2019) · Conference on Multimedia Modeling · doi:10.1007/978-3-030-37734-2_37 · Cited by 2,145 · see citing works
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
Self-citationA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Self-citationFoundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · see citing works
Can AI help in screening viral and COVID-19 pneumonia?
Chowdhury, M. E. H., Rahman, T., Khandakar, A., Mazhar, R., Kadir, M. A., Mahbub, Z. B., Islam, K. R., Khan, M. S., Iqbal, A., Emadi, N. A., et al. (2020) · IEEE Access · doi:10.1109/access.2020.3010287 · Cited by 1,913 · see citing works
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · see citing works
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Self-citationScaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., & Shum, H.-Y. (2022) · arXiv · Cited by 1,154 · see citing works
Exploring the effect of image enhancement techniques on COVID-19 detection using chest X-ray images
Rahman, T., Khandakar, A. A., Qiblawey, Y., Tahir, A., Serkan Kiranyaz, B., Kashem, S. B. A., Islam, M. T., Maadeed, S. A., Zughaier, S. M., Khan, M. S., et al. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104319 · Cited by 1,153 · see citing works
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · see citing works
Self-citationEndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos
Twinanda, A. P., Shehata, S., Mutter, D., Marescaux, J., de Mathelin, M., & Padoy, N. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2593957 · Cited by 1,116 · see citing works
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078 · see citing works
The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)Preprint
Yang, Z.-Y., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., & Wang, L.-J. (2023) · arXiv.org · doi:10.48550/arxiv.2309.17421 · Cited by 1,036 · see citing works
Self-citationGLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,028 · see citing works
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994 · see citing works
Self-citationA survey on multimodal large language models
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024) · National Science Review · doi:10.1093/nsr/nwae403 · Cited by 893 · see citing works
Self-citationSummary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855 · see citing works
Self-citationMedCLIP: Contrastive Learning from Unpaired Medical Images and Text
Wang, Z., Wu, Z., Agarwal, D., & Sun, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.256 · Cited by 838 · see citing works
Self-citationLabeled Optical Coherence Tomography (OCT) and Chest X-Ray Images for Classification
Kermany, D. (2018) · Data Archiving and Networked Services (DANS) · doi:10.17632/rscbjbr9sj.2 · Cited by 758 · see citing works
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737 · see citing works
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682 · see citing works
Explainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681 · see citing works
A review: Deep learning for medical image segmentation using multi-modality fusion
Zhou, T., Ruan, S., & Canu, S. (2019) · Array · doi:10.1016/j.array.2019.100004 · Cited by 665 · see citing works
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 626 · see citing works
A patient-centric dataset of images and metadata for identifying melanomas using clinical context
Rotemberg, V. M., Kurtansky, N. R., Betz‐Stablein, B., Caffery, L. J., Chousakos, E., Codella, N., Combalia, M., Dusza, S. W., Guitera, P., Gutman, D. A., et al. (2021) · Scientific Data · doi:10.1038/s41597-021-00815-z · Cited by 558 · see citing works
MMBench: Is Your Multi-modal Model an All-around Player?
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.06281 · Cited by 487 · see citing works
Self-citationMiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469 · see citing works
Tiu, E., Talius, E., Patel, P., Langlotz, C. P., Ng, A. Y., & Rajpurkar, P. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00936-9 · Cited by 442 · see citing works
CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection
Liu, J., Zhang, Y., Chen, J.-N., Xiao, J., Lu, Y., Landman, B. A., Yuan, Y., Yuille, A., Tang, Y., & Zhou, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01934 · Cited by 347 · see citing works
Self-citationTowards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338 · see citing works
Self-citationChatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture
Cui, J., Ning, M., Li, Z., Li, H., Yang, Y., Chen, B., Ling, B., Tian, Y., & Yuan, L. (2026) · Fundamental Research · doi:10.1016/j.fmre.2026.03.026 · Cited by 309 · see citing works
Review of large vision models and visual prompt engineering
Wang, J., Liu, Z., Zhao, L., Wu, Z., Ma, C., Yu, S., Dai, H., Yang, Q., Liu, Y., Zhang, S., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100047 · Cited by 231 · see citing works
Self-citationCOVID-19 infection localization and severity grading from chest X-ray images
Tahir, A., Chowdhury, M. E. H., Khandakar, A. A., Rahman, T., Qiblawey, Y., Khurshid, U., Serkan Kiranyaz, B., Ibtehaz, N., Rahman, M., Al-Maadeed, S. A., et al. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.105002 · Cited by 206 · see citing works
PMC-CLIP: Contrastive Language-Image Pre-training Using Biomedical Documents
Lin, W., Zhao, Z., Zhang, X., Wu, C., Zhang, Y., Wang, Y., & Xie, W. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43993-3_51 · Cited by 206 · see citing works
Self-citationUL2: Unifying Language Learning ParadigmsPreprint
Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Shakeri, S., Bahri, D., Schuster, T., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05131 · Cited by 194 · see citing works
Wang, Y.-Q., Chen, W.-T., Han, X.-T., Lin, X., Zhao, H., Liu, Y., Zhai, B.-H., Yuan, J., You, Q., & Yang, H. (2024) · arXiv.org · doi:10.48550/arxiv.2401.06805 · Cited by 188 · see citing works
Self-citationPeng, B., Li, C., He, P., Galley, M., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2304.03277 · Cited by 183 · see citing works
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Xu, L., Yang, J., Zheng, X., Li, K., Sun, X. W., et al. (2025) · neural information processing systems · doi:10.52202/085713-4899 · Cited by 148 · see citing works
Aligning Large Multimodal Models with Factually Augmented RLHF
Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L.-Y., Wang, Y.-X., Yang, Y., et al. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.775 · Cited by 146 · see citing works
Shi, P., Qiu, J., Abaxi, S. M. D., et al. (2023) · Diagnostics · doi:10.3390/diagnostics13111947 · Cited by 145 · see citing works
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning
Xu, R., Luo, F., Zhang, Z., Tan, C., Chang, B., Huang, S., & Huang, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.749 · Cited by 137 · see citing works
Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., Sun, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.00849 · Cited by 117 · see citing works
Self-citationCapabilities of Gemini Models in Medicine
Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.18416 · Cited by 117 · see citing works
COVID-19 infection map generation and detection from chest X-ray images
Degerli, A., Ahishali, M., Yamaç, M., Serkan Kiranyaz, B., Chowdhury, M. E. H., Hameed, K., Hamid, T., Mazhar, R., & Gabbouj, M. (2021) · Health Information Science and Systems · doi:10.1007/s13755-021-00146-8 · Cited by 116 · see citing works
Multi-Source and Multi-modal Deep Network Embedding for Cross-Network Node Classification
Yang, H., He, H. X., Zhang, W., Wang, Y., & Lin, J. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3653304 · Cited by 112 · see citing works
Self-citationCheong, I., Xia, K., Feng, K. J. K., Chen, Q. Z., & Zhang, A. X. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3659048 · Cited by 91 · see citing works
Multimodality of AI for Education: Towards Artificial General IntelligencePreprint
Lee, G.-G., Shi, L.-H., Latif, E., Gao, Y.-Z., Bewersdorff, A., Nyaaba, M., Guo, S.-C., Wu, Z.-H., Liu, Z., Wang, H., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2312.06037 · Cited by 84 · see citing works
Self-citationChatCAD+: Toward a Universal and Reliable Interactive CAD Using LLMs
Zhao, Z., Wang, S., Gu, J., Zhu, Y., Mei, L., Zhuang, Z., Cui, Z., Wang, Q., & Shen, D. G. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3398350 · Cited by 83 · see citing works
Self-citationDoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task
Xiong, H., Wang, S., Zhu, Y., Zhao, Z., Liu, Y., Huang, L., Wang, Q., & Shen, D. (2023) · arXiv · doi:10.48550/arxiv.2304.01097 · Cited by 81 · see citing works
Self-citationMM-Vet: Evaluating Large Multimodal Models for Integrated CapabilitiesPreprint
Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., & Wang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.02490 · Cited by 74 · see citing works
Self-citationEmpowering Education with LLMs - The Next-Gen Interface and Content Generation
Moore, S., Tong, R. M., Singh, A., Liu, Z., Hu, X., Lu, Y., Liang, J., Cao, C., Khosravi, H., Denny, P. C., et al. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-36336-8_4 · Cited by 64 · see citing works
Self-citationCausal reasoning: Charting a revolutionary course for next-generation AI-native wireless networks
Thomas, C. K., Chaccour, C., Saad, W., Debbah, M., & Hong, C. S. (2024) · IEEE Vehicular Technology Magazine · doi:10.1109/mvt.2024.3359357 · Cited by 46 · see citing works
Surviving ChatGPT in healthcare
Liu, Z., Zhang, L., Wu, Z., Yu, X., Cao, C., Dai, H., Liu, N., Liu, J., Liu, W., Li, Q., et al. (2024) · Frontiers in Radiology · doi:10.3389/fradi.2023.1224682 · Cited by 39 · see citing works
Self-citationXplainer: From X-Ray Observations to Explainable Zero-Shot Diagnosis
Pellegrini, C., Keicher, M., Özsoy, E., Jiraskova, P., Braren, R., & Navab, N. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43904-9_41 · Cited by 32 · see citing works
Zhang, H., Shao, W., Liu, H., Ma, Y., Luo, P., Qiao, Y., Zheng, N., & Zhang, K. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3520306 · Cited by 32 · see citing works
Jang, J., Kyung, D., Kim, S. H., Lee, H., Bae, K., & Choi, E. (2024) · Scientific Reports · doi:10.1038/s41598-024-73695-z · Cited by 28 · see citing works
Investigating the Catastrophic Forgetting in Multimodal Large Language Models
Zhai, Y., Tong, S., Li, X., Cai, M., Qu, Q., Lee, Y. J., & Ma, Y. (2023) · arXiv · doi:10.48550/arxiv.2309.10313 · Cited by 24 · see citing works
An Empirical Study of Scaling Instruct-Tuned Large Multimodal ModelsPreprint
Lu, Y., Li, C., Liu, H., Yang, J., Gao, J., & Shen, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.09958 · Cited by 21 · see citing works
On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous DrivingPreprint
Wen, L., Yang, X., Fu, D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.05332 · Cited by 21 · see citing works
Zhang, L., Liu, Z., Zhang, L., et al. (2024) · Medical Physics · doi:10.1002/mp.16965 · Cited by 21 · see citing works
Self-citationX-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentation
van Sonsbeek, T., & Worring, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34048-2_36 · Cited by 20 · see citing works
Evaluating Large Language Models for Radiology Natural Language ProcessingPreprint
Liu, Z., Zhong, T., Li, Y., Zhang, Y., Pan, Y., Zhao, Z., Dong, P., Cao, C., Liu, Y., Shu, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.13693 · Cited by 20 · see citing works
Self-citationPrompt Tuning for Generative Multimodal Pretrained ModelsPreprint
Yang, H., Lin, J., An, Y., Wang, P., Zhou, C., & Yang, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.02532 · Cited by 18 · see citing works
m2caiSeg: Semantic Segmentation of Laparoscopic Images using Convolutional Neural NetworksPreprint
Maqbool, S., Riaz, A., Sajid, H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.10134 · Cited by 17 · see citing works
Zhong, T., Zhao, W., Zhang, Y., Pan, Y., Dong, P., Jiang, Z., Kui, X., Shang, Y., Li, Y., Wei, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.05242 · Cited by 12 · see citing works
Self-citationLLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law DatasetPreprint
Izzidien, A., Sargeant, H., & Steffek, F. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4746305 · Cited by 12 · see citing works
X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval AugmentationPreprint
van Sonsbeek, T., & Worring, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.11352 · Cited by 7 · see citing works
Exploring Transfer Learning in Medical Image Segmentation using Vision-Language ModelsPreprint
Poudel, K., Dhakal, M., Bhandari, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.07706 · Cited by 7 · see citing works
Ophtha-LLaMA2: A Large Language Model for OphthalmologyPreprint
Zhao, H., Ling, Q., Pan, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.04906 · Cited by 7 · see citing works
Self-citationMa, S., Zhao, Y., Kang, Y., et al. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3410934 · Cited by 7 · see citing works
SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal InstructionsPreprint
Horawalavithana, S., Munikoti, S., Stewart, I. J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.01139 · Cited by 6 · see citing works
Boosting Few-Shot Semantic Segmentation With Prior-Driven Edge Feature Enhancement Network
Ma, J., Bai, S., & Pan, W. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3474650 · Cited by 6 · see citing works
Mishra, A., Mittal, R., Jestin, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.08000 · Cited by 5 · see citing works
Wang, Y., Wu, J., Dabral, T. S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.12100 · Cited by 5 · see citing works
LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law DatasetPreprint
Izzidien, A., Sargeant, H., & Steffek, F. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.04791 · Cited by 4 · see citing works
Causal Reasoning: Charting a Revolutionary Course for Next-Generation AI-Native Wireless NetworksPreprint
Thomas, C. K., Chaccour, C., Saad, W., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.13223 · Cited by 2 · see citing works
Aligning and Prompting Everything All at Once for Universal Visual PerceptionPreprint
Shen, Y., Fu, C., Chen, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.02153 · Cited by 2 · see citing works
LLMs for Coding and Robotics EducationPreprint
Shu, P., Zhao, H., Jiang, H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.06116 · Cited by 1 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Flagged first, then by citations
Cited by
Sorted by citations
Revolutionizing dermatopathology using AI in skin diagnostics: scoping review
Rammal, R., Mohy U Din, A., & Alam, T. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1614681 · Cited by 1 · see citing works
“Zhang et al. (34) improved Dermacen Analytica by integrating large language models with machine learning methods.”
- Authors
- Zhang, Y., Pan, Y., Zhong, T., Dong, P., Xie, K., Liu, Y., Jiang, H., Wu, Z., Liu, Z., Zhao, W., Zhang, W., Zhao, S., Zhang, T., Jiang, X., Shen, D., Liu, T., & Zhang, X.
- Affiliations
- Northwestern Polytechnical University (CN)
- University of Electronic Science and Technology of China (CN)
- University of Georgia (US)
- ShanghaiTech University (CN)
- Corresponding author
- Zhang, Yutong
- Zhang, Wei
- Zhang, Tuo
- Zhang, Xin
- ORCID
- Venue
- Meta-Radiology
- Publisher
- Elsevier BV
- Published
- 2024-12
- Volume
- 2
- Issue
- 4
- Pages
- 100103
- ISSN
- 2950-1628 (ISSN-L)
- OpenAlex
- W4402701904
- Semantic Scholar
- ee2ad6649b8c1df092fa93965ba4c7daa497c2c0
- Cited by
- 52
- References
- 100
- Citations by year
- 2026: 17 · 2025: 32 · 2024: 3
- FWCI
- 7.28
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Meta-Radiology (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62376221, 31971288, 62131009, 62476291)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- multimodal large language models, medical image analysis, radiology reports, Gemini, GPT-4, disease classification, lesion segmentation, anatomical localization, report generation, lesion detection, dermatology, radiology, ophthalmology, endoscopy, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
