GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition
Abstract
In recent years, the growing utilization of medical imaging is placing an increasing burden on radiologists. Deep learning provides a promising solution for automatic medical image analysis and clinical decision support. However, large-scale manually labeled datasets required for training deep neural networks are difficult and expensive to obtain for medical images. The purpose of this work is to develop label-efficient multimodal medical imaging representations by leveraging radiology reports. We propose an attention-based framework for learning global and local representations by contrasting image sub-regions and words in the paired report. In addition, we propose methods to leverage the learned representations for various downstream medical image recognition tasks with limited labels. Our results demonstrate high-performance and label-efficiency for image-text retrieval, classification (finetuning and zerosshot settings), and segmentation on different datasets.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Although gathering such large datasets will pose a substantial challenge, these data will generally not require costly expert labels, given the success of self-supervision.
“Although gathering such large datasets will pose a substantial challenge, these data will generally not require costly expert labels, given the success of self-supervision 9,54 .”
From Foundation models for generalist medical artificial intelligence (2023) · cites this paper as [54]
Huang et al. demonstrated the model's ability to localize chest abnormalities on chest x-rays without segmentation labels and revealed the possibility of zero-shot learning by converting the classification classes into textual captions and framing the image classification task as measuring image-text similarity.
“By leveraging radiology reports, Huang et al. demonstrated the model’s ability to localize chest abnormalities on chest x-rays without segmentation labels and revealed the possibility of zero-shot learning by converting the classification classes into textual captions and framing the image classification task as measuring the image-text similarity107.”
From Self-supervised learning for medical image classification: a systematic review and implementation guidelines (2023) · cites this paper as [107]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
Self-citationDeep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Jain, S., & Wallace, B. C. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1357 · Cited by 2,287 · see citing works
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923 · see citing works
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Harmeling, S. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206594 · Cited by 1,849 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · see citing works
Stacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402 · see citing works
DenseCap: Fully Convolutional Localization Networks for Dense Captioning
Johnson, J., Karpathy, A., & Fei-Fei, L. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.494 · Cited by 1,161 · see citing works
Publicly Available Clinical BERT EmbeddingsPreprint
Alsentzer, E., Murphy, J. R., Boag, W., Weng, W., Jin, D., Naumann, T., & McDermott, M. B. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03323 · Cited by 1,040 · see citing works
VSE++: Improving Visual-Semantic Embeddings with Hard NegativesPreprint
Faghri, F., Fleet, D. J., Kiros, J., & Fidler, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05612 · Cited by 1,002 · see citing works
Zero-Shot Learning Through Cross-Modal Transfer
Socher, R., Ganjoo, M., Sridhar, H., Bastani, O., Manning, C. D., & Ng, A. Y. (2013) · arXiv · doi:10.48550/arxiv.1301.3666 · Cited by 997 · see citing works
Image Generation from Scene Graphs
Johnson, J., Gupta, A., & Fei-Fei, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00133 · Cited by 836 · see citing works
Visual Semantic Reasoning for Image-Text Matching
Li, K., Zhang, Y., Li, K., Li, Y., & Fu, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00475 · Cited by 628 · see citing works
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Hu, R., Andreas, J., Rohrbach, M., Darrell, T., & Saenko, K. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.93 · Cited by 523 · see citing works
Attention is not ExplanationPreprint
Jain, S., & Wallace, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10186 · Cited by 468 · see citing works
Contrastive Learning of Medical Visual Representations from Paired Images and TextPreprint
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., & Langlotz, C. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00747 · Cited by 465 · see citing works
IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval
Chen, H., Ding, G., Liu, X., Lin, Z., Liu, J., & Han, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01267 · Cited by 451 · see citing works
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392 · see citing works
Self-citationSimilarity Reasoning and Filtration for Image-Text Matching
Diao, H., Zhang, Y., Ma, L., & Lu, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16209 · Cited by 384 · see citing works
Learning Semantic Concepts and Order for Image and Sentence Matching
Huang, Y., Wu, Q., Song, C., & Wang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00645 · Cited by 353 · see citing works
VirTex: Learning Visual Representations from Textual Annotations
Desai, K., & Johnson, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01101 · Cited by 350 · see citing works
Zero-data learning of new tasks
Larochelle, H., Erhan, D., & Bengio, Y. (2008) · Cited by 345 · see citing works
Shih, G., Wu, C. C., Halabi, S. S., Kohli, M. D., Prevedello, L. M. S., Cook, T. S., Sharma, A., Amorosa, J. K., Arteaga, V. A., Galperin-Aizenberg, M., et al. (2019) · Radiology Artificial Intelligence · doi:10.1148/ryai.2019180041 · Cited by 341 · see citing works
MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs
Johnson, A. E. W., Pollard, T. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Peng, Y., Lu, Z., Mark, R. G., Berkowitz, S. J., & Horng, S. (2019) · arXiv · doi:10.48550/arxiv.1901.07042 · Cited by 269 · see citing works
Self-citationLearning to Count Objects in Natural Images for Visual Question Answering
Zhang, Y., Hare, J. S., & Prügel‐Bennett, A. (2018) · ePrints Soton (University of Southampton) · doi:10.48550/arxiv.1802.05766 · Cited by 194 · see citing works
Learning Cross-Modal Embeddings With Adversarial Networks for Cooking Recipes and Food Images
Wang, H., Sahoo, D., Liu, C., Lim, E., & Hoi, S. C. H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01184 · Cited by 144 · see citing works
Learning Visual Representations with Caption Annotations
Sarıyıldız, M. B., Pérez, J., & Larlus, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58598-3_10 · Cited by 120 · see citing works
Finding Beans in Burgers: Deep Semantic-Visual Embedding with Localization
Engilberge, M., Chevallier, L., Pérez, P., & Cord, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00419 · Cited by 108 · see citing works
Learning Visual N-Grams from Web Data
Li, A., Jabri, A., Joulin, A., & van der Maaten, L. J. P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.449 · Cited by 91 · see citing works
Li, Y., Wang, H., & Luo, Y. (2020) · 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313289 · Cited by 81 · see citing works
Zhang, Z., Chen, P., Sapkota, M., & Yang, L. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66179-7_37 · Cited by 71 · see citing works
Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment
Chauhan, G., Liao, R., Wells, W. M., Andreas, J., Wang, X., Berkowitz, S. J., Horng, S., Szolovits, P., & Golland, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59713-9_51 · Cited by 59 · see citing works
Cross-Modal Recipe Retrieval: How to Cook this Dish?
Chen, J., Pang, L., & Ngo, C. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-51811-4_48 · Cited by 55 · see citing works
End-to-End Convolutional Semantic Embeddings
You, Q., Luo, J., & Zhang, Z. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00601 · Cited by 42 · see citing works
Unsupervised Multimodal Representation Learning across Medical Images and ReportsPreprint
Hsu, T.-M. H., Weng, W., Boag, W., McDermott, M. B. A., & Peter Szolovits, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.08615 · Cited by 33 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · see citing works
A visual-language foundation model for computational pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L. P., Gerber, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02856-4 · Cited by 966 · see citing works
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text
Wang, Z., Wu, Z., Agarwal, D., & Sun, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.256 · Cited by 838 · see citing works
Huang, S.-C., Pareek, A., Jensen, M., Lungren, M. P., Yeung, S., & Chaudhari, A. S. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00811-0 · Cited by 479 · see citing works
Self-citationTiu, E., Talius, E., Patel, P., Langlotz, C. P., Ng, A. Y., & Rajpurkar, P. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00936-9 · Cited by 442 · see citing works
LViT: Language Meets Vision Transformer in Medical Image Segmentation
Li, Z., Li, Y., Li, Q., Wang, P., Guo, D., Lü, L., Jin, D., Zhang, Y., & Hong, Q. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3291719 · Cited by 398 · see citing works
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing
Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20059-5_1 · Cited by 317 · see citing works
“Related self-supervised VLP work [30,31,45,56,85] has achieved impressive downstream classification and zero-shot classification performance.”
BackgroundMethodResultInfluentialA Multimodal Biomedical Foundation Model Trained from Fifteen Million Image–Text Pairs
Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., et al. (2025) · NEJM AI · doi:10.1056/aioa2400640 · Cited by 283 · see citing works
Self-citationReview of multimodal machine learning approaches in healthcare
Krones, F., Marikkar, U., Parsons, G., Szmul, A., & Mahdi, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102690 · Cited by 235 · see citing works
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
Bannur, S., Hyland, S., Liu, Q., Pérez-García, F., Ilse, M., Castro, D. C., Boecking, B., Sharma, H., Bouzid, K., Thieme, A., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01442 · Cited by 227 · see citing works
Self-citationKnowledge-enhanced visual-language pre-training on chest radiology images
Zhang, X., Wu, C., Zhang, Y., Xie, W., & Wang, Y. (2023) · Nature Communications · doi:10.1038/s41467-023-40260-7 · Cited by 207 · see citing works
PMC-CLIP: Contrastive Language-Image Pre-training Using Biomedical Documents
Lin, W., Zhao, Z., Zhang, X., Wu, C., Zhang, Y., Wang, Y., & Xie, W. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43993-3_51 · Cited by 206 · see citing works
CLIP in medical imaging: A survey
Zhao, Z., Liu, Y., Wu, H., Wang, M., Li, Y., Wang, S., Teng, L., Liu, D., Cui, Z., Wang, Q., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103551 · Cited by 175 · see citing works
A medical multimodal large language model for future pandemics
Liu, F., Zhu, T., Wu, X., Yang, B., You, C., Wang, C., Lu, L., Liu, Z., Zheng, Y., Sun, X., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00952-2 · Cited by 155 · see citing works
Foundation Model for Advancing Healthcare: Challenges, Opportunities and Future Directions
He, Y., Huang, F., Jiang, X., Nie, Y., Wang, M., Wang, J., & Chen, H. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3496744 · Cited by 130 · see citing works
Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images
Lu, M. Y., Chen, B., Zhang, A., Williamson, D. F. K., Chen, R. J., Ding, T., Le, L. P., Chuang, Y.-S., & Mahmood, F. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01893 · Cited by 122 · see citing works
MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training for X-ray Diagnosis
Wu, C., Zhang, X., Zhang, Y., Wang, Y., & Xie, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01954 · Cited by 122 · see citing works
Self-Supervised Multimodal Learning: A Survey
Zong, Y., Aodha, O. M., & Hospedales, T. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429301 · Cited by 122 · see citing works
Silva-Rodríguez, J., Chakor, H., Kobbi, R., Dolz, J., & Ayed, I. B. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103357 · Cited by 121 · see citing works
Exploring scalable medical image encoders beyond text supervision
Pérez-García, F., Sharma, H., Bond-Taylor, S., Bouzid, K., Salvatelli, V., Ilse, M., Bannur, S., Castro, D. C., Schwaighofer, A., Lungren, M. P., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00965-w · Cited by 115 · see citing works
Self-citationA Comprehensive Survey of Foundation Models in Medicine
Khan, W., Leem, S., See, K. B., Wong, J. K., Zhang, S., & Fang, R. (2025) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2025.3531360 · Cited by 107 · see citing works
CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training
You, K., Gu, J., Ham, J., Park, B., Kim, J., Hong, E. K., Baek, W., & Roh, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43895-0_10 · Cited by 97 · see citing works
Pahud de Mortanges, A., Luo, H., Shu, S. Z., Kamath, A., Suter, Y., Shelan, M., Pöllinger, A., & Reyes, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01190-w · Cited by 93 · see citing works
Multimodal Artificial Intelligence in Medical Diagnostics
Jandoubi, B., & Akhloufi, M. A. (2025) · Information · doi:10.3390/info16070591 · Cited by 88 · see citing works
PRIOR: Prototype Representation Joint Learning from Medical Images and Reports
Cheng, P., Lin, L., Lyu, J., Huang, Y., Luo, W., & Tang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01953 · Cited by 83 · see citing works
MedKLIP: Medical Knowledge Enhanced Language-Image Pre-TrainingPreprint
Wu, C., Zhang, X., Zhang, Y., Wang, Y., & Xie, W. (2023) · medRxiv · doi:10.1101/2023.01.10.23284412 · Cited by 80 · see citing works
Cross-Modal Causal Representation Learning for Radiology Report Generation
Chen, W., Liu, Y., Wang, C., Zhu, J., Li, G., Liu, C., & Lin, L. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3568746 · Cited by 78 · see citing works
Has multimodal learning delivered universal intelligence in healthcare? A comprehensive survey
Lin, Q., Zhu, Y., Mei, X., Huang, L., Ma, J., He, K., Peng, Z., Cambria, E., & Feng, M. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102795 · Cited by 76 · see citing works
Advancing medical imaging with language models: featuring a spotlight on ChatGPT
Hu, M., Qian, J. Y., Pan, S., Li, Y., Qiu, R. L. J., & Yang, X. (2024) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ad387d · Cited by 74 · see citing works
Open challenges and opportunities in federated foundation models towards biomedical healthcare
Li, X., Peng, L., Wang, Y., & Zhang, W. (2025) · BioData Mining · doi:10.1186/s13040-024-00414-9 · Cited by 71 · see citing works
Buess, L., Keicher, M., Navab, N., Maier, A., & Arasteh, S. T. (2025) · Biomedical Engineering Letters · doi:10.1007/s13534-025-00497-1 · Cited by 71 · see citing works
A scoping review on multimodal deep learning in biomedical images and texts
Sun, Z., Lin, M., Zhu, Q., Xie, Q., Wang, F., Lu, Z., & Peng, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104482 · Cited by 69 · see citing works
Are Natural Domain Foundation Models Useful for Medical Image Classification?
Huix, J. P., Ganeshan, A. R., Haslum, J. F., Söderberg, M. P., Matsoukas, C., & Smith, K. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00746 · Cited by 63 · see citing works
Semi-Supervised Medical Report Generation via Graph-Guided Hybrid Feature Consistency
Zhang, K., Jiang, H., Zhang, J., Huang, Q., Jianping Fan, B., Yu, J., & Han, W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3273390 · Cited by 62 · see citing works
Huang, W., Li, C., Zhou, H.-Y., Yang, H., Liu, J., Liang, Y., Zheng, H., Zhang, S., & Wang, S. (2024) · Nature Communications · doi:10.1038/s41467-024-51749-0 · Cited by 62 · see citing works
Medical Visual Question Answering via Conditional Reasoning and Contrastive Learning
Liu, B., Zhan, L.-M., Xu, L., & Wu, X.-M. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3232411 · Cited by 61 · see citing works
Sun, K., Xue, S., Sun, F., Sun, H., Luo, Y., Wang, L., Wang, S., Guo, N., Liu, L., Zhao, T., et al. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103265 · Cited by 61 · see citing works
Medical image identification methods: A review
Li, J., Jiang, P., An, Q., Wang, G., & Kong, H. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107777 · Cited by 57 · see citing works
Multi-Grained Radiology Report Generation With Sentence-Level Image-Language Contrastive Learning
Liu, A., Guo, Y., Yong, J., & Xu, F. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3372638 · Cited by 55 · see citing works
Liu, C., Cheng, S. L., Chen, C., Qiao, M., Zhang, W., Shah, A., Bai, W., & Arcucci, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43907-0_61 · Cited by 54 · see citing works
Continual Learning for Abdominal Multi-organ and Tumor Segmentation
Zhang, Y., Li, X., Chen, H.-X., Yuille, A., Liu, Y., & Zhou, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43895-0_4 · Cited by 53 · see citing works
Improving Medical Vision-Language Contrastive Pretraining With Semantics-Aware Triage
Liu, B., Lu, D., Wei, D., Wu, X., Wang, Y., Zhang, Y., & Zheng, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3294980 · Cited by 52 · see citing works
Exploring low-resource medical image classification with weakly supervised prompt learning
Zheng, F., Cao, J., Yu, W., Chen, Z., Xiao, N., & Lu, Y. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110250 · Cited by 52 · see citing works
Multi-task paired masking with alignment modeling for medical vision-language pre-training
Zhang, K., Yang, Y., Yu, J., Jiang, H., Fan, J., Huang, Q., & Han, W. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3325965 · Cited by 51 · see citing works
Guo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 50 · see citing works
Scaling-up medical vision-and-language representation learning with federated learning
Lu, S., Liu, Z., Liu, T., & Zhou, W. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107037 · Cited by 47 · see citing works
Cross-Modal Conditioned Reconstruction for Language-Guided Medical Image Segmentation
Huang, X., Li, H., Cao, M., Chen, L., You, C., & An, D. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3523333 · Cited by 47 · see citing works
Ghosh, S., Poynton, C. B., Visweswaran, S., & Batmanghelich, K. N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_59 · Cited by 46 · see citing works
A multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis
Liu, F., Zheng, L., Yin, Q., Huang, J., Luo, J., Thakur, A., Branson, K., Schwab, P., Yin, B., Wu, X., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01339-7 · Cited by 43 · see citing works
Developing medical imaging AI for emerging infectious diseases
Huang, S.-C., Chaudhari, A. S., Langlotz, C. P., Shah, N. H., Yeung, S. Y., & Lungren, M. P. (2022) · Nature Communications · doi:10.1038/s41467-022-34234-4 · Cited by 42 · see citing works
Self-citationZhou, Y., Huang, S.-C., Fries, J., Youssef, A., Amrhein, T. J., Chang, M., Banerjee, I., Rubin, D. L., Xing, L., Shah, N. H., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.11665 · Cited by 40 · see citing works
Self-citationScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embedding
Li, Z., Zheng, Y., Luo, X., Shan, D., & Hong, Q. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612056 · Cited by 39 · see citing works
RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports
Du, J., Guo, J., Zhang, W., Yang, S., Liu, H., Li, H., & Wang, N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_66 · Cited by 37 · see citing works
Multimodal Fusion Learning with Dual Attention for Medical Imaging
Dhar, J., Zaidi, N. A., Haghighat, M., Roy, S. S., Goyal, P., Alavi, A., & Kumar, V. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00428 · Cited by 35 · see citing works
MITER: Medical Image–TExt joint adaptive pretRaining with multi-level contrastive learning
Shu, C., Zhu, Y., Tang, X., Xiao, J., Chen, Y., Li, X., Zhang, Q., & Lu, Z. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121526 · Cited by 34 · see citing works
Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts
Chen, Z., Diao, S., Wang, B., Li, G., & Wan, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02139 · Cited by 34 · see citing works
Residual-based Language Models are Free Boosters for Biomedical Imaging Tasks
Lai, Z., Wu, J., Chen, S., Zhou, Y., & Hovakimyan, N. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00515 · Cited by 34 · see citing works
CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification
Lai, H., Yao, Q., Jiang, Z., Wang, R., He, Z., Tao, X., & Zhou, S. K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01059 · Cited by 33 · see citing works
Xplainer: From X-Ray Observations to Explainable Zero-Shot Diagnosis
Pellegrini, C., Keicher, M., Özsoy, E., Jiraskova, P., Braren, R., & Navab, N. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43904-9_41 · Cited by 32 · see citing works
Anatomy-aware self-supervised learning for anomaly detection in chest radiographs
Sato, J., Suzuki, Y., Wataya, T., Nishigaki, D., Kita, K., Yamagata, K., Tomiyama, N., & Kido, S. (2023) · iScience · doi:10.1016/j.isci.2023.107086 · Cited by 31 · see citing works
Mapping medical image-text to a joint space via masked modeling
Chen, Z., Du, Y., Hu, J., Liu, Y., Li, G., Wan, X., & Chang, T. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103018 · Cited by 30 · see citing works
ConTEXTual Net: A Multimodal Vision-Language Model for Segmentation of Pneumothorax
Huemann, Z., Tie, X., Hu, J., & Bradshaw, T. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01051-8 · Cited by 30 · see citing works
IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-Training
Liu, C., Cheng, S. L., Shi, M., Shah, A., Bai, W., & Arcucci, R. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3449690 · Cited by 30 · see citing works
Advancements in Medical Radiology Through Multimodal Machine Learning: A Comprehensive Overview
Haq, I. U., Mhamed, M., Alharbi, M., Osman, H., Hamd, Z. Y., & Liu, Z. (2025) · Bioengineering · doi:10.3390/bioengineering12050477 · Cited by 30 · see citing works
Harnessing Text Insights With Visual Alignment for Medical Image Segmentation
Zeng, Q., Luo, H., Lu, Z., Xie, Y., Wang, Z., Zhang, Y., & Xia, Y. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3601359 · Cited by 30 · see citing works
EVA-X: a foundation model for general chest x-ray analysis with self-supervised learning
Yao, J., Wang, X., Song, Y., Zhao, H., Ma, J., Chen, Y., Liu, W., & Wang, B. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02032-z · Cited by 30 · see citing works
Breaking with Fixed Set Pathology Recognition Through Report-Guided Contrastive Training
Seibold, C., Reiß, S., Sarfraz, M. S., Stiefelhagen, R., & Kleesiek, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_66 · Cited by 30 · see citing works
STPNet: Scale-Aware Text Prompt Network for Medical Image Segmentation
Shan, D., Li, Z., Li, Y., Li, Q., Tian, J., & Hong, Q. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3571672 · Cited by 29 · see citing works
Jang, J., Kyung, D., Kim, S. H., Lee, H., Bae, K., & Choi, E. (2024) · Scientific Reports · doi:10.1038/s41598-024-73695-z · Cited by 28 · see citing works
Multimodal Machine Learning Approaches in Predictive Healthcare Analytics: A Comprehensive Survey
Vavekanand, R., Kumar, T., Kumar, S., Kumar, G., & Laghari, A. A. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10560-4 · Cited by 29 · see citing works
ViLMedic: a framework for research at the intersection of vision and language in medical AI
Delbrouck, J.-B., Saab, K. K., Varma, M., Eyuboglu, S., Chambon, P., Dunnmon, J., Chaves, J. M. Z., Chaudhari, A. S., & Langlotz, C. P. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.3 · Cited by 28 · see citing works
Facing & mitigating common challenges when working with real-world data: The Data Learning Paradigm
Lever, J., Cheng, S. L., Quilodrán-Casas, C., Liu, C., Fan, H., Platt, R. W., Rakotoharisoa, A., Johnson, E., Li, S., Shang, Z., et al. (2025) · Journal of Computational Science · doi:10.1016/j.jocs.2024.102523 · Cited by 27 · see citing works
Rethinking masked image modelling for medical image representation
Xie, Y., Gu, L., Harada, T., Zhang, J., Xia, Y., & Wu, Q. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103304 · Cited by 26 · see citing works
Phan, V. M. H., Xie, Y., Qi, Y., Liu, L., Liu, L., Zhang, B., Liao, Z., Wu, Q., To, M., & Verjans, J. W. H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01092 · Cited by 26 · see citing works
Integrating language into medical visual recognition and reasoning: A survey
Lu, Y., & Wang, A. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103514 · Cited by 26 · see citing works
Medical Phrase Grounding with Region-Phrase Context Contrastive Alignment
Chen, Z., Zhou, Y., Tran, A., Zhao, J., Wan, L., Ooi, G. S. K., Cheng, L. T.-E., Thng, C. H., Xu, X., Liu, Y., et al. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43990-2_35 · Cited by 26 · see citing works
Beyond Medical Imaging - A Review of Multimodal Deep Learning in RadiologyPreprint
Heiliger, L., Sekuboyina, A., Menze, B. H., Egger, J., & Kleesiek, J. (2022) · doi:10.36227/techrxiv.19103432.v1 · Cited by 26 · see citing works
UniDCP: Unifying Multiple Medical Vision-Language Tasks via Dynamic Cross-Modal Learnable Prompts
Zhan, C., Zhang, Y., Lin, Y., Wang, G., & Wang, H. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3397191 · Cited by 25 · see citing works
Park, S. J., Lee, E. S., Shin, K. S., Lee, J. E., & Ye, J. C. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103021 · Cited by 25 · see citing works
Common Vision-Language Attention for Text-Guided Medical Image Segmentation of Pneumonia
Guo, Y., Zeng, X., Zeng, P., Fei, Y., Lu, W. S., Zhou, J., & Wang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72114-4_19 · Cited by 24 · see citing works
LVF: A language and vision fusion framework for tomato diseases segmentation
Hu, Y., Zhu, J., Zhou, G., He, M., Lv, M., Wang, J., Chen, A., Deng, J., & Jiang, Y. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109484 · Cited by 24 · see citing works
UniChest: Conquer-and-Divide Pre-Training for Multi-Source Chest X-Ray Classification
Dai, T., Zhang, R., Hong, F., Yao, J., Zhang, Y., & Wang, Y. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3381123 · Cited by 23 · see citing works
Visual-Textual Matching Attention for Lesion Segmentation in Chest Images
Bui, P.-N., Le, D.-T., & Choo, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72114-4_67 · Cited by 23 · see citing works
Hu, J., Li, Y., Sun, H., Song, Y., Zhang, C., Lin, L., & Chen, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_57 · Cited by 23 · see citing works
HERGen: Elevating Radiology Report Generation with Longitudinal Data
Wang, F., Du, S., & Yu, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73001-6_11 · Cited by 23 · see citing works
Sun, Y., Wen, X., Zhang, Y., Jin, L., Yang, C., Zhang, Q., Jiang, M. C., Xu, Z., Guo, W., Su, J., et al. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108870 · Cited by 23 · see citing works
Contrastive Masked Image-Text Modeling for Medical Visual Representation Learning
Chen, C., Zhong, A., Wu, D., Luo, J., & Li, Q. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43904-9_48 · Cited by 23 · see citing works
Turgut, Ö., Müller, P., Hager, P., Shit, S., Starck, S., Menten, M. J., Martens, E., & Rueckert, D. (2025) · Medical Image Analysis · doi:10.1016/j.media.2024.103451 · Cited by 22 · see citing works
Zhao, X., & Wang, X. Y. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107018 · Cited by 21 · see citing works
A systematic review of multi-modal large language models on domain-specific applications
Li, S., Wong, K. W., Wang, G., & Duong, T.-T. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11398-1 · Cited by 21 · see citing works
Automated image label extraction from radiology reports — A review
Pereira, S. C., Mendonça, A. M., Campilho, A., Sousa, P., & Lopes, C. T. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102814 · Cited by 21 · see citing works
He, J., Li, P., Liu, G., & Zhong, S. (2025) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi60581.2025.10981029 · Cited by 20 · see citing works
Yan, R., Zhang, X., Jiang, Z., Wang, B., Bian, X., Ren, F., & Zhou, S. K. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3611531 · Cited by 20 · see citing works
Anatomical Structure-Guided Medical Vision-Language Pre-training
Li, Q., Yan, X., Xu, J., Yuan, R., Zhang, Y., Feng, R., Shen, Q., Zhang, X., & Wang, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72120-5_8 · Cited by 20 · see citing works
Multimodal deep learning-based diagnostic model for BPPV
Lu, H., Mao, Y., Li, J., & Zhu, L. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02438-x · Cited by 19 · see citing works
A Multimodal Transfer Learning Approach Using PubMedCLIP for Medical Image Classification
Dao, H. N., Nguyen, T. D., Mugisha, C., & Paik, I. (2024) · IEEE Access · doi:10.1109/access.2024.3401777 · Cited by 19 · see citing works
Automatic Medical Report Generation: Methods and Applications
Li, G., Tahir, A. M., Zhang, D., Wang, Z. J., & Ward, R. K. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.20240044 · Cited by 19 · see citing works
Improving Medical Multi-modal Contrastive Learning with Expert Annotations
Kumar, Y., & Marttinen, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72661-3_27 · Cited by 19 · see citing works
AO-UOD: A Novel Paradigm for Underwater Object Detection Using Acousto–Optic Fusion
Yu, F., Xiao, F., Li, C., Cheng, E., & Yuan, F. (2025) · IEEE Journal of Oceanic Engineering · doi:10.1109/joe.2025.3529121 · Cited by 18 · see citing works
Coarse-to-Fine Covid-19 Segmentation via Vision-Language Alignment
Shan, D., Li, Z., Chen, W., Li, Q., Tian, J., & Hong, Q. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096683 · Cited by 18 · see citing works
- Authors
- Huang, S.-C., Shen, L., Lungren, M. P., & Yeung, S.
- Affiliations
- Stanford University (US)
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 3922-3931
- OpenAlex
- W3201906559
- Cited by
- 489
- References
- 41
- Citations by year
- 2026: 122 · 2025: 195 · 2024: 106 · 2023: 51 · 2022: 14 · 2021: 1
- FWCI
- 14.04
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal representation learning, label-efficient learning, medical image recognition, radiology reports, image-text retrieval, zero-shot classification, medical image segmentation, limited labeled data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
