Cited by 493Multimodal Machine Learning ApplicationsImage Retrieval and Classification TechniquesDomain Adaptation and Few-Shot Learning

The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.

GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition

Huang, Shih-Cheng, Shen, Liyue, Lungren, Matthew P., Yeung, Serena·2021·2021 IEEE/CVF International Conference on Computer Vision (ICCV)

DOI 10.1109/iccv48922.2021.00391

Open this paper in CoChat — free account

Abstract

In recent years, the growing utilization of medical imaging is placing an increasing burden on radiologists. Deep learning provides a promising solution for automatic medical image analysis and clinical decision support. However, large-scale manually labeled datasets required for training deep neural networks are difficult and expensive to obtain for medical images. The purpose of this work is to develop label-efficient multimodal medical imaging representations by leveraging radiology reports. We propose an attention-based framework for learning global and local representations by contrasting image sub-regions and words in the paired report. In addition, we propose methods to leverage the learned representations for various downstream medical image recognition tasks with limited labels. Our results demonstrate high-performance and label-efficiency for image-text retrieval, classification (finetuning and zerosshot settings), and segmentation on different datasets.

Integrity signals

Registry

Crossref · 10.1109/iccv48922.2021.00391

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
Self-citation
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Harmeling, S. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206594 · Cited by 1,849

Show all 41 references

Research works printed but not linked: 1

frome, 2013, DeViSE A deep visual-semantic embedding model [J]

Sorted by citations

Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028
A visual-language foundation model for computational pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L. P., Gerber, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02856-4 · Cited by 966
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text
Wang, Z., Wu, Z., Agarwal, D., & Sun, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.256 · Cited by 1,011
Self-supervised learning for medical image classification: a systematic review and implementation guidelines
Huang, S.-C., Pareek, A., Jensen, M., Lungren, M. P., Yeung, S., & Chaudhari, A. S. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00811-0 · Cited by 479
Self-citation
Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning
Tiu, E., Talius, E., Patel, P., Langlotz, C. P., Ng, A. Y., & Rajpurkar, P. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00936-9 · Cited by 442
LViT: Language Meets Vision Transformer in Medical Image Segmentation
Li, Z., Li, Y., Li, Q., Wang, P., Guo, D., Lü, L., Jin, D., Zhang, Y., & Hong, Q. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3291719 · Cited by 416
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing
Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20059-5_1 · Cited by 317
“Related self-supervised VLP work [30,31,45,56,85] has achieved impressive downstream classification and zero-shot classification performance.”
BackgroundMethodResultInfluential
A Multimodal Biomedical Foundation Model Trained from Fifteen Million Image–Text Pairs
Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., et al. (2025) · NEJM AI · doi:10.1056/aioa2400640 · Cited by 283
Self-citation
Review of multimodal machine learning approaches in healthcare
Krones, F., Marikkar, U., Parsons, G., Szmul, A., & Mahdi, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102690 · Cited by 235
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
Bannur, S., Hyland, S., Liu, Q., Pérez-García, F., Ilse, M., Castro, D. C., Boecking, B., Sharma, H., Bouzid, K., Thieme, A., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01442 · Cited by 227
Self-citation
Knowledge-enhanced visual-language pre-training on chest radiology images
Zhang, X., Wu, C., Zhang, Y., Xie, W., & Wang, Y. (2023) · Nature Communications · doi:10.1038/s41467-023-40260-7 · Cited by 207
PMC-CLIP: Contrastive Language-Image Pre-training Using Biomedical Documents
Lin, W., Zhao, Z., Zhang, X., Wu, C., Zhang, Y., Wang, Y., & Xie, W. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43993-3_51 · Cited by 249
CLIP in medical imaging: A survey
Zhao, Z., Liu, Y., Wu, H., Wang, M., Li, Y., Wang, S., Teng, L., Liu, D., Cui, Z., Wang, Q., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103551 · Cited by 248
A medical multimodal large language model for future pandemics
Liu, F., Zhu, T., Wu, X., Yang, B., You, C., Wang, C., Lu, L., Liu, Z., Zheng, Y., Sun, X., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00952-2 · Cited by 155
Foundation Model for Advancing Healthcare: Challenges, Opportunities and Future Directions
He, Y., Huang, F., Jiang, X., Nie, Y., Wang, M., Wang, J., & Chen, H. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3496744 · Cited by 162
Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images
Lu, M. Y., Chen, B., Zhang, A., Williamson, D. F. K., Chen, R. J., Ding, T., Le, L. P., Chuang, Y.-S., & Mahmood, F. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01893 · Cited by 122
MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training for X-ray Diagnosis
Wu, C., Zhang, X., Zhang, Y., Wang, Y., & Xie, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01954 · Cited by 148
Self-Supervised Multimodal Learning: A Survey
Zong, Y., Aodha, O. M., & Hospedales, T. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429301 · Cited by 165
A Foundation Language-Image Model of the Retina (FLAIR): encoding expert knowledge in text supervision
Silva-Rodríguez, J., Chakor, H., Kobbi, R., Dolz, J., & Ayed, I. B. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103357 · Cited by 134
Exploring scalable medical image encoders beyond text supervision
Pérez-García, F., Sharma, H., Bond-Taylor, S., Bouzid, K., Salvatelli, V., Ilse, M., Bannur, S., Castro, D. C., Schwaighofer, A., Lungren, M. P., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00965-w · Cited by 115
Self-citation
A Comprehensive Survey of Foundation Models in Medicine
Khan, W., Leem, S., See, K. B., Wong, J. K., Zhang, S., & Fang, R. (2025) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2025.3531360 · Cited by 137
CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training
You, K., Gu, J., Ham, J., Park, B., Kim, J., Hong, E. K., Baek, W., & Roh, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43895-0_10 · Cited by 109
Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging
Pahud de Mortanges, A., Luo, H., Shu, S. Z., Kamath, A., Suter, Y., Shelan, M., Pöllinger, A., & Reyes, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01190-w · Cited by 93
Multimodal Artificial Intelligence in Medical Diagnostics
Jandoubi, B., & Akhloufi, M. A. (2025) · Information · doi:10.3390/info16070591 · Cited by 97
PRIOR: Prototype Representation Joint Learning from Medical Images and Reports
Cheng, P., Lin, L., Lyu, J., Huang, Y., Luo, W., & Tang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01953 · Cited by 93
MedKLIP: Medical Knowledge Enhanced Language-Image Pre-TrainingPreprint
Wu, C., Zhang, X., Zhang, Y., Wang, Y., & Xie, W. (2023) · medRxiv · doi:10.1101/2023.01.10.23284412 · Cited by 80
Cross-Modal Causal Representation Learning for Radiology Report Generation
Chen, W., Liu, Y., Wang, C., Zhu, J., Li, G., Liu, C., & Lin, L. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3568746 · Cited by 91
Has multimodal learning delivered universal intelligence in healthcare? A comprehensive survey
Lin, Q., Zhu, Y., Mei, X., Huang, L., Ma, J., He, K., Peng, Z., Cambria, E., & Feng, M. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102795 · Cited by 86
Advancing medical imaging with language models: featuring a spotlight on ChatGPT
Hu, M., Qian, J. Y., Pan, S., Li, Y., Qiu, R. L. J., & Yang, X. (2024) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ad387d · Cited by 97
Open challenges and opportunities in federated foundation models towards biomedical healthcare
Li, X., Peng, L., Wang, Y., & Zhang, W. (2025) · BioData Mining · doi:10.1186/s13040-024-00414-9 · Cited by 84
From large language models to multimodal AI: a scoping review on the potential of generative AI in medicine
Buess, L., Keicher, M., Navab, N., Maier, A., & Arasteh, S. T. (2025) · Biomedical Engineering Letters · doi:10.1007/s13534-025-00497-1 · Cited by 79
A scoping review on multimodal deep learning in biomedical images and texts
Sun, Z., Lin, M., Zhu, Q., Xie, Q., Wang, F., Lu, Z., & Peng, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104482 · Cited by 69
Are Natural Domain Foundation Models Useful for Medical Image Classification?
Huix, J. P., Ganeshan, A. R., Haslum, J. F., Söderberg, M. P., Matsoukas, C., & Smith, K. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00746 · Cited by 67
Semi-Supervised Medical Report Generation via Graph-Guided Hybrid Feature Consistency
Zhang, K., Jiang, H., Zhang, J., Huang, Q., Jianping Fan, B., Yu, J., & Han, W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3273390 · Cited by 64
Enhancing representation in radiography-reports foundation model: a granular alignment algorithm using masked contrastive learning
Huang, W., Li, C., Zhou, H.-Y., Yang, H., Liu, J., Liang, Y., Zheng, H., Zhang, S., & Wang, S. (2024) · Nature Communications · doi:10.1038/s41467-024-51749-0 · Cited by 62
Medical Visual Question Answering via Conditional Reasoning and Contrastive Learning
Liu, B., Zhan, L.-M., Xu, L., & Wu, X.-M. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3232411 · Cited by 63
Medical multimodal foundation models in clinical diagnosis and treatment: Applications, challenges, and future directions
Sun, K., Xue, S., Sun, F., Sun, H., Luo, Y., Wang, L., Wang, S., Guo, N., Liu, L., Zhao, T., et al. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103265 · Cited by 69
Medical image identification methods: A review
Li, J., Jiang, P., An, Q., Wang, G., & Kong, H. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107777 · Cited by 61
Multi-Grained Radiology Report Generation With Sentence-Level Image-Language Contrastive Learning
Liu, A., Guo, Y., Yong, J., & Xu, F. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3372638 · Cited by 55
M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization
Liu, C., Cheng, S. L., Chen, C., Qiao, M., Zhang, W., Shah, A., Bai, W., & Arcucci, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43907-0_61 · Cited by 54
Continual Learning for Abdominal Multi-organ and Tumor Segmentation
Zhang, Y., Li, X., Chen, H.-X., Yuille, A., Liu, Y., & Zhou, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43895-0_4 · Cited by 53
Improving Medical Vision-Language Contrastive Pretraining With Semantics-Aware Triage
Liu, B., Lu, D., Wei, D., Wu, X., Wang, Y., Zhang, Y., & Zheng, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3294980 · Cited by 56
Exploring low-resource medical image classification with weakly supervised prompt learning
Zheng, F., Cao, J., Yu, W., Chen, Z., Xiao, N., & Lu, Y. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110250 · Cited by 54
Multi-task paired masking with alignment modeling for medical vision-language pre-training
Zhang, K., Yang, Y., Yu, J., Jiang, H., Fan, J., Huang, Q., & Han, W. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3325965 · Cited by 51
A survey on advancements in image–text multimodal models: From general techniques to biomedical implementations
Guo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 58
Scaling-up medical vision-and-language representation learning with federated learning
Lu, S., Liu, Z., Liu, T., & Zhou, W. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107037 · Cited by 49
Cross-Modal Conditioned Reconstruction for Language-Guided Medical Image Segmentation
Huang, X., Li, H., Cao, M., Chen, L., You, C., & An, D. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3523333 · Cited by 47
Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammography
Ghosh, S., Poynton, C. B., Visweswaran, S., & Batmanghelich, K. N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_59 · Cited by 46
A multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis
Liu, F., Zheng, L., Yin, Q., Huang, J., Luo, J., Thakur, A., Branson, K., Schwab, P., Yin, B., Wu, X., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01339-7 · Cited by 50
Developing medical imaging AI for emerging infectious diseases
Huang, S.-C., Chaudhari, A. S., Langlotz, C. P., Shah, N. H., Yeung, S. Y., & Lungren, M. P. (2022) · Nature Communications · doi:10.1038/s41467-022-34234-4 · Cited by 42
Self-citation
RadFusion: Benchmarking Performance and Fairness for Multimodal Pulmonary Embolism Detection from CT and EHRPreprint
Zhou, Y., Huang, S.-C., Fries, J., Youssef, A., Amrhein, T. J., Chang, M., Banerjee, I., Rubin, D. L., Xing, L., Shah, N. H., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.11665 · Cited by 60
Self-citation
ScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embedding
Li, Z., Zheng, Y., Luo, X., Shan, D., & Hong, Q. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612056 · Cited by 39
RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports
Du, J., Guo, J., Zhang, W., Yang, S., Liu, H., Li, H., & Wang, N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_66 · Cited by 37
Multimodal Fusion Learning with Dual Attention for Medical Imaging
Dhar, J., Zaidi, N. A., Haghighat, M., Roy, S. S., Goyal, P., Alavi, A., & Kumar, V. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00428 · Cited by 35
MITER: Medical Image–TExt joint adaptive pretRaining with multi-level contrastive learning
Shu, C., Zhu, Y., Tang, X., Xiao, J., Chen, Y., Li, X., Zhang, Q., & Lu, Z. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121526 · Cited by 34
Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts
Chen, Z., Diao, S., Wang, B., Li, G., & Wan, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02139 · Cited by 34
Residual-based Language Models are Free Boosters for Biomedical Imaging Tasks
Lai, Z., Wu, J., Chen, S., Zhou, Y., & Hovakimyan, N. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00515 · Cited by 42
CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification
Lai, H., Yao, Q., Jiang, Z., Wang, R., He, Z., Tao, X., & Zhou, S. K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01059 · Cited by 42
Xplainer: From X-Ray Observations to Explainable Zero-Shot Diagnosis
Pellegrini, C., Keicher, M., Özsoy, E., Jiraskova, P., Braren, R., & Navab, N. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43904-9_41 · Cited by 32
Anatomy-aware self-supervised learning for anomaly detection in chest radiographs
Sato, J., Suzuki, Y., Wataya, T., Nishigaki, D., Kita, K., Yamagata, K., Tomiyama, N., & Kido, S. (2023) · iScience · doi:10.1016/j.isci.2023.107086 · Cited by 38
Mapping medical image-text to a joint space via masked modeling
Chen, Z., Du, Y., Hu, J., Liu, Y., Li, G., Wan, X., & Chang, T. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103018 · Cited by 31
ConTEXTual Net: A Multimodal Vision-Language Model for Segmentation of Pneumothorax
Huemann, Z., Tie, X., Hu, J., & Bradshaw, T. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01051-8 · Cited by 32
IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-Training
Liu, C., Cheng, S. L., Shi, M., Shah, A., Bai, W., & Arcucci, R. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3449690 · Cited by 30
Advancements in Medical Radiology Through Multimodal Machine Learning: A Comprehensive Overview
Haq, I. U., Mhamed, M., Alharbi, M., Osman, H., Hamd, Z. Y., & Liu, Z. (2025) · Bioengineering · doi:10.3390/bioengineering12050477 · Cited by 32
Harnessing Text Insights With Visual Alignment for Medical Image Segmentation
Zeng, Q., Luo, H., Lu, Z., Xie, Y., Wang, Z., Zhang, Y., & Xia, Y. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3601359 · Cited by 33
EVA-X: a foundation model for general chest x-ray analysis with self-supervised learning
Yao, J., Wang, X., Song, Y., Zhao, H., Ma, J., Chen, Y., Liu, W., & Wang, B. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02032-z · Cited by 36
Breaking with Fixed Set Pathology Recognition Through Report-Guided Contrastive Training
Seibold, C., Reiß, S., Sarfraz, M. S., Stiefelhagen, R., & Kleesiek, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_66 · Cited by 30
STPNet: Scale-Aware Text Prompt Network for Medical Image Segmentation
Shan, D., Li, Z., Li, Y., Li, Q., Tian, J., & Hong, Q. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3571672 · Cited by 29
Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders
Jang, J., Kyung, D., Kim, S. H., Lee, H., Bae, K., & Choi, E. (2024) · Scientific Reports · doi:10.1038/s41598-024-73695-z · Cited by 34
Multimodal Machine Learning Approaches in Predictive Healthcare Analytics: A Comprehensive Survey
Vavekanand, R., Kumar, T., Kumar, S., Kumar, G., & Laghari, A. A. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10560-4 · Cited by 30
ViLMedic: a framework for research at the intersection of vision and language in medical AI
Delbrouck, J.-B., Saab, K. K., Varma, M., Eyuboglu, S., Chambon, P., Dunnmon, J., Chaves, J. M. Z., Chaudhari, A. S., & Langlotz, C. P. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.3 · Cited by 30
Facing & mitigating common challenges when working with real-world data: The Data Learning Paradigm
Lever, J., Cheng, S. L., Quilodrán-Casas, C., Liu, C., Fan, H., Platt, R. W., Rakotoharisoa, A., Johnson, E., Li, S., Shang, Z., et al. (2025) · Journal of Computational Science · doi:10.1016/j.jocs.2024.102523 · Cited by 29
Rethinking masked image modelling for medical image representation
Xie, Y., Gu, L., Harada, T., Zhang, J., Xia, Y., & Wu, Q. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103304 · Cited by 30
Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-Training Framework
Phan, V. M. H., Xie, Y., Qi, Y., Liu, L., Liu, L., Zhang, B., Liao, Z., Wu, Q., To, M., & Verjans, J. W. H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01092 · Cited by 26
Integrating language into medical visual recognition and reasoning: A survey
Lu, Y., & Wang, A. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103514 · Cited by 32
Medical Phrase Grounding with Region-Phrase Context Contrastive Alignment
Chen, Z., Zhou, Y., Tran, A., Zhao, J., Wan, L., Ooi, G. S. K., Cheng, L. T.-E., Thng, C. H., Xu, X., Liu, Y., et al. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43990-2_35 · Cited by 26
Beyond Medical Imaging - A Review of Multimodal Deep Learning in RadiologyPreprint
Heiliger, L., Sekuboyina, A., Menze, B. H., Egger, J., & Kleesiek, J. (2022) · doi:10.36227/techrxiv.19103432.v1 · Cited by 28
UniDCP: Unifying Multiple Medical Vision-Language Tasks via Dynamic Cross-Modal Learnable Prompts
Zhan, C., Zhang, Y., Lin, Y., Wang, G., & Wang, H. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3397191 · Cited by 25
Self-supervised multi-modal training from uncurated images and reports enables monitoring AI in radiology
Park, S. J., Lee, E. S., Shin, K. S., Lee, J. E., & Ye, J. C. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103021 · Cited by 27
Common Vision-Language Attention for Text-Guided Medical Image Segmentation of Pneumonia
Guo, Y., Zeng, X., Zeng, P., Fei, Y., Lu, W. S., Zhou, J., & Wang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72114-4_19 · Cited by 24
LVF: A language and vision fusion framework for tomato diseases segmentation
Hu, Y., Zhu, J., Zhou, G., He, M., Lv, M., Wang, J., Chen, A., Deng, J., & Jiang, Y. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109484 · Cited by 24
UniChest: Conquer-and-Divide Pre-Training for Multi-Source Chest X-Ray Classification
Dai, T., Zhang, R., Hong, F., Yao, J., Zhang, Y., & Wang, Y. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3381123 · Cited by 31
Visual-Textual Matching Attention for Lesion Segmentation in Chest Images
Bui, P.-N., Le, D.-T., & Choo, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72114-4_67 · Cited by 23
LGA: A Language Guide Adapter for Advancing the SAM Model’s Capabilities in Medical Image Segmentation
Hu, J., Li, Y., Sun, H., Song, Y., Zhang, C., Lin, L., & Chen, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_57 · Cited by 28
HERGen: Elevating Radiology Report Generation with Longitudinal Data
Wang, F., Du, S., & Yu, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73001-6_11 · Cited by 33
Visual-language foundation models in medical imaging: A systematic review and meta-analysis of diagnostic and analytical applications
Sun, Y., Wen, X., Zhang, Y., Jin, L., Yang, C., Zhang, Q., Jiang, M. C., Xu, Z., Guo, W., Su, J., et al. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108870 · Cited by 27
Contrastive Masked Image-Text Modeling for Medical Visual Representation Learning
Chen, C., Zhong, A., Wu, D., Luo, J., & Li, Q. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43904-9_48 · Cited by 23
Unlocking the diagnostic potential of electrocardiograms through information transfer from cardiac magnetic resonance imaging
Turgut, Ö., Müller, P., Hager, P., Shit, S., Starck, S., Menten, M. J., Martens, E., & Rueckert, D. (2025) · Medical Image Analysis · doi:10.1016/j.media.2024.103451 · Cited by 22
Multi-label chest X-ray image classification based on long-range dependencies capture and label relationships learning
Zhao, X., & Wang, X. Y. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107018 · Cited by 21
A systematic review of multi-modal large language models on domain-specific applications
Li, S., Wong, K. W., Wang, G., & Duong, T.-T. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11398-1 · Cited by 24
Automated image label extraction from radiology reports — A review
Pereira, S. C., Mendonça, A. M., Campilho, A., Sousa, P., & Lopes, C. T. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102814 · Cited by 29
Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Grounding
He, J., Li, P., Liu, G., & Zhong, S. (2025) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi60581.2025.10981029 · Cited by 20
Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis
Yan, R., Zhang, X., Jiang, Z., Wang, B., Bian, X., Ren, F., & Zhou, S. K. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3611531 · Cited by 20
Anatomical Structure-Guided Medical Vision-Language Pre-training
Li, Q., Yan, X., Xu, J., Yuan, R., Zhang, Y., Feng, R., Shen, Q., Zhang, X., & Wang, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72120-5_8 · Cited by 20
Multimodal deep learning-based diagnostic model for BPPV
Lu, H., Mao, Y., Li, J., & Zhu, L. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02438-x · Cited by 19
A Multimodal Transfer Learning Approach Using PubMedCLIP for Medical Image Classification
Dao, H. N., Nguyen, T. D., Mugisha, C., & Paik, I. (2024) · IEEE Access · doi:10.1109/access.2024.3401777 · Cited by 19
Automatic Medical Report Generation: Methods and Applications
Li, G., Tahir, A. M., Zhang, D., Wang, Z. J., & Ward, R. K. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.20240044 · Cited by 20
Improving Medical Multi-modal Contrastive Learning with Expert Annotations
Kumar, Y., & Marttinen, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72661-3_27 · Cited by 29
AO-UOD: A Novel Paradigm for Underwater Object Detection Using Acousto–Optic Fusion
Yu, F., Xiao, F., Li, C., Cheng, E., & Yuan, F. (2025) · IEEE Journal of Oceanic Engineering · doi:10.1109/joe.2025.3529121 · Cited by 20
Coarse-to-Fine Covid-19 Segmentation via Vision-Language Alignment
Shan, D., Li, Z., Chen, W., Li, Q., Tian, J., & Hong, Q. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096683 · Cited by 18

Authors: Huang, S.-C., Shen, L., Lungren, M. P., & Yeung, S.
Affiliations: Stanford University (US)
ORCID: Huang, Shih-Cheng
Shen, Liyue
Lungren, Matthew P.
Yeung, Serena
Venue: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher: IEEE
Published: 2021-10
Pages: 3922-3931
DOI: 10.1109/iccv48922.2021.00391
OpenAlex: W3201906559
Cited by: 489
References: 41
Citations by year: 2026: 122 · 2025: 195 · 2024: 106 · 2023: 51 · 2022: 14 · 2021: 1
FWCI: 14.04
Type: conference paper
Language: en
Open access: closed
Licence: doi.org/10.15223/policy-029 (stm-asf)
doi.org/10.15223/policy-037 (stm-asf)
Locations: IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
Topics: Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords: multimodal representation learning, label-efficient learning, medical image recognition, radiology reports, image-text retrieval, zero-shot classification, medical image segmentation, limited labeled data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks