The DOI matches its Crossref record; 63 of the 68 references it cites were found in a registry or an index.
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing
Open accessAbstract
Multi-modal data abounds in biomedicine, such as radiology images and reports. Interpreting this data at scale is essential for improving clinical care and accelerating clinical research. Biomedical text with its complex semantics poses additional challenges in vision--language modelling compared to the general domain, and previous work has used insufficiently adapted models that lack domain-specific language understanding. In this paper, we show that principled textual semantic modelling can substantially improve contrastive learning in self-supervised vision--language processing. We release a language model that achieves state-of-the-art results in radiology natural language inference through its improved vocabulary and novel language pretraining objective leveraging semantics and discourse characteristics in radiology reports. Further, we propose a self-supervised joint vision--language approach with a focus on better text modelling. It establishes new state of the art results on a wide range of publicly available benchmarks, in part by leveraging our new domain-specific language model. We release a new dataset with locally-aligned phrase grounding annotations by radiologists to facilitate the study of complex semantic modelling in biomedical vision--language processing. A broad evaluation, including on this new dataset, shows that our contrastive learning approach, aided by textual-semantic modelling, outperforms prior methods in segmentation tasks, despite only using a global-alignment objective.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We note that the official MIMIC-CXR test split is highly enriched for abnormal cases, 26 hence prior studies are more common (Table 1).”
From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“By inviting a domain expert to review and adapt individual phrase outputs, our approach to qualitative evaluation of report generation differs from most existing works that primarily ask experts to rate overall report quality via a (Likert-) scale; 59 provide total error counts/ categorisation; 48,60 or engage in comparison rankings across different report candidates.61”
From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [48]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 4×
“We use a ResNet-50 [28] architecture as our image encoder and pretrain the model on MIMIC-CXR images using SimCLR [6] with domain-specific augmentations as detailed in Section 4.”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · cited 2×
“[64] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.”
BackgroundMethodInfluentialLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 5×
“A variety of self-supervised VLP approaches have been proposed towards jointly learning visual and textual representations of paired data without supervision, such as frameworks using contrastive objectives [26, 42, 60], approaches based on joint transformer architectures [40, 41, 51, 69], self-supervised VLP with wor…”
BackgroundMethodInfluentialGoldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891 · cited 6×
“network (CNN) [37] image encoder Eimg, our CXR-BERT text encoder Etxt, and projection models Pimg and Ptxt to learn representations in a joint space.”
BackgroundMethodInfluentialMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 10×
“The phases proceed as follows: (I) First, we construct a custom WordPiece [80] vocabulary of 30k to-kens from PubMed abstracts 4 (15 GB), MIMIC-III [35] clinical notes (3.5 GB), and MIMIC-CXR radiology reports (0.1 GB).”
BackgroundMethodInfluentialBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Related medical VLP work commonly uses publicly available contextual word embedding models including BioBERT [39], ClinicalBERT [2], BioClini-calBERT [2], or PubMedBERT [26].”
BackgroundMethodInfluential
Research works printed but not linked: 21
- Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13–18 July 2020, Virtual Event. Proceedings of Machine Learning Research, vol. 119, pp. 1597–1607. PMLR (13–18 Jul 2020), http://proceedings.mlr.press/v119/chen20j.html
- Graf, B., et al.: Pneumothorax and chest tube classification on chest X-rays for detection of missed pneumothorax. Machine Learning for Health (ML4H) NeurIPS Workshop: Extended Abstract (2020). https://arxiv.org/abs/2011.07353 · arXiv
- Hayat, N., Lashen, H., Shamout, F.E.: Multi-label generalized zero shot learning for the classiffcation of disease in chest radiographs. In: Machine Learning for Healthcare Conference, pp. 461–477. PMLR (2021)
- Hsu, T.M.H., Weng, W.H., Boag, W., McDermott, M., Szolovits, P.: Unsupervised multimodal representation learning across medical images and reports. Machine Learning for Health (ML4H) NeurIPS Workshop (2018). https://arxiv.org/abs/1811.08615 · arXiv
- Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456. PMLR (2015)
- Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: VisualBERT: a simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019) · arXiv
- Li, Y., et al.: Supervision exists everywhere: a data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208 (2021) · arXiv
- Liu, X., et al.: Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994 (2020) · arXiv
- Liu, Y., et al.: RoBERTa: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) · arXiv
- Logeswaran, L., Lee, H.: An efficient framework for learning sentence representations. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, 30 April - 3 May 2018, Conference Track Proceedings. OpenReview.net (2018). https://openreview.net/forum?id=rJvJXZb0W
- Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=Bkg6RiCqY7
- Lu, J., Batra, D., Parikh, D., Lee, S.: ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Advances in Neural Information Processing Systems, vol. 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019 (December), pp. 8–14 (2019). Vancouver, BC, Canada, pp. 13–23 (2019). https://proceedings.neurips.cc/paper/2019/hash/c74d97b01eae257e44aa9d5bade97baf-Abstract.html
- Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018) · arXiv
- Preechakul, K., Piansaddhayanon, C., Naowarat, B., Khandhawit, T., Sriswasdi, S., Chuangsuwanich, E.: Set prediction in the latent space. In: Advances in Neural Information Processing Systems, vol. 34 (2021)
- Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018) · arXiv
- Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28: Annual Conference on Neural Information Processing Systems 2015(December), pp. 7–12, 2015. Montreal, Quebec, Canada 28, pp. 91–99 (2015). https://proceedings.neurips.cc/paper/2015/hash/14bfa6bb14875e45bba028a21ed38046-Abstract.html
- Simard, P., Steinkraus, D., Platt, J.: Best practices for convolutional neural networks applied to visual document analysis. In: Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings, pp. 958–963. IEEE (2003)
- Wolf, T., et al.: Huggingface’s transformers: state-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019) · arXiv
- You, Y., Gitman, I., Ginsburg, B.: Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 (2017) · arXiv
- Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: Bengio, Y., LeCun, Y. (eds.) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2–4 May 2016, Conference Track Proceedings (2016). http://arxiv.org/abs/1511.07122 · arXiv
- Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747 (2020) · arXiv
Cited by
Sorted by citations
Items 101–200 of 315
Towards Medical Vision-Language Contrastive Pre-training via Study-Oriented Semantic Exploration
Liu, B., Lu, Z., & Wang, Y. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681531 · Cited by 8
“For the dual-encoder-based paradigm [2, 20, 28, 32, 41, 51], the pretraining objective is to align visual and textual features extracted by two separate encoders in the embedding space.”
InfluentialEnhancing Medical Vision-Language Contrastive Learning via Inter-Matching Relation Modeling
Li, M., Meng, M., FULHAM, M. J., Feng, D. D., Bi, L., & Kim, J. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3534436 · Cited by 9
“BERT [23] and its variants, e.g., BioClinical-BERT [24] and CXR-BERT [25] which are pretrained on medical text, have been widely employed by mVLCL meth-ods as domain-specific language encoders.”
Withdrawn: Large Language Models in Healthcare Decision Support: A ReviewPreprint
Vavekanand, R., Karttunen, P., Xu, Y., Milani, S., & Li, H. (2024) · Preprints.org · doi:10.20944/preprints202407.1842.v1 · Cited by 7
Li, Y., Kong, C., Zhao, G., & Zhao, Z. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11337-0 · Cited by 7
Multimodal transformers for joint diagnosis and retrieval from chest X-rays and radiology reports
Jahanian, M., Karimi, A., Eraghi, N. O., & Zarafshan, F. (2025) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2025.101672 · Cited by 7
Özdemir, Z., Keleş, H. Y., & Tanriöver, Ö. Ö. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3615479 · Cited by 10
“The third method, Deep Transfer Learning (DTL), omits episodic training and instead fine-tunes ImageNet-pretrained models directly on base classes.”
Kim, J., & Park, H. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635671 · Cited by 7
MEDBind: Unifying Language and Multimodal Medical Data Embeddings
Gao, Y., Kim, S., Austin, D. E., & McIntosh, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_21 · Cited by 9
Dombrowski, M., Reynaud, H., Baugh, M., & Kainz, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00097 · Cited by 9
“Then we report the pixel-wise AUC-ROC on MS-CXR [5], a subset of MIMIC with bounding box labels for diseases.”
MethodWhen Visual Grounding Meets Gigapixel-Level Large-Scale Scenes: Benchmark and Approach
Tao, M., Bai, B., Lin, H., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02088 · Cited by 7
“In addition, there are related benchmarks in fields such as remote sensing [15, 16] and biomedical science [17, 18], with their resolutions also limited to 1024 × 1024 or lower.”
BackgroundHybrid cross-modality fusion network for medical image segmentation with contrastive learning
Zhou, X., Song, Q., Nie, J., Feng, Y., Liu, H., Liang, F., Chen, L., & Xie, J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110073 · Cited by 8
Text-Driven Medical Image Segmentation With Text-Free Inference via Knowledge Distillation
Zhao, P., Hou, Y., Yan, Z., & Huo, S. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3545506 · Cited by 8
“A text prompt t ∈ R L with a length of L is first processed by the CXR-BERT [7] to extract the text feature t ∈ R L × 768 , and then projected by a nonlinear layer and layer normalization (LN) to incorporate into the decoder block.”
Du, S., Liang, S., & Gu, Y. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104811 · Cited by 6
He, Z., Wong, A. N. N., & Yoo, J. S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110625 · Cited by 9
ViTexNet: Vision-Text Guided Dynamic Convolution Network for Medical Image Segmentation
Bhardwaj, R., Tambe, U. Y., & Neog, D. R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04981-0_65 · Cited by 6
A multimodal vision–language model for generalizable annotation-free pathology localization
Yang, H., Zhou, H.-Y., Liu, J., Huang, W., Li, C., Li, Z., Gao, Y., Liu, Q., Liang, Y., Yang, Q., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01574-7 · Cited by 11
“MS-CXR [28].”
InfluentialSpiegler, P., Koleilat, T., Harirpoush, A., Miller, C., Rivaz, H. J., Kersten‐Oertel, M., & Xiao, Y. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00104 · Cited by 10
“In the biomedical space, several CLIP variants, such as BioViL [1], PubMedCLIP [7], and BiomedCLIP [41], have incorporated medical corpora to improve domain alignment.”
Multimodal Self-Supervised Learning for Lesion Localization
Yang, H., Zhou, H.-Y., Li, C., Huang, W., Liu, J., Liang, Y., Shi, G., Zheng, H., Liu, Q., & Wang, S. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635268 · Cited by 7
“To overcome this problem, some research methods have been improved by aligning words with local regions of the image [8, 11].”
BackgroundMethodMCAD: Multi-modal Conditioned Adversarial Diffusion Model for High-Quality PET Image Reconstruction
Cui, J., Zeng, X., Zeng, P., Liu, B., Wu, X., Zhou, J., & Wang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72104-5_45 · Cited by 6
Chen, C., Zhao, L., Lang, Q., & Xu, Y. (2024) · Bioengineering · doi:10.3390/bioengineering11100993 · Cited by 5
“The emergence of multimodal methods in the medical domain, particularly language– vision models, has demonstrated a remarkable capability to extract and integrate representations from both images and text descriptions [41].”
BackgroundMG-UNet: A Memory-Guided UNet for Lesion Segmentation in Chest Images
Ding, S., Li, M., & Wang, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04927-8_34 · Cited by 6
Jiang, H., Hao, X., Huang, Y., Ma, C., Zhang, J., Pan, Y. F., & Zhang, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91721-9_2 · Cited by 6
Normal-Abnormal Decoupling Memory for Medical Report Generation
Zhao, G., Yan, Y., & Zhao, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.131 · Cited by 5
Disease-Informed Adaptation of Vision-Language Models
Zhang, J., Wang, G., Kalra, M. K., & Yan, P. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3484294 · Cited by 7
“For BioViL, we adhered to the approach described in the original paper [11], setting a grid of P = 16 × 16 on the output feature map produced by ResNet-50 vision encoder.”
InfluentialSymbiotic Fusion of Evolving Anatomical Landscapes for Intelligent Diagnostic Narratives
Du, H., Zhuang, Y., & Xu, J. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822272 · Cited by 5
“For the CE metrics, we fine-tuned the CXR-Bert model [28] for multi-label classification on 18 abnormality categories within the radiology reports.”
MOSMOS: Multi-organ segmentation facilitated by medical report supervision
Tian, W., Huang, X., Hou, J., Ren, C., Jiang, L., Zhao, R.-W., Jin, G., Zhang, Y., & Geng, D. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.107743 · Cited by 6
Yu, Y., Wang, J., Liu, W., Mien, I. H., Krishnaswamy, P., Yang, X., & Cheng, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130018 · Cited by 5
Compositional Zero-Shot Domain Transfer with Text-to-Text Models
Liu, F., Liu, Q., Bannur, S., Pérez‐García, F., Usuyama, N., Zhang, S., Naumann, T., Nori, A., Poon, H., Alvarez-Valle, J., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00585 · Cited by 7
Self-citationA Novel Multimodal Prototype Network for Interpretable Medical Image Classification
Wang, G., Li, J., Cheng, T., Ma, X., & Liu, S. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394356 · Cited by 6
“We follow CXR-BERT [17] to use a contrastive loss function to learn the shared latent semantics between image and text. where τ 2 is the scaling parameter and N is the batch size.”
MethodDisease-Informed Adaptation of Vision-Language Models
Zhang, J., Wang, G., Kalra, M. K., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72120-5_22 · Cited by 5
OmicsNavigator : An auditable scientific partner for scalable hypothesis validation in spatial omicsPreprint
Yiyao, L., Vakharia, N., Liang, W., Mayer, A. T., Luo, R., Trevino, A. E., & Wu, Z. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.21.665821 · Cited by 5
Yang, Z., & Shen, L. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00454 · Cited by 4
“For the text, we use a BERT-based encoder [11], specifically CXR-BERT [4], pre-trained on large-scale medical datasets.”
InfluentialGrounded knowledge-enhanced medical vision–language pre-training for chest X-ray
Deng, Q., Huang, Z., Wang, Y., Wang, Z., Wang, Z., Zhang, X., Dou, Q., Hui, Y. Y., & Hui, E. S. K. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108416 · Cited by 7
“ConVIRT [3] employs a contrastive loss to align global representations, while GLoRIA [5], LovT [6] and BioViL [7] incorporate instance-level local contrastive learning on image sub-regions and corresponding radiology report sentence representations.”
Xu, H., Nie, Y., Wang, H., Chen, Y., Li, W., Ning, J., Liu, L., Wang, H., Zhu, L., Liu, J., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04971-1_37 · Cited by 6
Shehzad, F., Mennella, C., Esposito, M., & Minutolo, A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01011-x · Cited by 4
Harib, W., Fouad, S., & Mahmoud, T. F. (2026) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi61048.2026.11515438 · Cited by 4
“Feature extraction is performed using eight pre-trained models for pneumonia detection on CXR: a CNN [22], ViT [23], MobileNetV2 [24], ResNet101V2 [25], Inception– ResNetV2 [26], a simple TensorFlow CNN [27], and two vision–language heads based on BioViL-T [28] and SigLIP [29].”
Automated Spinal MRI Labelling from Reports Using a Large Language Model
Park, R. Y., Windsor, R., Jamaludin, A., & Zisserman, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72086-4_10 · Cited by 4
“However, these are often trained using clinical abstracts and papers, which have a different form to radiological reports [9, 10, 12] or use reports from a single modality, e.g. Chest X-rays [1, 2].”
BackgroundTowards an Exhaustive Evaluation of Vision-Language Foundation Models
Salin, E., Ayache, S., & Favre, B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00041 · Cited by 4
Application of Vision-Language Models for Assessing Osteoarthritis
Felfeliyan, B., Zhou, Y., Ghosh, S., Küpper, J., Liu, S., Hareendranathan, A. R., & Jaremko, J. L. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635386 · Cited by 4
“In this study, a pre-trained CXR-BERT-general from the [15] was used as the backbone text encoder, which was pre-trained on PubMed abstracts and clinical notes from two other publicly available datasets (MIMIC-IIICXR).”
BackgroundMethodGEM: Context-Aware Gaze EstiMation with Visual Search Behavior Matching for Chest Radiograph
Liu, S., Chen, W., Liu, J., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72378-0_49 · Cited by 4
Unified Medical Image Pre-training in Language-Guided Common Semantic Space
He, X., Yang, Y., Jiang, X., Luo, X., Hu, H. R., Zhao, S., Li, D., Yang, Y., & Qiu, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73004-7_8 · Cited by 5
Using Multiple Instance Learning to Build Multimodal Representations
Wang, P., Wells, W. M., Berkowitz, S., Horng, S., & Golland, P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34048-2_35 · Cited by 4
“We employ ResNet-50 [12] as the image region encoder and CXR-BERT [2] as the sentence encoder.”
BackgroundMethodResultInfluentialSun, J., Wei, D., Xu, Z., Lu, D., Liu, H., Wang, L., & Zheng, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43907-0_35 · Cited by 4
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling
Thawani, A., Ghanekar, S., Zhu, X., & Pujara, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.662 · Cited by 12
Huang, W., Li, C., Zhou, H.-Y., Liu, J., Yang, H., Liang, Y., Shi, G., Zheng, H., & Wang, S. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635303 · Cited by 5
“GLoRIA [11] introduces a local alignment framework that contrasts image sub-regions with words in the paired report, leading to associations be-tween words and their corresponding image regions.”
BackgroundDomain Adaptation and Generalization Using Foundation Models in Healthcare ImagingPreprint
Ying, H., Lia, Y., & Fu, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5345726 · Cited by 4
Li, K., Li, H., Cui, M., Li, J., Lv, P., Zhou, B., & Xu, M. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3535318 · Cited by 4
Scaling Chest X-Ray Foundation Models From Mixed Supervisions for Dense Prediction
Wang, F., & Yu, L. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3589928 · Cited by 5
“5, following previous studies [9].”
InfluentialMulti-Grained Vision-and-Language Model for Medical Image and Text Alignment
Yan, H., Yang, X., Bai, L., Li, J., & Liang, J. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3590930 · Cited by 4
RRG-DPO: Direct Preference Optimization for Clinically Accurate Radiology Report Generation
Liu, H., Wei, D., Xu, Z., Wu, X., Zheng, Y., & Wang, L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04971-1_52 · Cited by 4
Retrieval-Augmented Representation for Semi-Supervised Medical Image Segmentation
Luo, H., Zeng, Q., Zhang, Y., & Xia, Y. (2026) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2026.3712008 · Cited by 3
“For textual modeling, we adopted CXR-BERT [46] as the text encoder.”
Kulsoom, U., Glavin, F. G., & Bendechache, M. (2025) · IEEE Access · doi:10.1109/access.2025.3582728 · Cited by 5
Suo, J., Dong, K., He, K., He, K., & Qionghai, D. (2025) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01456-y · Cited by 5
“We used the results obtained by the U-Net-based segmentation network trained from the 817 samples in the MS-CXR dataset as benchmark, and respectively added 800 and 1,600 generated data to test performance changes.”
InfluentialChen, Y., Xu, S., Sellergren, A., Matias, Y., Hassidim, A., Shetty, S., Golden, D. I., Yuille, A., & Yang, L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04978-0_8 · Cited by 5
Feature Decomposition via Shared Low-Rank Matrix Recovery for CT Report Generation
Tian, Y., & Song, Y. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3628159 · Cited by 3
“For CT-RATE, we use GREEN 6 [71] and additionally report precision (P), recall (R), and F1 on 18 abnormalities [11] using a CXR-BERT-based model [72] following the standard process.”
GTGM: Generative Text-Guided 3D Vision-Language Pretraining for Medical Image Segmentation
Chen, Y., Liu, C., Huang, W., Liu, X., Shi, H., Cheng, S. L., Arcucci, R., & Xiong, Z. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00699 · Cited by 3
KU-DMIS-MSRA at RadSum23: Pre-trained Vision-Language Model for Radiology Report Summarization
Kim, G., Kim, H., Ji, L., Bae, S., Kim, C., Sung, M., Kim, H., Yan, K., Chang, E., & Kang, J. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.59 · Cited by 3
Kandiyana, A., Mouton, P. R., Hall, L., & Goldgof, D. B. (2024) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms61543.2024.00042 · Cited by 3
“Another example is the outperformance of BiomedCLIP [3] over bioVIL [22], a radiology-specific model for tasks such as pneumonia detection.”
MethodReport-Concept Textual-Prompt Learning for Enhancing X-ray Diagnosis
Zhao, X., Liu, Z., Лю, Ф., Li, G., Dou, Y., & Peng, S. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681568 · Cited by 3
Sultana, J., Qin, R., & Yin, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0901-7_9 · Cited by 4
“Notable works such as GloRIA [16] and BioVIL [6] have demonstrated the potential of aligning chest X-ray images with radiological reports through self-supervised contrastive learning, effectively capturing both local and global visual features.”
Das, J., & Garai, P. (2025) · International Conference on Intelligent Systems, Advanced Computing and Communication (ISACC) · doi:10.1109/isacc65211.2025.10969237 · Cited by 3
“Recent adaptations incorporate domain-specific knowledge, ensuring clinical relevance in the generated representations [12].”
InfluentialCSMAE : Cataract Surgical Masked Autoencoder (MAE) Based Pre-Training
Shah, N. A., Bandara, C., Sikder, S., Vedula, S. S., & Patel, V. M. (2025) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi60581.2025.10981288 · Cited by 3
“Medical imaging often combines images and text, as seen in X-ray diagnosis [3] and radiology report generation [4] with vision-language models.”
The role of multimodality in clinical disease diagnosis: advances, challenges, and opportunities
Nawaz, A., Edinat, A., Rana, M. R. R., Ali, T., Mustafa, G., Tahir, S., & Lee, S. W. (2026) · Frontiers in public health · doi:10.3389/fpubh.2026.1788454 · Cited by 3
Gowda, S. N., & Clifton, D. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72120-5_40 · Cited by 2
Yang, C., Jin, Q., Du, F., Guo, J., & Zhou, Y. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01717-4 · Cited by 3
MMPL-Seg: Prompt Learning with Missing Modalities for Medical Segmentation
Qu, R., Wu, W., Guan, Y., & Shu, L. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822668 · Cited by 2
“So far, research on text-guided medical segmentation has been limited to trying to get a better representation [13] or optimizing architectures for information fusion [14] [15].”
Mathematically-Grounded Multimodal Attention Network for Breast Cancer Prognosis
Guo, M., Luo, Z., Liu, J., et al. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822185 · Cited by 2
A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
Silva-Rodríguez, J., Dolz, J., & Ayed, I. B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-96625-5_20 · Cited by 2
Chen, R., Min, W., Han, Q., Li, J., Li, L., & Wu, H. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112263 · Cited by 4
Shin, C., Eom, D., Lee, S. M., Park, J. E., Kim, K., & Lee, K. H. (2025) · Scientific Reports · doi:10.1038/s41598-025-16213-z · Cited by 5
Zhu, Y. Y., Wang, X., Huang, X., Xia, M., Mu, P., Hu, H., & Zhang, X. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3606881 · Cited by 3
InstructX2X: An Interpretable Local Editing Model for Counterfactual Medical Image Generation
Min, H., You, T., Lee, H., Cho, Y., & Cho, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05325-1_27 · Cited by 2
The Impact of Multimodal Fusion methods and Debiasing Techniques for Medical VQA
Nuseir, A., & Ebner, M. (2025) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa65407.2025.11166283 · Cited by 2
Han, J., Kim, M. J., Kim, Y., & Lee, W. H. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02326-5 · Cited by 3
“Ana - tomical localization performance was evaluated using MS-CXR [ 45 ] expert annotations as ground truth, calculating IoU and Dice coefficients between predicted and reference bounding boxes.”
Recent Progress in Deep Learning for Chest X-Ray Report Generation
Salhi, M., & Akhloufi, M. A. (2026) · BioMedInformatics · doi:10.3390/biomedinformatics6010003 · Cited by 3
“MLRG uses RAD-DINO [72] for vision encoding, a CXR-BERT [73] for text encoding, and a DistilGPT2-based text generator [74].”
Cheema, M. N., Nazir, A., Harmanci, A. O., Harmanci, A. O., Cheema, Y., Masood, S., & Khokhar, F. A. (2026) · The Visual Computer · doi:10.1007/s00371-026-04385-2 · Cited by 2
“For instance, MAIRA-1 uses radiology-specific prompts for chest X-ray summaries, while Dia-LLaMA aligns segmentation maps with domain-specific reports [25, 135].”
HiMix : Hierarchical Visual-Textual Mixing Network for Lesion Segmentation
Hwang, S., Sim, J., & Kim, W. H. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00517 · Cited by 2
“We adopt CXR-BERT [3] as the default text encoder and ad-5338 ditionally evaluate BioClinicalBERT [16], PubMedBERT [11], BioBERT [26], and BioMed-RoBERTa [12].”
Zhong, X., Batmanghelich, K. N., & Sun, L. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00097 · Cited by 2
“Together with the specialized text model, they presented a self-supervised vision-language approach BioViL using contrastive learning, demonstrating state-of-the-art performance in various biomedical benchmarks.”
InfluentialYe, Y., Zhang, J., & Shi, H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8496-7_20 · Cited by 2
Hyperbolic vision language representation learning on chest radiology images
Zhang, Z., Qiao, Z., Han, L., Yang, H. B., Qian, Z., & Wu, J. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00341-x · Cited by 2
“recent years for medical image-text representation learning as baseline methods, ConVIRT [48] and BioViL [49].”
Semantic-Aware Hard Negative Mining for Medical Vision-Language Contrastive Pretraining
Li, Y., Cheng, Y., Pan, Y., He, W., Wang, Q., Feng, R., & Zhang, X. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3754991 · Cited by 2
Park, J., Choi, K., Yoon, B., Cho, H. G., & Hwang, B. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00702 · Cited by 2
“While VLMs have been extensively applied to 2D modalities such as chest X-rays [5, 26, 27], their extension to 3D medical imaging remains relatively sparse.”
Automated Brain Abnormality Detection using a Self-Supervised Text-Vision FrameworkPreprint
Wood, D. A., Guilhem, E., Kafiabadi, S., Busaidi, A. A., Hammam, A., Mansoor, N. M., Townend, M., Agarwal, S., Wei, Y., Mazumder, A., et al. (2024) · doi:10.22541/au.171571444.45086524/v1 · Cited by 2
Exploring Vision Language Pretraining with Knowledge Enhancement via Large Language Model
Tung, C., Lin, Y., Yin, J., Ye, Q., & Chen, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-67751-9_7 · Cited by 1
Sun, J., Wei, D., Xu, Z., Lu, D., Liu, H., Wang, H., Tsaftaris, S. A., McDonagh, S., Zheng, Y., & Wang, L. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3516954 · Cited by 1
“We conduct experiments on the chest radiography dataset MIMIC-CXR [20], [57] with the detection annotations provided by MS-CXR [18] and a pneumo-disease CXR (PD-CXR) dataset focused on pneumonia and pneumothorax [19].”
Generating MRI Images Based on Clinical Text Using Diffusion Models for Breast Cancer
Alali, A., & Hewahi, N. (2024) · International Conference on Data Analytics for Business and Industry (ICDABI) · doi:10.1109/icdabi63787.2024.10800586 · Cited by 1
“Self-supervised vision-language processing (VLP) models, as discussed by [8], enhance text and image integration for various applications including report generation and disease localization.”
Full fine-tuning strategy for endoscopic foundation models with expanded learnable offset parameters
Dong, M., Zheng, X., Zhang, X., Zhang, X., & Zhang, M. (2025) · Biomedical Physics & Engineering Express · doi:10.1088/2057-1976/adaec3 · Cited by 1
Anatomical Grounding Pre-Training for Medical Phrase Grounding
Zhang, W., Chandra, S. S., & Nicolson, A. (2025) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi60581.2025.10980652 · Cited by 1
“The self-supervised pre-trained models GLoRIA [17] and BioViL [9] were used for comparison.”
InfluentialPicha, S. G., Chanti, D. A., & Caplier, A. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100684 · Cited by 1
“A separate text encoder, often derived from a pretrained language model like BERT Devlin et al. [2019], processes the input text query, embedding it into a feature space aligned with the visual representations.”
InfluentialAnatomy-Aware Mixture of Experts for Medical Vision-Language Pre-training
Shi, K., Pan, H., & Zhang, K. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0027-7_10 · Cited by 1
Aksu, F., Gelardi, F., Chiti, A., & Soda, P. (2025) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi64645.2025.00079 · Cited by 2
“They employed CT-ViT’s image encoder [31], which is a vision transformer, as the image encoder and CXR-BERT [32] as the text encoder.”
Hierarchical Contrastive Learning for Precise Whole-Body Anatomical Localization in PET/CT Imaging
Gao, Y., Shu, Y., Yu, M., Chen, Y., Liu, J., Zhong, S., Zhang, W., Zhan, Y., Zhou, X. S., Wang, X., et al. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3599197 · Cited by 2
“Building on these foundations, BioViL [22], GLoRIA [23], and LoVT [24] have further advanced visual–textual alignment by incorporating global and local contrastive objectives, yielding more semantically consistent cross-modal representations for diverse medical tasks.”
Fairness and Robustness of CLIP-Based Models for Chest X-Rays
Sourget, T., Restrepo, D. S., Hudelot, C., Ferrante, E., Christodoulidis, S., & Vakalopoulou, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05870-6_2 · Cited by 1
SimCroP: Radiograph Representation Learning with Similarity-Driven Cross-Granularity Pre-training
Wang, R., Tang, F., Yao, Q., Yan, R., Zhang, X., Huang, Z., Lai, H., He, Z., Tao, X., Jiang, Z., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04971-1_53 · Cited by 1
“Additionally, we employ pre-trained CXR-BERT [2] as our text encoder and a 6-layer BERT as our text decoder.”
Li, Y., Lai, H., Zhou, X., Ming, S., Ma, W., Wei, W., & Zhou, S. K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04981-0_33 · Cited by 2
“Following CT-CLIP [12,20], we use CXR-BERT [5] as our text encoder.”
PathVG: A New Benchmark and Dataset for Pathology Visual Grounding
Zhong, C., Hao, S., Wu, J., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05169-1_44 · Cited by 1
CXR-TFT: Multi-modal Temporal Fusion Transformer for Predicting Chest X-Ray Trajectories
Arora, M., Ali, A., Wu, K., Davis, C., Shimazui, T., Alwakeel, M., Moas, V. M., Yang, P., Esper, A. M., & Kamaleswaran, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05182-0_16 · Cited by 1
Foundation Model for Medical Imaging: A Comprehensive Review
Jiao, L., Yang, J., Li, R., Liu, F., Liu, X., Chen, P., Guo, Y., Li, L., Shang, R., & Ma, W. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3618796 · Cited by 4
“The model achieved better performance than biomedical vision models such as PubMedCLIP [139] and MedCLIP [140], and surpassed BioViL [141] in radiology-specific tasks such as RSNA Pneumonia Detection [142].”
Text-Driven Medical Image Segmentation With LLM Semantic Bridge and LLM Prompt Bridge
Liu, Z., Wu, J., Fang, X., et al. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3639352 · Cited by 1
Li, J., Li, Z., Chi, J., & Bi, Z. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356033 · Cited by 1
“The latest advances in multimodal information analysis have significantly contributed to medical image diagnosis [1]–[3].”
Metadata
- Authors
- Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., Poon, H., & Oktay, O.
- Affiliations
- Microsoft (United States) (US)
- Corresponding author
- Oktay, Ozan
- ORCID
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer Nature Switzerland
- Published
- 2022
- Pages
- 1-21
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W4312533035
- Semantic Scholar
- a1aeb5442e31276b197696f49b3243112a4049ce
- Cited by
- 317
- References
- 68
- Citations by year
- 2026: 88 · 2025: 122 · 2024: 73 · 2023: 34
- FWCI
- 20.76
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- radiology reports, contrastive learning, domain-specific language models, natural language inference, phrase grounding, multimodal medical data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
