The DOI matches its Crossref record; 63 of the 68 references it cites were found in a registry or an index.
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing
Open accessAbstract
Multi-modal data abounds in biomedicine, such as radiology images and reports. Interpreting this data at scale is essential for improving clinical care and accelerating clinical research. Biomedical text with its complex semantics poses additional challenges in vision--language modelling compared to the general domain, and previous work has used insufficiently adapted models that lack domain-specific language understanding. In this paper, we show that principled textual semantic modelling can substantially improve contrastive learning in self-supervised vision--language processing. We release a language model that achieves state-of-the-art results in radiology natural language inference through its improved vocabulary and novel language pretraining objective leveraging semantics and discourse characteristics in radiology reports. Further, we propose a self-supervised joint vision--language approach with a focus on better text modelling. It establishes new state of the art results on a wide range of publicly available benchmarks, in part by leveraging our new domain-specific language model. We release a new dataset with locally-aligned phrase grounding annotations by radiologists to facilitate the study of complex semantic modelling in biomedical vision--language processing. A broad evaluation, including on this new dataset, shows that our contrastive learning approach, aided by textual-semantic modelling, outperforms prior methods in segmentation tasks, despite only using a global-alignment objective.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We note that the official MIMIC-CXR test split is highly enriched for abnormal cases, 26 hence prior studies are more common (Table 1).”
From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“By inviting a domain expert to review and adapt individual phrase outputs, our approach to qualitative evaluation of report generation differs from most existing works that primarily ask experts to rate overall report quality via a (Likert-) scale; 59 provide total error counts/ categorisation; 48,60 or engage in comparison rankings across different report candidates.61”
From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [48]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 4×
“We use a ResNet-50 [28] architecture as our image encoder and pretrain the model on MIMIC-CXR images using SimCLR [6] with domain-specific augmentations as detailed in Section 4.”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · cited 2×
“[64] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.”
BackgroundMethodInfluentialLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 5×
“A variety of self-supervised VLP approaches have been proposed towards jointly learning visual and textual representations of paired data without supervision, such as frameworks using contrastive objectives [26, 42, 60], approaches based on joint transformer architectures [40, 41, 51, 69], self-supervised VLP with wor…”
BackgroundMethodInfluentialGoldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891 · cited 6×
“network (CNN) [37] image encoder Eimg, our CXR-BERT text encoder Etxt, and projection models Pimg and Ptxt to learn representations in a joint space.”
BackgroundMethodInfluentialMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 10×
“The phases proceed as follows: (I) First, we construct a custom WordPiece [80] vocabulary of 30k to-kens from PubMed abstracts 4 (15 GB), MIMIC-III [35] clinical notes (3.5 GB), and MIMIC-CXR radiology reports (0.1 GB).”
BackgroundMethodInfluentialBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Related medical VLP work commonly uses publicly available contextual word embedding models including BioBERT [39], ClinicalBERT [2], BioClini-calBERT [2], or PubMedBERT [26].”
BackgroundMethodInfluential
Research works printed but not linked: 21
- Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13–18 July 2020, Virtual Event. Proceedings of Machine Learning Research, vol. 119, pp. 1597–1607. PMLR (13–18 Jul 2020), http://proceedings.mlr.press/v119/chen20j.html
- Graf, B., et al.: Pneumothorax and chest tube classification on chest X-rays for detection of missed pneumothorax. Machine Learning for Health (ML4H) NeurIPS Workshop: Extended Abstract (2020). https://arxiv.org/abs/2011.07353 · arXiv
- Hayat, N., Lashen, H., Shamout, F.E.: Multi-label generalized zero shot learning for the classiffcation of disease in chest radiographs. In: Machine Learning for Healthcare Conference, pp. 461–477. PMLR (2021)
- Hsu, T.M.H., Weng, W.H., Boag, W., McDermott, M., Szolovits, P.: Unsupervised multimodal representation learning across medical images and reports. Machine Learning for Health (ML4H) NeurIPS Workshop (2018). https://arxiv.org/abs/1811.08615 · arXiv
- Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456. PMLR (2015)
- Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: VisualBERT: a simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019) · arXiv
- Li, Y., et al.: Supervision exists everywhere: a data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208 (2021) · arXiv
- Liu, X., et al.: Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994 (2020) · arXiv
- Liu, Y., et al.: RoBERTa: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) · arXiv
- Logeswaran, L., Lee, H.: An efficient framework for learning sentence representations. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, 30 April - 3 May 2018, Conference Track Proceedings. OpenReview.net (2018). https://openreview.net/forum?id=rJvJXZb0W
- Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=Bkg6RiCqY7
- Lu, J., Batra, D., Parikh, D., Lee, S.: ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Advances in Neural Information Processing Systems, vol. 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019 (December), pp. 8–14 (2019). Vancouver, BC, Canada, pp. 13–23 (2019). https://proceedings.neurips.cc/paper/2019/hash/c74d97b01eae257e44aa9d5bade97baf-Abstract.html
- Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018) · arXiv
- Preechakul, K., Piansaddhayanon, C., Naowarat, B., Khandhawit, T., Sriswasdi, S., Chuangsuwanich, E.: Set prediction in the latent space. In: Advances in Neural Information Processing Systems, vol. 34 (2021)
- Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018) · arXiv
- Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28: Annual Conference on Neural Information Processing Systems 2015(December), pp. 7–12, 2015. Montreal, Quebec, Canada 28, pp. 91–99 (2015). https://proceedings.neurips.cc/paper/2015/hash/14bfa6bb14875e45bba028a21ed38046-Abstract.html
- Simard, P., Steinkraus, D., Platt, J.: Best practices for convolutional neural networks applied to visual document analysis. In: Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings, pp. 958–963. IEEE (2003)
- Wolf, T., et al.: Huggingface’s transformers: state-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019) · arXiv
- You, Y., Gitman, I., Ginsburg, B.: Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 (2017) · arXiv
- Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: Bengio, Y., LeCun, Y. (eds.) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2–4 May 2016, Conference Track Proceedings (2016). http://arxiv.org/abs/1511.07122 · arXiv
- Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747 (2020) · arXiv
Cited by
Sorted by citations
Items 201–300 of 315
Enhancing Radiography-Report foundation model via Multi-View masked contrastive learning
Li, X., Wang, M., Huang, Z., Wang, S., Liu, Z., Yu, Z. H., Hou, M., Wang, J., Yan, Y., & Liu, Y.-S. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113322 · Cited by 1
Meta-Entity Driven Triplet Mining for Aligning Medical Vision-Language Models
Yilmaz, M., Öztürk, Ş., Kara, M., Yavuz, M. T., Gümeler, E., Koç, A., & Çukur, T. (2026) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2026.3678306 · Cited by 2
TGME-Net: Text-Guided Multi-Scale Enhancement Network for Semi-Supervised Medical Image Segmentation
Li, F., Sun, C., Wang, B., et al. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463310 · Cited by 1
MAFM 3 : Modular Adaptation of Foundation Models for Multi-Modal Medical AI
Qazi, M. A., Nwadike, M., Almakky, I., Yaqub, M. K., & Saeed, N. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00341 · Cited by 1
“Inspired by these works, recent medical models such as BioViL-T [4] and MedCLIP [34] adapt contrastive learning to radiology reports and imaging data, demonstrating that language can provide crucial contextual guidance for medical tasks.”
Li, J., Liu, C. H., Bai, W., Liu, M., Arcucci, R., Bercea, C. I., & Schnabel, J. A. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00233 · Cited by 1
“RadVLM, a 7B model, is trained on 1,022,742 image-instruction pairs curated from MIMIC-CXR, CheX-pert [18], CheXpert-Plus [10], MS-CXR [7], PadChest-GR [9], and VinDr-CXR [34].”
Wan, H., Xu, W., Zou, Y., Zeng, J., & Wei, X. (2026) · IEEE Signal Processing Letters · doi:10.1109/lsp.2026.3697688 · Cited by 1
“2, VTMedSeg consists of a geometry-guided text generator (Geo-TG), a transformer backbone PVT-v2-B2 [17], [18], [19], BiomedVLP [20], a concept-aware cross-modal fusion module (CCFM), and an adaptive decoder.”
Language-Guided Segmentation of Medical Images: A Review of Foundation Models
Qamar, S. (2026) · Bioengineering · doi:10.3390/bioengineering13070803 · Cited by 1
“…pairs to overcome the small size of paired medical data [14], GLoRIA introduced global-local contrastive learning to align text tokens with image regions [16], and BioViL/BioViL-T refined the approach for chest radiology, with BioViL-T modeling temporal change across follow-up scans [17,18].”
A vision transformer based leakage free concept bottleneck model for clinical skin lesion diagnosis
Kandpal, K. C., & Verma, P. (2026) · Discover Computing · doi:10.1007/s10791-026-10315-6 · Cited by 1
“By mapping clinical concepts to final diagnoses, they aim to create a transparent reasoning pipeline [16, 24, 25].”
Dai, W., Wang, C., Ghosh, S., & Batmanghelich, K. N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37517-9_22 · Cited by 1
MedTri: Structured Medical Report Normalization for Enhanced Vision–Language Pretraining
Chu, Y., Ma, X., Jin, X., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38059-3_35 · Cited by 1
In-Depth Exploration of the Advantages of Neural Networks in English Machine Translation
Li, L. (2024) · International Conference on Electronics and Sustainable Communication Systems (ICESC) · doi:10.1109/icesc60852.2024.10689832 · Cited by 1
TiBiX: Leveraging Temporal Information for Bidirectional X-Ray and Report Generation
Sanjeev, S., Maani, F. A., Abzhanov, A., Papineni, V. R. K., Almakky, I., Papież, B. W., & Yaqub, M. K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72744-3_17 · Cited by 3
Enhancing visual and semantic alignment of multimodal large models in medical images
Liu, Q., Fang, Y., & Liu, X. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07859-7 · Cited by 1
“BioViL [14] leverages biomedical images and their corresponding radiology reports, using only global image–text representations for contrastive matching.”
A Survey of Generalization and Adaptation in Medical Imaging Foundation ModelsPreprint
Ying, H., Lia, Y., & Fu, Z. (2025) · Preprints.org · doi:10.20944/preprints202507.0942.v1 · Cited by 2
Automated Brain Abnormality Detection using a Self-Supervised Text-Vision FrameworkPreprint
Wood, D. A., Guilhem, E., Kafiabadi, S., Busaidi, A. A., Hammam, A., Mansoor, N. M., Townend, M., Agarwal, S., Wei, Y., Mazumder, A., et al. (2024) · doi:10.22541/au.171571444.45086524/v2
A self-supervised framework for abnormality detection from brain MRIPreprint
Wood, D. A., Guilhem, E., Kafiabadi, S., Busaidi, A. A., Dissanayake, K., Hammam, A., Mansoor, N. M., Townend, M., Agarwal, S., Wei, Y., et al. (2024) · doi:10.22541/au.171900128.89392883/v1
TextCAVs: Debugging Vision Models Using Text
Nicolson, A., Gal, Y., & Noble, J. A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-77610-6_10
Improving CLIP for Biomedical Retrieval: A Multi-Margin Contrastive Loss Approach
Shakya, E., Rayakota, H. P., & Huang, P.-C. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825400
Bilateral network with text guided aggregation architecture for lung infection image segmentation
Pan, X., Mei, H., Zheng, J., & Zheng, H. (2025) · Biomedical Physics & Engineering Express · doi:10.1088/2057-1976/adb290
Multi-Modality Knowledge Transfer for Generalizable Chest Radiograph Interpretation
Hou, X., Li, Y., Mei, Y., Liu, D., & Wang, S. (2024) · International Conference on Biomedical and Bioinformatics Engineering (ICBBE) · doi:10.1145/3707127.3707128
Jakobsen, T. S. T., Cristòbal-Cóppulo, E., Rasmussen, S., & Benros, M. E. (2025) · medRxiv · doi:10.1101/2025.03.07.25323558
Liu, X., & Li, C. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-6963-9_5
Enhancing Clinical Reasoning in Medical Vision-Language Model through Structured PromptsPreprint
Prakash, K. D., Kim, K., & Han, Y. (2025) · medRxiv · doi:10.1101/2025.07.31.25331901
“Models such as BioViL [13] and MedCLIP [14] used contrastive learning to match radiological images with corresponding reports, enhancing visual representations in clinical settings.”
Temporal multimodal transformer for automated radiology reporting in traumatic brain injuries
Bouslimi, R., Medini, M., & Fradj, M. B. (2025) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-025-00579-1
Lie Group-Based Multimodal Dynamic Attention for Chest X-ray Diagnosis
Fang, T., & Liu, G. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-0011-6_8
Knowledge-Enhanced Hyperbolic Language-Image Pretraining for Zero-Shot Learning
Han, L., Qiao, Z., Zhen, X., Gao, J., & Qian, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-96625-5_14
Eliminating Non-Overlapping Semantic Misalignment for Cross-Modal Medical Retrieval
Wei, Z., Hou, Z., & Zhou, X. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3602392
“However, these methods primarily focus on global feature alignment, which often proves insufficient for capturing the fine-grained semantic details crucial for medical data [14], particularly local pathological features.”
Enhancing Radiology Report Interpretation through Modality-Specific RadGraph Fine-Tuning
Guan, H., Dai, Y., Afyouni, S., Kain, A., Hsu, W.-C., Cheng, J., Yao, S., Wang, Y., Wu, J., Pulakhandam, R., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04981-0_21
IP-CRR: Information Pursuit for Interpretable Classification of Chest Radiology Reports
Ge, Y., Chan, K. H. R., Messina, P., & Vidal, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05185-1_30
Dual-Guided 3D Liver CT Image Generation for Medical Analysis
Song, Z., Zhu, J., Huang, W., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06624-4_13
Injarabian, D. A., Ariva, J., Šuvalov, H., & Fishman, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05479-1_10
Improving Cross-Modal Consistency in Multimodal Biomedical AI: A Comprehensive Review of Techniques
Basak, A., Choudhury, S. N., & Garai, P. (2025) · International Conference on Computing, Intelligence, and Application (CIACON) · doi:10.1109/ciacon65473.2025.11189769
“Multimodal biomedical information processing, especially combining chest X-ray images with radiology reports, rep-Authorized MedCLIP [10], BioViL [7], BioMedCLIP [9], CXR-CLIP [30], GLoRIA [12], CONVIRT [13], MedKLIP [11], KAD [17], LViT [19] Cross Entropy Loss CheXNet [14], CheXZero [15], TransVW…”
REMix: Refinement-Enhanced Visual-Textual Mixing for Lesion Segmentation
Hwang, S., Sim, J., & Kim, W. H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07502-4_5
Specialised or Generic? Tokenization Choices for Radiology Language Models
Warr, H., Xu, W., Anthony, H., Ibrahim, Y., McGowan, D. R., & Kamnitsas, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07502-4_8
Hao, X., Liu, J., & Yang, C. (2025) · Frontiers in Physiology · doi:10.3389/fphys.2025.1661960
Cui, J., Zeng, X., Zeng, P., Liu, B., Wu, X., Deng, X., Zhou, J., Wang, Y., & Shen, D. G. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3625572
Gong, Z., He, L., Zhou, N., Liao, P., Yang, H., Wu, Z., Chen, H., & Zhang, Y. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228394
MultiModal craniocerebral diagnose based on 3D CT and image reports
He, W., Chen, Q., Gao, W., Xu, H., & Wei, Z. X. (2025) · Scientific Reports · doi:10.1038/s41598-025-30249-1
“Representative frameworks include ConVIRT 38 , which first demonstrated that aligning chest images with free-text reports yields robust visual encoders under limited labels , GLoRIA 39 , which extends the paradigm by jointly optimizing global–local alignment between image regions and report tokens to improve retrieval…”
Huang, Y., Zuo, Y., Xu, M., Chen, K., Wu, M., Zhang, C., Ma, Z., & Guo, J. H. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3639888
Longjiang, E., Cai, Y., Zhao, B., Zhang, M., Lin, J., & Wang, T. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00402-x
“2020; Boecking Et al.”
Chest X-ray foundation models: A survey and future directions
Wang, A., Yang, Z., Rais-Bahrami, S., & Yan, P. (2025) · Meta-Radiology · doi:10.1016/j.metrad.2025.100203 · Cited by 1
Vision-Language Sliding Cross Attention for Text-Guided Pneumonia Segmentation
Yao, F., Zhang, X., Jiang, X., Xiao, Y., Fan, L., Zhou, S. K., & Liu, S. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-09569-5_23
Ontology-Based Concept Distillation for Radiology Report Retrieval and Labeling
Nützel, F., Dombrowski, M., & Kainz, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-09513-8_52
Radixpert: A Multimodal Approach to Medical Imaging
Farooqi, M. A. R., Chaudhry, A. A., Ghaffar, M. Y., Perwaiz, N., & Kiani, H. M. (2025) · International Conference on Frontiers of Information Technology (FIT) · doi:10.1109/fit67061.2025.11333658
“Building on this, [9] proposed BioViL and its successor BioViL-T [10], which incorporate temporal data (prior images and reports) into the pre-training objective, enhancing the model’s ability to reason about disease progression and text semantics.”
Medical Vision Language Model With Multi-granularity Alignment and Learning Data Augmentation
Gao, D., Hua, Y., Zhang, W., & Song, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5634-2_34
Cai, J., Lian, S., Pan, D., Chen, G., Li, L., Luo, Z., & Li, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5631-1_3
MMKPL-Seg: Knowledge Proxy Learning with Missing Modalities for Medical Image Segmentation
Ding, S., Fan, M., & Li, M. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356586
“In our implementation, the channel dimensions are configured as: Fortext modality processing, we adopt CXR-BERT [24],”
BiFormer-Seg: Dual-Branch Visual Encoding and Text-Guided Decoding for Radiological Segmentation
Yang, C., Shi, J., Gong, H., Duan, Y., & Yun, L. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356458
“PubMedCLIP [51] and BioViL [52] further introduced domain-specific corpora and radiological ontologies to improve biomedical alignment.”
Classifying Pulmonary Diseases Using Small Multimodal Model on X-Ray Images and Reports
Thai, M. K., & Tran, D.-K. (2025) · RIVF International Conference on Computing and Communication Technologies (RIVF) · doi:10.1109/rivf68649.2025.11365036
“Huang et al. developed GLoRIA , which aligns local image regions with textual tokens for fine-grained supervision [12], while Boecking et al. released BioViL-T , a transformer-based model achieving state-of-the-art performance in zero-shot and fine-tuned radiology tasks [13].”
Distribution-Based Masked Medical Vision-Language Model Using Structured Reports
Gowda, S. N., Zhang, R., Gu, X., Weng, Y., & Yang, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-17611-0_1
Goetze, M., Eschweiler, D., Huang, B., Müller‐Franzes, G. A., Ramirez, C., Hess, M., Goldermann, L., Majumdar, S., & Truhn, D. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00701
“The text encoder is initialized to BiomedVLP-CXR-BERT-general [3] and follows the BERT architecture, as also used by CT-CLIP.”
Fine-Grained Visual Enhancement for Multimodal Biomedical Question Answering
Li, R. (2025) · International Conference on Machine Learning, Computational Intelligence and Pattern Recognition (MLCIPR) · doi:10.1109/mlcipr68329.2025.11407346
“Previous work has also demonstrated that pre-training fine-tuned with domain knowledge yields significant results in biomedical natural language processing (NLP) [15, 16] and biomedical vision-language (VL) tasks [17, 18].”
AHMED, M. S., Hamid, M. A., Monowar, M. M., & Yousuf, M. A. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-01914-2
Dai, X., Rajendran, P., Jiang, Z., et al. (2026) · SPIE Medical Imaging: Imaging Informatics for Healthcare, Research, and Applications · doi:10.1117/12.3087908
“Multimodal learning approaches that combine imaging with structured clinical variables or free-text reports have shown promise in improving performance and robustness in medical imaging tasks [14, 15].”
Study of the multimodal understanding of vision-language transformer models
Salin, E. (2023) · theses.fr (ABES) · doi:10.70675/2beca704z5d3bz421dzbf61z61e00610aaeb
Multimodal Contrastive Learning for Clinical Data Alignment via Fat Composition Representations
Sanchez-Sos, B., Tomé-Fernández, M., Beviá-Ballesteros, I., et al. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-16992-1_5
Symptom-Aware Adaptation of CLIP for Few-Shot Medical Image Classification
Li, Z., Wang, W., Liu, C., He, J., Zhao, Y., Zhai, Y., & Tang, R. (2026) · IEEE Signal Processing Letters · doi:10.1109/lsp.2026.3685174
“Notably, while biomedical-specific VLMs [14], [15], [16], [17], [18] show stronger adaptability in medical tasks, existing studies adapting general-purpose CLIP to healthcare [19] still suffer from critical gaps: narrow task coverage, over-reliance on large-scale annotated datasets, and limited…”
Rukon, M. M. R., & Islam, N. (2026) · IEEE SoutheastCon · doi:10.1109/southeastcon63549.2026.11476472
Eom, T. S., & Song, B. C. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464520
Salient Diagnostic Value Perception For Preoperative Posterior Fossa Tumor Diagnosis
Chen, Z., Li, P., Zhang, Z., Wu, S., Zhang, J., Hu, C., Ou, Y., & Sun, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460465
DPA-HiVQA: Enhancing Structured Radiology Reporting with Dual-Path Cross-Attention
Do, N. T., Quang, M. N., & Pham, H. V. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8050113
“Text Encoder : CXR-BERT [36], which is pre-trained on massive free-text radiology reports aligned with chest X-ray images, encoding the current question with progressive history ( T ∈ R L × 768 ).”
InfluentialVision Language Models for Medical Imaging
Tyagi, V. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7315-7.ch008
Uncertainty-Aware Vision-Language Segmentation for Medical Imaging
Das, A., Rachamalla, T., Biswas, K., Roy, S. K., & Verma, V. K. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00819 · Cited by 1
“We utilize two pre-trained models to encode the input modalities: ConvNeXt-Tiny [37] as the visual encoder and BioViL CXR-BERT [38] as the text encoder.”
Bhat, S., Mansoor, A., Georgescu, B., Zinnen, M., Sahu, P., Panambur, A. B., Ghesu, F. C., Grbić, S., & Maier, A. (2026) · Scientific Reports · doi:10.1038/s41598-026-52235-x · Cited by 1
“On the other hand, BioViL 35 presents that “principled textual semantic modeling” can substantially improve contrastive learning in SSL visual language processing.”
SLG-Flow: A Unified Framework from Semantic Labeling to Visual Generation
Feng, F., Wang, L., Chen, J., Wang, L., & Wang, L. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-8978-4_29
Jiang, J. C., Zhou, Q., & He, C. (2026) · Symmetry · doi:10.3390/sym18050859
“…geometric information from the ground-truth mask Y is incorporated to construct a posterior distribution: The mask is downsampled and flattened into f y , and the posterior parameters are obtained by Latent samples are then generated using the reparameterization trick: where ϵ ∼ N ( 0 , I ) .”
TG-MUNet: A Lightweight Text-Guided Mamba Unet for Semi-Supervised Medical Image Segmentation
Li, F., Sun, C., Wang, B., et al. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810688
Learning Directional Semantic Transitions for Longitudinal Chest X-Ray Analysis
Hu, Z., Yang, Z., Wang, G., Syeda-Mahmood, T. F., Burade, A., Kalra, M. K., & Yan, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38172-9_36
Medical Vision-Language Models: Existing Technologies, Clinical Applications and Future Directions
Zou, L., Ma, M., Li, J. B., Chen, H., & Peng, S. (2026) · Sensors · doi:10.3390/s26133998
“GLoRIA [50] 2021 CheXpert AUC = 75.0 - MedCLIP [82] 2022 CheXpert AUC = 74.4 - CheXzero [89] 2022 CheXpert AUC = 88.9 - BiomedCLIP [90] 2023 CheXpert AUC = 67.7 - MedKLIP [85] 2023 CheXpert AUC = 87.9 - KAD [91] 2023 CheXpert AUC = 90.5 - BioViL [92] 2023 CheXpert AUC = 78.9 - CARZero [44] 2024”
InfluentialGeneralized Medical Phrase Grounding
Zhang, W., Chandra, S. S., & Nicolson, A. (2026) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2026.3707568 · Cited by 2
“The MS-CXR benchmark [8] was, in fact, introduced to evaluate this emergent localisation capability against ground-truth boxes, rather than as a primary training set for grounding models.”
Residual Adapter Learning with Attention Pooling for Medical Visual Question Answering
Nuseir, A., & Ebner, M. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-30612-8_32
Zheng, B., & Xu, C. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3507-0_24
Artificial intelligence-assisted imaging technologies for disease diagnosis
Coelho, L. P. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-43934-6.00007-9
Spot the Difference: Bilateral Contrastive Representation Learning for Nodule Classification
Haynes, S. C., Mekala, M. S., & Elyan, E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31927-2_20
TRACE: Temporal Radiology with Anatomical Change Explanation for Grounded X-Ray Report Generation
Aranya, O. F. M. R. R., & Desai, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31933-3_7 · Cited by 3
Improving Negation Understanding in Medical Vision-Language Models via Contrastive Fine-Tuning
Vu, J., & Sheshappanavar, S. V. (2026) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw68408.2026.00046
“While there have been many adaptations of CLIP for use in medical settings, like CXR-CLIP [22], GLORIA [6], MedKlip [21], BioViL [2], MLIP [10], Xlip [20], they often overlook the importance of ensuring accuracy in the con-1 text of negation.”
Rohilla, H., Singh, A. P., & Ratna, S. (2026) · International Conference on Data Science for Cyber-Physical Systems Resilience using Advanced Applications (ICDCA) · doi:10.1109/icdca69396.2026.11620238
“Vision–language models such as BioViL further improve alignment between image features and radiology text [18].”
Pseudocode Generation from Clinical Protocol Flowchart using Large Vision-Language Models
Zhou, L., Luo, X., Chandra, C., & Zhang, Z. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00113
“Additionally, the embedding of medical ontologies or structured domain knowledge in LVLM could enhance their contextual understanding of clinical concepts [3], improving the accuracy of pseudocode generation.”
Li, H., Guo, R., & Li, C. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00257
“The text encoder uses BiomedVLP-CXR-BERT [28] with fixed token length 24.”
Li, H., Xing, W., & Li, C. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00068
“We adopt BiomedVLP-CXR-BERT [37], which has been specifically designed to maximize text semantics for biomedical vision-language tasks.”
A New Generative Generalised Zero‐Shot Recognition for Chest X‐Ray Images
Zhu, H., Zhang, G., & Wang, Y. (2026) · IET Image Processing · doi:10.1049/ipr2.70431
SA-LanSeg: Bridging semi-automatic interaction and language guidance for medical image segmentation
Huang, Z., Peng, Y., Xu, M.-M., Jung, Y., Ahn, E., Kim, J., & Bi, L. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114665
Comparative Study of CNN, Hybrid, and Transformer Architectures in Medical Image Classification
Elkholy, D. A., Shehata, M. S., & Braun, J. W. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02154-0
Wang, Y., Yang, J., & Zhang, H. (2026) · Fractal and Fractional · doi:10.3390/fractalfract10090597
Jandoubi, B., & Akhloufi, M. A. (2026) · Bioengineering · doi:10.3390/bioengineering13090993
“In parallel, clinical language models [12,13] have been used to extract information from clinical notes and radiology reports.”
Integration of artificial intelligence and multi-omics for precision medicine
Marei, H. E. S., Cenciarelli, C., & Vagni, D. (2026) · Functional & Integrative Genomics · doi:10.1007/s10142-026-02024-6
Rahman, H., Mahmood, T., & Raza, A. (2026) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/u145014357
Khalid, O., Nadeem, S., Farooq, B., & Saeed, H. (2024) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/s049146327
Prosima: Reliable Semantic-Aware Medical Image Recovery Via Lightweight Distributed Ledger
Rasheed, M., & Al-Mamun, A. (2026) · IEEE International Conference on Decentralized Applications and Infrastructures (DAPPS) · doi:10.1109/dapps69803.2026.00014
“Semantic encoders such as CLIP [28] and domain-specific models like BioViL [29], [30] and MedCLIP [28], [31] capture anatomical semantics beyond pixel-level losses, but are primarily used for classification or retrieval and rarely integrated into reconstruction pipelines with explicit semantic…”
Anatomy-Aware Hierarchical Contrastive Hashing for Efficient Radiograph Classification and Retrieval
Ahmad, J., Almetnawy, H., Orabi, A., Khan, M., Farman, H., & Ullah, F. (2026) · Technologies · doi:10.3390/technologies14090566
Bercean, B. A., Medelean, R., Tenescu, A., et al. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02196-4
MedCure: Medical Data Curation for Efficient Vision–Language Pretraining
Wang, C., Liu, F., Liu, Y., Chen, Y., Tian, Y., & Fang, L. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3730839
“CXR-CLIP [32] extended this paradigm to both image-to-image and text-to-text contrastive objectives, while MRM [33] together with BioViL [34] leveraged masked modeling techniques to reconstruct masked components in both CXR images and reports.”
Frequency-semantic Joint Calibration for Text-guided Medical Image Segmentation
Mo, J., Du, S., Chen, R., et al. (2026) · doi:10.1109/mlmc69743.2026.11686813
“Language Encoder: To process clinical text prompts T ∈ L of length L , we utilize the domain-specific pre-trained CXR-BERT model [24].”
Bidirectional Anatomy-Aware Post-Training for Longitudinal Chest X-Ray Progression Modeling
Chen, Y., Wang, H., To, M., Zhang, J., Wu, Q., & Xie, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38172-9_8
PCRP: Progressive Coarse-to-Fine Refinement for Pathology Grounding
Peng, L., Li, C., Dong, S., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38224-5_36
Long, J., Zeng, X., & Wang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38082-1_59
Learning Role-Conditioned Alignment for Medical Image Referring Segmentation
Li, K., Wang, F., Zhou, F., Zhao, H., Xu, X., Meng, Y., Zhao, Y., & Zheng, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38082-1_27
Owais, M., Shafay, M., Velayudhan, D., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38062-3_48
TraceCXR: Verifiable Chest X-Ray Reports via Evidence-Addressable Graph Prompting
Dong, H., Sun, C., Yan, H., Hu, W., & Du, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38062-3_57
Metadata
- Authors
- Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., Poon, H., & Oktay, O.
- Affiliations
- Microsoft (United States) (US)
- Corresponding author
- Oktay, Ozan
- ORCID
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer Nature Switzerland
- Published
- 2022
- Pages
- 1-21
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W4312533035
- Semantic Scholar
- a1aeb5442e31276b197696f49b3243112a4049ce
- Cited by
- 317
- References
- 68
- Citations by year
- 2026: 88 · 2025: 122 · 2024: 73 · 2023: 34
- FWCI
- 20.76
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- radiology reports, contrastive learning, domain-specific language models, natural language inference, phrase grounding, multimodal medical data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
