The DOI matches its Crossref record; 63 of the 68 references it cites were found in a registry or an index.
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing
Open accessAbstract
Multi-modal data abounds in biomedicine, such as radiology images and reports. Interpreting this data at scale is essential for improving clinical care and accelerating clinical research. Biomedical text with its complex semantics poses additional challenges in vision--language modelling compared to the general domain, and previous work has used insufficiently adapted models that lack domain-specific language understanding. In this paper, we show that principled textual semantic modelling can substantially improve contrastive learning in self-supervised vision--language processing. We release a language model that achieves state-of-the-art results in radiology natural language inference through its improved vocabulary and novel language pretraining objective leveraging semantics and discourse characteristics in radiology reports. Further, we propose a self-supervised joint vision--language approach with a focus on better text modelling. It establishes new state of the art results on a wide range of publicly available benchmarks, in part by leveraging our new domain-specific language model. We release a new dataset with locally-aligned phrase grounding annotations by radiologists to facilitate the study of complex semantic modelling in biomedical vision--language processing. A broad evaluation, including on this new dataset, shows that our contrastive learning approach, aided by textual-semantic modelling, outperforms prior methods in segmentation tasks, despite only using a global-alignment objective.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We note that the official MIMIC-CXR test split is highly enriched for abnormal cases, 26 hence prior studies are more common (Table 1).”
From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [26]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“By inviting a domain expert to review and adapt individual phrase outputs, our approach to qualitative evaluation of report generation differs from most existing works that primarily ask experts to rate overall report quality via a (Likert-) scale; 59 provide total error counts/ categorisation; 48,60 or engage in comparison rankings across different report candidates.61”
From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [48]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 4×
“We use a ResNet-50 [28] architecture as our image encoder and pretrain the model on MIMIC-CXR images using SimCLR [6] with domain-specific augmentations as detailed in Section 4.”
BackgroundMethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · cited 2×
“[64] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.”
BackgroundMethodInfluentialLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 5×
“A variety of self-supervised VLP approaches have been proposed towards jointly learning visual and textual representations of paired data without supervision, such as frameworks using contrastive objectives [26, 42, 60], approaches based on joint transformer architectures [40, 41, 51, 69], self-supervised VLP with wor…”
BackgroundMethodInfluentialGoldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891 · cited 6×
“network (CNN) [37] image encoder Eimg, our CXR-BERT text encoder Etxt, and projection models Pimg and Ptxt to learn representations in a joint space.”
BackgroundMethodInfluentialMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 10×
“The phases proceed as follows: (I) First, we construct a custom WordPiece [80] vocabulary of 30k to-kens from PubMed abstracts 4 (15 GB), MIMIC-III [35] clinical notes (3.5 GB), and MIMIC-CXR radiology reports (0.1 GB).”
BackgroundMethodInfluentialBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Related medical VLP work commonly uses publicly available contextual word embedding models including BioBERT [39], ClinicalBERT [2], BioClini-calBERT [2], or PubMedBERT [26].”
BackgroundMethodInfluential
Research works printed but not linked: 21
- Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13–18 July 2020, Virtual Event. Proceedings of Machine Learning Research, vol. 119, pp. 1597–1607. PMLR (13–18 Jul 2020), http://proceedings.mlr.press/v119/chen20j.html
- Graf, B., et al.: Pneumothorax and chest tube classification on chest X-rays for detection of missed pneumothorax. Machine Learning for Health (ML4H) NeurIPS Workshop: Extended Abstract (2020). https://arxiv.org/abs/2011.07353 · arXiv
- Hayat, N., Lashen, H., Shamout, F.E.: Multi-label generalized zero shot learning for the classiffcation of disease in chest radiographs. In: Machine Learning for Healthcare Conference, pp. 461–477. PMLR (2021)
- Hsu, T.M.H., Weng, W.H., Boag, W., McDermott, M., Szolovits, P.: Unsupervised multimodal representation learning across medical images and reports. Machine Learning for Health (ML4H) NeurIPS Workshop (2018). https://arxiv.org/abs/1811.08615 · arXiv
- Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456. PMLR (2015)
- Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: VisualBERT: a simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019) · arXiv
- Li, Y., et al.: Supervision exists everywhere: a data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208 (2021) · arXiv
- Liu, X., et al.: Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994 (2020) · arXiv
- Liu, Y., et al.: RoBERTa: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) · arXiv
- Logeswaran, L., Lee, H.: An efficient framework for learning sentence representations. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, 30 April - 3 May 2018, Conference Track Proceedings. OpenReview.net (2018). https://openreview.net/forum?id=rJvJXZb0W
- Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=Bkg6RiCqY7
- Lu, J., Batra, D., Parikh, D., Lee, S.: ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Advances in Neural Information Processing Systems, vol. 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019 (December), pp. 8–14 (2019). Vancouver, BC, Canada, pp. 13–23 (2019). https://proceedings.neurips.cc/paper/2019/hash/c74d97b01eae257e44aa9d5bade97baf-Abstract.html
- Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018) · arXiv
- Preechakul, K., Piansaddhayanon, C., Naowarat, B., Khandhawit, T., Sriswasdi, S., Chuangsuwanich, E.: Set prediction in the latent space. In: Advances in Neural Information Processing Systems, vol. 34 (2021)
- Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018) · arXiv
- Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28: Annual Conference on Neural Information Processing Systems 2015(December), pp. 7–12, 2015. Montreal, Quebec, Canada 28, pp. 91–99 (2015). https://proceedings.neurips.cc/paper/2015/hash/14bfa6bb14875e45bba028a21ed38046-Abstract.html
- Simard, P., Steinkraus, D., Platt, J.: Best practices for convolutional neural networks applied to visual document analysis. In: Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings, pp. 958–963. IEEE (2003)
- Wolf, T., et al.: Huggingface’s transformers: state-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019) · arXiv
- You, Y., Gitman, I., Ginsburg, B.: Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 (2017) · arXiv
- Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: Bengio, Y., LeCun, Y. (eds.) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2–4 May 2016, Conference Track Proceedings (2016). http://arxiv.org/abs/1511.07122 · arXiv
- Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747 (2020) · arXiv
Cited by
Sorted by citations
Items 301–315 of 315
Mahapatra, D., Das, A., Roy, S., Pandey, M., Dhar, J., Zaidi, N., Ge, Z., Bozorgtabar, B., Reyes, M., & Razzak, I. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38062-3_46
Multimodal learning for imbalanced multi-label chest X-ray classification with radiology reports
Eddine, L. B., Gallego, A., & Pertusa, A. (2026) · Pattern Analysis and Applications · doi:10.1007/s10044-026-01774-0
“For the text modality, we consider trans-former-based language models, namely BERT Base [33], BioBERT [34], and CXR-BERT [35] in both its General and Specialized variants.”
AtlasCT: Report-Conditioned 3D CT Synthesis with a Learnable Population Atlas Prior
Hou, J., Moraros, J., & Wang, S. (2026) · International Conference on Computer Vision and Data Mining (ICCVDM) · doi:10.1109/iccvdm71812.2026.11709283
“Reports are encoded with a domain-specific CXR-BERT encoder [11] into features f text , and a bidirectional text-volume fusion [12] exchanges information layer-wise.”
Liu, H., Wei, D., Zheng, Y., Wu, X., & Wang, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37531-5_21
Cantú, E. D., Acharya, M., Berend, J., et al. (2026) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3776591.3836730
Disease-Aware Visual Token Memory for Chest X-Ray Report Generation with VLMs
Fichadia, R. H. (2026) · IEEE World Conference on Applied Intelligence and Computing (AIC) · doi:10.1109/aic70896.2026.11716964
“Domain-specific pretraining is preferred over a generic CLIP backbone because biomedical contrast (e.g. BioViL [9], CheXzero [10]) yields features markedly more sensitive to pathology cues than natural-image features. b) Disease-conditioned token distillation.”
AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretation
Hashmi, A. U. R., Saeed, N., & Lippert, C. (2026) · doi:10.1109/cvprf71909.2026.00577
“The details of each task are provided in the supplementary material section S3.”
R3-Gen: Enhancing medical report generation with reasoning-reinforced fine-tuning
Xiao, T., Qiu, Y., Yang, H., & Wang, Z. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.115111
CXR-CoRe: Two-Stage View- and Section-Aware Study-Level Chest X-ray Report Retrieval
Tian, Y., & Ma, J. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02377-1
Evaluating a Multimodal Foundation Model for Glaucoma Classification from Fundus Images
Serio, F. D., Gravina, M., Moscato, V., Carlo, S., Gonzalo‐Martín, C., & Garcia-Pedrero, A. (2026) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms69103.2026.00096
“Models such as BioViL [17], MedCLIP [18], and MedFLAMINGO [19] leverage large-scale paired image-text data, contrastive objectives, or caption supervision to learn generalizable representations.”
A SAM2 and VLM-Based Automatic Radiological Reporting System for Pneumothorax Detection
Tunçer, S., Yıldırım, M., & Yıldırım, Ö. (2026) · Biomedicines · doi:10.3390/biomedicines14102312
Foundation Models and Vision-Language Models in Medical Imaging: A Review
Mohsen, S., Lulu, L. M., Eassa, M., Raafat, R., Tawdrous, M. I., & Ibrahim, A. F. (2026) · Computers · doi:10.3390/computers15100707
Concept-Enhanced Hypergraph-Guided Medical Image Retrieval Using CUI Semantics
Saad, M., Emam, M. H., Sani, M. I., & Momen, S. (2026) · IEEE Access · doi:10.1109/access.2026.3743341
“…used large-scale scientific image-text pairs [14]; MedCLIP learned from unpaired images and text [15]; BioViL exploited radiology text semantics [16]; GLoRIA learned global and local image-text representations [17]; and CheXzero showed strong zero-shot pathology detection without manual…”
BILATERAL VISION-LANGUAGE FRAMEWORK WITH CROSS-LATERAL ATTENTION FOR MAMMOGRAPHY
Abdikenov, B., Imasheva, A., & Zhaksylyk, T. (2026) · Scientific Journal of Astana IT University · doi:10.37943/hlva7583
Cai, Y. (2026) · International Conference on Artificial Intelligence Technologies and Applications (ICAITA) · doi:10.1109/icaita71179.2026.11737020
“BioViL[1] showed that stronger biomedical vision–language representations can be learned via improved text semantics and domain-specific objectives, but it models longitudinal studies as independent samples, limiting its ability to capture disease progression.”
Metadata
- Authors
- Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., Poon, H., & Oktay, O.
- Affiliations
- Microsoft (United States) (US)
- Corresponding author
- Oktay, Ozan
- ORCID
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer Nature Switzerland
- Published
- 2022
- Pages
- 1-21
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W4312533035
- Semantic Scholar
- a1aeb5442e31276b197696f49b3243112a4049ce
- Cited by
- 317
- References
- 68
- Citations by year
- 2026: 88 · 2025: 122 · 2024: 73 · 2023: 34
- FWCI
- 20.76
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- radiology reports, contrastive learning, domain-specific language models, natural language inference, phrase grounding, multimodal medical data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
