Multimodal transformers for joint diagnosis and retrieval from chest X-rays and radiology reports
Open accessAbstract
Background: Radiological diagnosis frequently hinges on the joint interpretation of medical images and their corresponding narrative reports. However, most existing automated approaches treat these modalities in isolation, thereby overlooking their rich semantic interplay—an essential element for robust clinical understanding. Methods: We introduce RadFuse , a unified multimodal transformer framework that concurrently analyzes chest radiographs and associated free-text radiology reports. The model integrates a Vision Transformer (ViT) for visual representation learning and ClinicalBERT for processing clinical narratives. These modalities are fused through stacked self-attention and bidirectional cross-attention mechanisms, enabling fine-grained semantic alignment. RadFuse is optimized via a composite objective that combines binary cross-entropy for multi-label disease classification and a contrastive InfoNCE loss for image-text retrieval. Results: We evaluate RadFuse on two benchmark datasets— MIMIC-CXR and IU X-ray —across two key tasks: disease classification and cross-modal retrieval. On MIMIC-CXR, RadFuse achieves an AUC of 0.847 and a Recall@10 of 78.2 , outperforming prior unimodal and shallow multimodal baselines. Ablation studies demonstrate the critical contribution of cross-attention and contrastive training to the model’s performance. Conclusion: RadFuse provides a scalable and interpretable solution for vision-language integration in clinical settings. Its unified architecture enables multi-task learning, facilitates robust semantic grounding, and sets a new benchmark for multimodal modeling in radiology AI.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Attention-based transformer architectures can discover clinically relevant feature interactions directly from structured EHR data without explicit graph construction.
“Nevertheless, recently attention-based transformer architectures have shown that they can discover clinically relevant feature interactions directly from structured EHR data even without the need for explicit graph construction43.”
From K-STAMM: a knowledge-enhanced spatial - temporal attention model with multimodal fusion for pneumonia prediction (2026) · cites this paper as [43]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · see citing works
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · see citing works
Preparing a collection of radiology examinations for distribution and retrieval
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv080 · Cited by 1,164 · see citing works
Clinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910 · see citing works
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text
Wang, Z., Wu, Z., Agarwal, D., & Sun, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.256 · Cited by 838 · see citing works
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618 · see citing works
Huang, S.-C., Shen, L., Lungren, M. P., & Yeung, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00391 · Cited by 489 · see citing works
Contrastive Learning of Medical Visual Representations from Paired Images and TextPreprint
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., & Langlotz, C. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00747 · Cited by 465 · see citing works
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392 · see citing works
When Radiology Report Generation Meets Knowledge Graph
Zhang, Y., Wang, X., Xu, Z., Yu, Q., Yuille, A., & Xu, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6989 · Cited by 348 · see citing works
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing
Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20059-5_1 · Cited by 317 · see citing works
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
Bannur, S., Hyland, S., Liu, Q., Pérez-García, F., Ilse, M., Castro, D. C., Boecking, B., Sharma, H., Bouzid, K., Thieme, A., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01442 · Cited by 227 · see citing works
Chest X-ray analysis empowered with deep learning: A systematic review
Meedeniya, D., Kumarasinghe, H., Kolonne, S., Fernando, C., Díez, I. D. L. T., & Marques, G. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109319 · Cited by 129 · see citing works
Liu, J., Zhao, G., Fei, Y., Zhang, M., Wang, Y., & Yu, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01073 · Cited by 99 · see citing works
Basu, S., Gupta, M., Rana, P., Gupta, P. C., & Arora, C. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102676 · Cited by 56 · see citing works
U-Net Based Chest X-ray Segmentation with Ensemble Classification for Covid-19 and Pneumonia
Kumarasinghe, H., Kolonne, S., Fernando, C., et al. (2022) · International Journal of Online and Biomedical Engineering (iJOE) · doi:10.3991/ijoe.v18i07.30807 · Cited by 51 · see citing works
Chest Radiographs Classification Using Multi-model Deep Learning: A Comparative Study
Fernando, C., Kolonne, S., Kumarasinghe, H., & Meedeniya, D. (2022) · International Conference on Advanced Research in Computing (ICARC) · doi:10.1109/icarc54489.2022.9753811 · Cited by 23 · see citing works
MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray imagesPreprint
Hayat, N., Geras, K. J., & Shamout, F. E. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.07027 · Cited by 15 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Anbukkarasi, S., Hemalatha, S., Balakrishnan, A., Varadhaganapathy, S., & Easwaramoorthy, S. V. (2026) · Scientific reports · doi:10.1038/s41598-026-47146-w · Cited by 1 · see citing works
“Nevertheless, recently attention-based transformer architectures have shown that they can discover clinically relevant feature interactions directly from structured EHR data even without the need for explicit graph construction 43 .”
- Authors
- Jahanian, M., Karimi, A., Eraghi, N. O., & Zarafshan, F.
- Affiliations
- Islamic Azad University, Arak (IR)
- Corresponding author
- Jahanian, Mojtaba
- Karimi, Abbas
- Venue
- Informatics in Medicine Unlocked
- Publisher
- Elsevier BV
- Published
- 2025
- Volume
- 58
- Pages
- 101672
- ISSN
- 2352-9148 (ISSN-L)
- OpenAlex
- W4413009320
- Cited by
- 7
- References
- 21
- Citations by year
- 2026: 5 · 2025: 2
- FWCI
- 2.47
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Informatics in Medicine Unlocked (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- Funders
- Islamic Azad University
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Keywords
- multimodal transformer, chest radiography, radiology reports, image-text retrieval, disease classification, vision-language models, cross-attention, contrastive learning, clinical text understanding, MIMIC-CXR
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
