Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment
Open accessAbstract
Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is calculated using the attention mechanism without direct supervision about their mapping. The pre-trained multi-modal representation learning paves the way for downstream tasks concerning image and/or text encoding. We demonstrate the representation learning quality by cross-modality retrievals and multi-label classifications on two datasets: OpenI-IU and MIMIC-CXR. Our code is available at https://github.com/mshaikh2/JoImTeR_MLMI_2021.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several studies have utilized medical reports to provide supervision signals during self-supervised learning and shown label efficiency for downstream tasks.
“Several studies have utilized these medical reports to provide supervision signals during self-supervised learning and shown label efficiency for downstream tasks60,106.”
From Self-supervised learning for medical image classification: a systematic review and implementation guidelines (2023) · cites this paper as [60]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · see citing works
Self-citationUNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · see citing works
Learning deep structured semantic models for web search using clickthrough data
Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L. P. (2013) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2505515.2505665 · Cited by 2,054 · see citing works
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.10 · Cited by 2,036 · see citing works
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923 · see citing works
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · see citing works
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · see citing works
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298754 · Cited by 1,305 · see citing works
Preparing a collection of radiology examinations for distribution and retrieval
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv080 · Cited by 1,164 · see citing works
Deep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048 · see citing works
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
Li, G., Duan, N., Fang, Y., Gong, M., & Jiang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6795 · Cited by 795 · see citing works
ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing
Neumann, M., King, D., Beltagy, I., & Ammar, W. (2019) · BioNLP Workshop and Shared Task (BioNLP) · doi:10.18653/v1/w19-5034 · Cited by 772 · see citing works
Wang, X., Peng, Y., Lü, L., Lu, Z., & Summers, R. M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00943 · Cited by 574 · see citing works
Self-citationCheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392 · see citing works
Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
Li, C. Y., Liang, X., Hu, Z., & Xing, E. P. (2018) · arXiv · doi:10.48550/arxiv.1805.08298 · Cited by 271 · see citing works
Li, Y., Wang, H., & Luo, Y. (2020) · 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313289 · Cited by 81 · see citing works
Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment
Chauhan, G., Liao, R., Wells, W. M., Andreas, J., Wang, X., Berkowitz, S. J., Horng, S., Szolovits, P., & Golland, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59713-9_51 · Cited by 59 · see citing works
Attention based Writer Independent Verification
Shaikh, M. A., Duan, T., Chauhan, M., & Srihari, S. N. (2020) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr2020.2020.00074 · Cited by 43 · see citing works
Self-citation
Flagged first, then by citations
1 of the 25 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- [12]Johnson AE,: MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 (2019) PMC6908718 31831740 · arXiv
- [15]Kingma DP, Ba J: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) · arXiv
- [17]Li LH, Yatskar M, Yin D, Hsieh CJ, Chang KW: VisualBERT: a simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019) · arXiv
Cited by
Sorted by citations
Huang, S.-C., Pareek, A., Jensen, M., Lungren, M. P., Yeung, S., & Chaudhari, A. S. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00811-0 · Cited by 479 · see citing works
“In radiology, Ji et al. used multimodal contrastive learning by matching X-rays with corresponding radiology reports 60 .”
BackgroundA scoping review on multimodal deep learning in biomedical images and texts
Sun, Z., Lin, M., Zhu, Q., Xie, Q., Wang, F., Lu, Z., & Peng, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104482 · Cited by 69 · see citing works
Self-citationVanberlo, B., Hoey, J., & Wong, A. (2024) · BMC Medical Imaging · doi:10.1186/s12880-024-01253-0 · Cited by 59 · see citing works
MITER: Medical Image–TExt joint adaptive pretRaining with multi-level contrastive learning
Shu, C., Zhu, Y., Tang, X., Xiao, J., Chen, Y., Li, X., Zhang, Q., & Lu, Z. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121526 · Cited by 34 · see citing works
On the use of contrastive learning for standard-plane classification in fetal ultrasound imaging
Migliorelli, G., Fiorentino, M. C., Cosmo, M. D., Villani, F. P., Mancini, A., & Moccia, S. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108430 · Cited by 29 · see citing works
Category supervised cross-modal hashing retrieval for chest X-ray and radiology reports
Zhang, Y., Ou, W., Zhang, J., & Deng, J. (2022) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2021.107673 · Cited by 18 · see citing works
Semantic interdisciplinary evaluation of image captioning models
Sirisha, U., & Chandana, B. S. (2022) · Cogent Engineering · doi:10.1080/23311916.2022.2104333 · Cited by 18 · see citing works
Automatic Medical Report Generation: Methods and Applications
Li, G., Tahir, A. M., Zhang, D., Wang, Z. J., & Ward, R. K. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.20240044 · Cited by 19 · see citing works
A Systematic Review and Implementation Guidelines of Multimodal Foundation Models in Medical ImagingPreprint
Huang, S.-C., Jensen, M., Yeung, S. Y., Lungren, M. P., Poon, H., & Chaudhari, A. S. (2025) · Research Square · doi:10.21203/rs.3.rs-5537908/v1 · Cited by 17 · see citing works
Multimodal Foundation Models for Medical Imaging - A Systematic Review and Implementation GuidelinesPreprint
Huang, S.-C., Jensen, M., Yeung, S. Y., Lungren, M. P., Poon, H., & Chaudhari, A. S. (2024) · medRxiv · doi:10.1101/2024.10.23.24316003 · Cited by 15 · see citing works
O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models
Sharifdeen, A., Munir, M. A., Baliah, S., Khan, S. A., & Khan, M. H. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01857 · Cited by 11 · see citing works
Ketabi, S., Wagner, M. W., Hawkins, C. E., Tabori, U. Y., Ertl-Wagner, B. B., & Khalvati, F. (2025) · Scientific Reports · doi:10.1038/s41598-025-94806-4 · Cited by 12 · see citing works
Multi-view and Multi-scale Alignment for Contrastive Language-Image Pre-training in Mammography
Du, Y., Onofrey, J. A., & Dvornek, N. C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-96625-5_17 · Cited by 12 · see citing works
Few-Shot Learning Geometric Ensemble for Multi-label Classification of Chest X-Rays
Moukheiber, D., Mahindre, S., Moukheiber, L., Moukheiber, M., Wang, S., Ma, C., Shih, G., Peng, Y., & Gao, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17027-0_12 · Cited by 7 · see citing works
Self-citationJaiswal, T., Pandey, M., & Tripathi, P. (2025) · IETE Journal of Research · doi:10.1080/03772063.2025.2519536 · Cited by 4 · see citing works
Kulsoom, U., Glavin, F. G., & Bendechache, M. (2025) · IEEE Access · doi:10.1109/access.2025.3582728 · Cited by 5 · see citing works
Meta-Entity Driven Triplet Mining for Aligning Medical Vision-Language Models
Yilmaz, M., Öztürk, Ş., Kara, M., Yavuz, M. T., Gümeler, E., Koç, A., & Çukur, T. (2026) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2026.3678306 · Cited by 2 · see citing works
Multimodal Representation Learning for Blastocyst Assessment
Wang, Y., Zheng, Z., Ni, N., Tong, G., Cheng, N., Li, K., Yin, P., Chen, Y., Wu, Y., & Xie, G. (2023) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi53787.2023.10230468 · Cited by 1 · see citing works
Deep understanding of radiology reports: leveraging dynamic convolution in chest X-ray images
Jaiswal, T., Pandey, M., & Tripathi, P. (2023) · Data Technologies and Applications · doi:10.1108/dta-07-2023-0307 · Cited by 3 · see citing works
CSIP: Contrastive learning with Graph Neural Network enables chemical structure to image mapping
Liu, L. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/86/20241606
- Authors
- Ji, Z., Shaikh, M. A., Moukheiber, D., Srihari, S. N., Peng, Y., & Gao, M.
- Affiliations
- University at Buffalo, State University of New York (US)
- Weill Cornell Medicine (US)
- Corresponding author
- Ji, Zhanghexuan
- Shaikh, Mohammad Abuzar
- ORCID
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2021
- Volume
- 12966
- Pages
- 110-119
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- PMC
- PMC9134785
- OpenAlex
- W3202198427
- Cited by
- 21
- References
- 25
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 6 · 2023: 5 · 2022: 3
- FWCI
- 1.14
- Type
- conference paper
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- Locations
- Lecture notes in computer science (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1910492)
- U.S. National Library of Medicine (R00LM013001)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- chest radiography, radiology reports, self-supervised pre-training, cross-entropy loss, triplet loss, attention mechanism, multi-label classification, multimodal representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
