The DOI matches its Crossref record; 79 of the 80 references it cites were found in a registry or an index.
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
Open accessAbstract
Self-supervised learning in vision-language processing (VLP) exploits semantic alignment between imaging and text modalities. Prior work in biomedical VLP has mostly relied on the alignment of single image and report pairs even though clinical notes commonly refer to prior images. This does not only introduce poor alignment between the modalities but also a missed opportunity to exploit rich self-supervision through existing temporal content in the data. In this work, we explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN-Transformer hybrid multi-image encoder trained jointly with a text model. It is designed to be versatile to arising challenges such as pose variations and missing input images across time. The resulting model excels on downstream tasks both in single- and multi-image setups, achieving state-of-the-art (SOTA) performance on (I) progression classification, (II) phrase grounding, and (III) report generation, whilst offering consistent improvements on disease classification and sentence-similarity tasks. We release a novel multi-modal temporal benchmark dataset, MS-CXR-T, to quantify the quality of vision-language representations in terms of temporal semantics. Our experimental results show the advantages of incorporating prior images and reports to make most use of the data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Comparison to previous imaging studies is crucial for tracking the development of disease or impact of treatment, and references to prior studies are frequent in radiology reporting. 16,17”
From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [17]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Such references can be removed to reduce hallucinations when prior studies are not available, 9,14,18 or used in conjunction with prior images to enable descriptions of change.17,19,20”
From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“Comparison to previous imaging studies is crucial for tracking the development of disease or impact of treatment, and references to prior studies are frequent in radiology reporting. 16,17”
From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“It is not possible to quantitatively compare to models trained to generate other sections, such as Impression 17 or both Findings and Impression together.15,58”
From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“It is not possible to quantitatively compare to models trained to generate other sections, such as Impression 17 or both Findings and Impression together.15,58”
From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [58]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Empirically, in the MIMIC-CXR dataset, when the Comparison section is equivalent to 'No comparison available', references to prior studies are rarely observed, in contrast to the background rate of 40% in the full MIMIC-CXR dataset.17”
From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [17]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762
Flagged first, then by citations
1 of the 80 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- brown, 2020, Language models are few-shot learners Advances in neural information processing systems, 33, 1877
- boecking, 2022, MS-CXR: Making the most of text semantics to improve biomedical vision-language processing (version 0.1), PhysioNet
- 2020, American College of Radiology (ACR), ACR practice guideline for communication of diagnostic imaging findings
- johnson, 2019, MIMIC-CXR database (version 2 0 0) PhysioNet
- wu, 2021, Chest imagenome dataset (version 1.0.0), PhysioNet
- endo, 2021, Retrieval-based chest x-ray report generation using a pre-trained contrastive language-image model, Machine Learning for Health, 209
Cited by
Sorted by citations
Items 201–231 of 231
Dual-View Vision Transformer with Cross-Modal Memory for Radiology Report Generation
Lamred, N., Ayed, N. G. B., & Charfi, N. (2026) · International Conference on Control, Decision and Information Technologies (CoDIT) · doi:10.1109/codit70676.2026.11631055
Dieng, A. N. P., Jaballi, S., Nicolas, H., & Zrigui, M. (2026) · International Conference on Control, Decision and Information Technologies (CoDIT) · doi:10.1109/codit70676.2026.11631022
Mathis, C., Wang, S., Hou, J., Wen, J., Wang, Z., Yang, Z., Bhattacharya, R., Sweet, S. M., Elfiky, A. A., Wang, L., et al. (2026) · doi:10.2196/preprints.109672
Temporally Entangled Documents: Multimodal AI for ICU Records Under Label Ambiguity
Butler, L. (2026) · ACM Symposium on Document Engineering (DocEng) · doi:10.1145/3820755.3821644
Medrecord-CLIP: enhancing fundus disease diagnosis via EHR-guided vision-language pre-training
Shi, L., Zhai, W., Yu, L., Wen, D., Wang, J., Qiu, S., & Gao, C. (2026) · Health Information Science and Systems · doi:10.1007/s13755-026-00476-5
Comparative Study of CNN, Hybrid, and Transformer Architectures in Medical Image Classification
Elkholy, D. A., Shehata, M. S., & Braun, J. W. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02154-0
Reddy, K. A. N., Lakshmi, L., Prasad, G. S. C., Priyadharshini, M., Murugesh, V., & Tilaye, G. F. (2026) · Discover Computing · doi:10.1007/s10791-026-10489-z
GroundMed-SAM: Prompt-based Zero-shot Medical Image Segmentation
Nguyen, V. D., Luong, H. Q., & Pham, P. N. (2026) · International Conference on Communications and Electronics (ICCE) · doi:10.1109/icce70609.2026.11658424
Wang, Y., Yang, J., & Zhang, H. (2026) · Fractal and Fractional · doi:10.3390/fractalfract10090597
From Labels to Language: Zero-Shot Radiology Report Generation with Hybrid Retrieval Augmentation
Nishizawa, T., Issah, A., Fuertes-Brito, A., Manzoor, K., Jiao, Z., & Qamar, N. (2026) · IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies · doi:10.1109/chase69719.2026.00062
Standardizing longitudinal chest X-ray report evaluation via large language model annotation
Wang, X., Figueredo, G., Li, R., & Chen, X. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134237
Rahman, H., Mahmood, T., & Raza, A. (2026) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/u145014357
Prosima: Reliable Semantic-Aware Medical Image Recovery Via Lightweight Distributed Ledger
Rasheed, M., & Al-Mamun, A. (2026) · IEEE International Conference on Decentralized Applications and Infrastructures (DAPPS) · doi:10.1109/dapps69803.2026.00014
Fanale, R. (2026) · Frontiers in Imaging · doi:10.3389/fimag.2026.1885292
Işık, Ş., & Eren, H. A. (2026) · Tomography · doi:10.3390/tomography12090129
MedCure: Medical Data Curation for Efficient Vision–Language Pretraining
Wang, C., Liu, F., Liu, Y., Chen, Y., Tian, Y., & Fang, L. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3730839
Zraqou, J. S. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0136-3.ch010
A Lightweight Approach for Improving VLM Accuracy for Medical Usage
Gherghinescu, M., Ivaşcu, T., Ştefănigă, S.-A., & Negru, V. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-37933-7_34
Bidirectional Anatomy-Aware Post-Training for Longitudinal Chest X-Ray Progression Modeling
Chen, Y., Wang, H., To, M., Zhang, J., Wu, Q., & Xie, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38172-9_8
How Good Are Foundation Models in Longitudinal MRI Disease Progression Reasoning?
Ghallabi, W. A., Thawkar, R., Ghaboura, S., Thawakar, O., Saeed, N., Nuaimi, D. A., Alkatheeri, A., Khan, S., & Khan, F. S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38172-9_30
A Review of Deep Learning Models and Explainable Artificial Intelligence for Pathology Recognition
Toala-Paz, A., Becerra-Andrade, A., Cevallos-Levicek, D., Peláez, E., Domínguez, F., & Loayza, F. R. (2026) · International Conference on eDemocracy & eGovernment · doi:10.1109/icedeg70169.2026.11695396
An Adaptive Framework for Evaluating Radiology Reports with Limited Annotated Data
Novikova, V., Pchelintsev, Y. A., Khvostikov, A. V., & Krylov, A. S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39382-1_17
Explainable Multimodal Brain Tumor Classification via Co-Attention Gated Fusion
Syed, A. A., & kumar, K. R. V. G. (2026) · International Conference on Soft Computing for Security Applications (ICSCSA) · doi:10.1109/icscsa69367.2026.11708114
Delrer: Disease Evolution-Informed Longitudinal Radiology Report Generation
Wang, K., Wang, B., Li, C., Lu, Y., Wang, Y., Zhang, H., & Li, X. (2026) · doi:10.1109/cvprf71909.2026.00515
CXR-CoRe: Two-Stage View- and Section-Aware Study-Level Chest X-ray Report Retrieval
Tian, Y., & Ma, J. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02377-1
Foundation Models and Vision-Language Models in Medical Imaging: A Review
Mohsen, S., Lulu, L. M., Eassa, M., Raafat, R., Tawdrous, M. I., & Ibrahim, A. F. (2026) · Computers · doi:10.3390/computers15100707
Chaturvedy, A., Mangalagowri, R., Sivakumar, S., & Chakravarthy, V. D. (2026) · International Conference on Integrated Circuits, Communication, and Computing Systems (ICIC3S) · doi:10.1109/icic3s72305.2026.11737203
LLaMIA: Few-Shot Medical Image Lesion Localization Analysis with Large Language Models
Xie, R., Zeng, A., Pan, D., & Zhao, Y. (2026) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-026-00871-2
Anatomy-Guided Gated Cross-Attention for Chest X-Ray Visual Question Answering
Koç, A. C., Kokach, V., Yιldιrιm, H. M., & Akbulut, F. P. (2026) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11727976
Koyuncu, Z. M., & Aras, S. (2026) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11728081
Parameter-Efficient Adaptation of Vision-Language Models for Medical Visual Question Answering
Barone, M., Serio, F. D., Riccio, G., Romano, A., Gravina, M., Postiglione, M., Moscato, V., & Sansone, C. (2026) · IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc70364.2026.11740016
Metadata
- Authors
- Bannur, S., Hyland, S., Liu, Q., Pérez-García, F., Ilse, M., Castro, D. C., Boecking, B., Sharma, H., Bouzid, K., Thieme, A., Schwaighofer, A., Wetscherek, M., Lungren, M. P., Nori, A., Alvarez-Valle, J., & Oktay, O.
- Affiliations
- Microsoft (United States) (US)
- Microsoft Health Futures
- ORCID
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 15016-15027
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386075916
- Cited by
- 227
- References
- 80
- Citations by year
- 2026: 93 · 2025: 85 · 2024: 40 · 2023: 9
- FWCI
- 24.28
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- temporal structure, self-supervised learning, clinical reports, phrase grounding, report generation, chest radiography
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
