CoChat
Sign Up
Cited by 41Multimodal Machine Learning ApplicationsArtificial Intelligence in Healthcare and EducationRadiomics and Machine Learning in Medical Imaging

Vision-language foundation model for 3D medical imaging

Open access
Wu, Jing, Wang, Yuli, Zhong, Zhusi, Liao, Weihua, Trayanova, Natalia, Jiao, Zhicheng, Bai, Harrison X.·2025·npj Artificial Intelligence
DOI 10.1038/s44387-025-00015-9
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advances in AI, especially vision-language foundation models (VLFMs), show promise in automating radiology report generation from complex 3D medical imaging data. Our review analyzes 23 studies on VLFMs, focusing on model architectures, capabilities, training datasets, and evaluation metrics. We discuss AI’s evolution in radiology, emphasizing the need for diverse datasets and standardized metrics, as challenges remain in producing consistent, high-quality reports.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s44387-025-00015-9

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Large language models, computer vision systems, and clinical decision-support systems have demonstrated substantial clinical value across medical specialties.

    “Large language models (LLMs), computer vision systems, and clinical decision-support systems (CDSS) have demonstrated substantial clinical value across medical specialties 1 -4 .”

    From AI-induced never-skilling in medical education (2026) · cites this paper as [4]

    The Abstract describes potential clinical benefits of vision-language models in medical imaging, and ‘Addressing computational and clinical challenges in 3D’ identifies possible supportive uses. But that section says prospective trials are still needed to demonstrate clinical value. The paper does not establish that LLMs, computer vision systems, and CDSS have already demonstrated substantial clinical value across medical specialties; page numbers were not available in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 78 · Cited by 41

Research works printed but not linked: 57

  1. Singh, S. P. et al. 3D deep learning on medical images: a review. Sensors 20, 5097 (2020).
  2. Liu, X. et al. Advances in deep learning-based medical image analysis. Health Data Sci. 2021, 8786793 (2021).
  3. Yildirim, N. et al. Multimodal healthcare AI: identifying and designing clinically relevant vision-language applications for radiology. in: Proceedings of the CHI Conference on Human Factors in Computing Systems, pp 1-22 (2024).
  4. Hartsock, I. & Rasool, G. Vision-language models for medical report generation and visual question answering: a review. Front Artif Intell. 7, 1430984 (2024).
  5. Wu, C., Zhang X., Zhang Y., Wang Y. & Xie W. Towards generalist foundation model for radiology. Preprint at arXiv https://doi.org/10. 48550/arXiv.2308.02463 (2023). · DOI
  6. Moor, M. et al. Med-flamingo: a multimodal medical few-shot learner. Machine Learning for Health (ML4H). PMLR. (2023).
  7. Hamamci, I. E. et al. A foundation model utilizing chest CT volumes and radiology reports for supervised-level zero-shot detection of abnormalities. Preprint at arXiv https://arxiv.org/html/2403.17834v1 (2024).
  8. Huang, S.-C. et al. INSPECT: a multimodal dataset for pulmonary embolism diagnosis and prognosis. Preprint at arXiv https://doi.org/ 10.48550/arXiv.2311.10798 (2023). · DOI
  9. Bai, F., Du, Y., Huang, T., Meng, MQ-H. & Zhao, B. M3D: advancing 3D medical image analysis with multi-modal large language models. Preprint at arXiv https://doi.org/10.48550/arXiv.2404.00578 (2024). · DOI · Paper page
  10. Zhang, X. et al. RadGenome-Chest CT: a grounded vision-language dataset for chest CT analysis. Preprint at arXiv https://doi.org/10. 48550/arXiv.2404.16754 (2024). · DOI
  11. Xu, C., Tao, D. & Xu, C. A survey on multi-view learning. Preprint at arXiv https://doi.org/10.48550/arXiv.1304.5634 (2013). · DOI
  12. Yariv, L., Gu, J., Kasten, Y. & Lipman, Y. Volume rendering of neural implicit surfaces. Adv. Neural Inf. Process Syst. 34, 4805-4815 (2021).
  13. Park, T. et al. Contrastive learning for unpaired image-to-image translation. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16. (Springer, 2020).
  14. Sun, Y. et al. Roi pooled correlation filters for visual tracking. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5776--5784 (2019).
  15. Papineni, K. et al. Bleu: a method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics. (2002).
  16. Banerjee, S. et al. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the Acl Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. (2005).
  17. Vedantam, R. et al. Cider: consensus-based image description evaluation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4566-4575 (2015). https://doi.org/10.1038/s44387-025-00015-9
  18. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human languagetechnologies, vol 1 (long and short papers) (2019).
  19. Yang, L. et al. Advancing multimodal medical capabilities of Gemini. Preprint at arXiv https://doi.org/10.48550/arXiv.2405. 03162 (2024). · DOI · Paper page
  20. Wang, S. et al. CopilotCAD: empowering radiologists with report completion models and quantitative evidence from medical image foundation models. Preprint at arXiv https://doi.org/10.48550/arXiv. 2404.07424 (2024). · DOI
  21. Lu, Y. et al. Machine learning for synthetic data generation: a review. Preprint at arXiv https://doi.org/10.48550/arXiv.2302.04062 (2023). · DOI
  22. Castillo, C., Steffens, T., Sim, L. & Caffery, L. The effect of clinical information on radiology reporting: a systematic review. J. Med. Radiat. Sci. 68, 60-74 (2021).
  23. Sloan, P., Clatworthy, P., Simpson, E. & Mirmehdi, M. Automated radiology report generation: a review of recent advances. IEEE Rev. Biomed. Eng. (2024).
  24. Tremblay, J. et al. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. (2018).
  25. Khader, F. et al. Denoising diffusion probabilistic models for 3D medical image generation. Sci. Rep. 13, 7303 (2023).
  26. Thambawita, V. et al. SinGAN-Seg: synthetic training data generation for medical image segmentation. PloS ONE 17, e0267976 (2022).
  27. Cirillo, M. D. et al. Vox2Vox: 3D-GAN for brain tumour segmentation. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 6th International Workshop, BrainLes 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4, 2020, Revised Selected Papers, Part I 6. (Springer, 2021).
  28. Wu J. et al. Medsegdiff: medical image segmentation with diffusion probabilistic model. Medical Imaging with Deep Learning. PMLR. (2024).
  29. Chen, Z., Ma, M., Li, T., Wang, H. & Li, C. Long sequence time-series forecasting with deep learning: a survey. Inf. Fusion 97, 101819 (2023).
  30. Li, J. et al. Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives. Med. Image Anal. 85, 102762 (2023).
  31. Ahmad, I. S., Dai, J., Xie, Y. & Liang, X. Deep learning models for CT image classification: a comprehensive literature review. Quant. imaging Med. Surg. 15, 962-1011 (2025).
  32. Tang, Y. et al. Self-supervised pre-training of swin transformers for 3d medical image analysis. In: Proceedings of the Ieee/cvf Conference on Computer Vision and Pattern Recognition (2022).
  33. Ren, H. et al. Combiner: Full attention transformer with sparse computation cost. Adv. Neural Inf. Process. Syst. 34, 22470-22482 (2021).
  34. Pinasthika, K., Laksono, B. S. P., Irsal, R. B. P., Shabiyya, S. H. & Yudistira, N. SparseSwin: swin transformer with sparse transformer block. Neurocomputing 580, 127433 (2024).
  35. Deng, Y. et al. TChange: a hybrid transformer-CNN change detection network. Remote Sens. 15, 1219 (2023).
  36. Zhao, M., Cao, G., Huang, X. & Yang, L. Hybrid transformer-CNN for real image denoising. IEEE Signal Process. Lett. 29, 1252-1256 (2022).
  37. Zhou, W. et al. HENet: hierarchical enhancement network for pulmonary vessel segmentation in non-contrast CT images. International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer, 2023).
  38. Chen, Z., Luo, L., Bie, Y. & Chen, H. Dia-LLaMA: towards large language model-driven CT report generation. Preprint at arXiv https:// doi.org/10.48550/arXiv.2403.16386 (2024). · DOI
  39. Blankemeier, L. et al. Merlin: a vision language foundation model for 3D computed tomography. Research Square, rs-3 (2024).
  40. Zhang, T. et al. Incorporating clinical guidelines through adapting multi-modal large language model for prostate cancer PI-RADS scoring. International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 360-370 (2024).
  41. Liu, C. et al. Benchmarking and boosting radiology report generation for 3D high-resolution medical images. Preprint at arXiv https://arxiv. org/html/2406.07146v2 (2024).
  42. Zhong, Z. et al. Abn-BLIP: abnormality-aligned bootstrapping language-image pre-training for pulmonary embolism diagnosis and report generation from CTPA. Preprint at arXiv https://doi.org/10. 48550/arXiv.2503.02034 (2025). · DOI
  43. Shui, Z. et al. Large-scale and fine-grained vision-language pre- training for enhanced CT image understanding. Preprint at arXiv https://doi.org/10.48550/arXiv.2501.14548 (2025). · DOI
  44. Lee, C. et al. Read like a radiologist: efficient vision-language model for 3D medical imaging interpretation. Preprint at arXiv https://doi.org/ 10.48550/arXiv.2412.13558 (2024). · DOI
  45. Ni, X. et al. MG-3D: Multi-grained knowledge-enhanced 3D medical vision-language pre-training. Preprint at arXiv https://doi.org/10. 48550/arXiv.2412.05876 (2024). · DOI
  46. Lee, J. et al. Improving Factuality of 3D Brain MRI Report Generation with Paired Image-domain Retrieval and Text-domain Augmentation. arXiv preprint arXiv:15490. (2024).
  47. Shi, Y. et al. Med-2E3: a 2D-enhanced 3D medical multimodal large language model. Preprint at arXiv https://doi.org/10.48550/arXiv. 2411.12783 (2024). · DOI
  48. Lai, H. et al. E3D-GPT: enhanced 3D visual foundation for medical vision-language model. Preprint at arXiv https://doi.org/10.48550/ arXiv.2410.14200 (2024). · DOI
  49. Luo, L., Tang, B., Chen, X., Han, R. & Chen, T. VividMed: vision language model with versatile visual grounding for medicine. Preprint at arXiv https://doi.org/10.48550/arXiv.2410.12694 (2024). · DOI
  50. Li, S., Xu, B., Luo, Y., Nie, D. & Zhang, L. ViT3D alignment of LLaMA3: 3D medical image report generation. Preprint arXiv https://doi.org/10. 48550/arXiv.2410.08588 (2024). https://doi.org/10.1038/s44387-025-00015-9 · DOI
  51. Zheng, C., Ji, J., Shi, Y., Zhang, X. & Qu L. See detail say clear: towards brain CT report generation via pathological clue-driven representation learning. Preprint at arXiv https://doi.org/10.48550/arXiv.2409.19676 (2024). · DOI
  52. Chen H. et al. 3d-ct-gpt: generating 3d radiology reports through integration of large vision-language models. Preprint at arXiv https:// doi.org/10.48550/arXiv.2409.19330 (2024). · DOI
  53. Alkhaldi A. et al. Minigpt-med: Large language model as a general interface for radiology diagnosis. Preprint at arXiv https://doi.org/10. 48550/arXiv.2407.04106 (2024). · DOI
  54. Rajpurkar, M. et al. CheXNet: radiologist-level pneumonia detection on chest X-rays with deep learning. Preprint at arXiv https://doi.org/ 10.48550/arXiv.1711.05225 (2017). · DOI
  55. Manzari, O. N., Ahmadabadi, H., Kashiani, H., Shokouhi, S. B. & Ayatollahi, A. MedViT: a robust vision transformer for generalized medical image classification. Comput. Biol. Med. 157, 106791 (2023).
  56. Ghosh A., Acharya A., Saha S., Jain V. & Chadha A. Exploring the frontier of vision-language models: a survey of current methodologies and future directions. Preprint at arXiv https://doi.org/10.48550/arXiv. 2404.07214 (2024). · DOI
  57. Mennella, C., Maniscalco, U., De Pietro, G. & Esposito, M. Ethical and regulatory challenges of AI technologies in healthcare: a narrative review. Heliyon 10, e26297 (2024).

Authors
Wu, J., Wang, Y., Zhong, Z., Liao, W., Trayanova, N., Jiao, Z., & Bai, H. X.
Affiliations
  • Central South University (CN)
  • Johns Hopkins University (US)
  • Rhode Island Hospital (US)
Corresponding author
Bai, Harrison X.
ORCID
Venue
npj Artificial Intelligence
Publisher
Springer Science and Business Media LLC
Published
2025-08-06
Volume
1
Issue
1
ISSN
3005-1460 (ISSN-L)
DOI
10.1038/s44387-025-00015-9
OpenAlex
W4413030191
Cited by
41
References
78
Citations by year
2026: 34 · 2025: 7
FWCI
14.44
Type
journal article
Language
en
Open access
Licence
Locations
npj Artificial Intelligence (publishedVersion, cc-by-nc-nd)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
vision-language foundation models, 3D medical imaging, radiology report generation, automated reporting, model architecture, evaluation metrics, 3D medical image analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review