LViT: Language Meets Vision Transformer in Medical Image Segmentation
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Deep learning has been widely used in medical image segmentation and other aspects. However, the performance of existing medical image segmentation models has been limited by the challenge of obtaining sufficient high-quality labeled data due to the prohibitive data annotation cost. To alleviate this limitation, we propose a new text-augmented medical image segmentation model LViT (Language meets Vision Transformer). In our LViT model, medical text annotation is incorporated to compensate for the quality deficiency in image data. In addition, the text information can guide to generate pseudo labels of improved quality in the semi-supervised learning. We also propose an Exponential Pseudo label Iteration mechanism (EPI) to help the Pixel-Level Attention Module (PLAM) preserve local image features in semi-supervised LViT setting. In our model, LV (Language-Vision) loss is designed to supervise the training of unlabeled images using text information directly. For evaluation, we construct three multimodal medical segmentation datasets (image + text) containing X-rays and CT images. Experimental results show that our proposed LViT has superior segmentation performance in both fully-supervised and semi-supervised setting. The code and datasets are available at https://github.com/HUANGLIZI/LViT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
CBAM: Convolutional Block Attention Module
Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_1 · Cited by 25,853
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021) · Nature Methods · doi:10.1038/s41592-020-01008-z · Cited by 10,477
UNet++: A Nested U-Net Architecture for Medical Image Segmentation
Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., & Liang, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00889-5_1 · Cited by 9,610
Swin-Unet: Unet-like pure transformer for medical image segmentation
Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., & Wang, M. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-25066-8_9 · Cited by 5,640
Attention U-Net: Learning Where to Look for the Pancreas
Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N. Y., Kainz, B., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.03999 · Cited by 5,458
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206 · cited 2×
“As can be seen from the red boxes in Figure 4, UNet ++ , nnUNet and TransUNet all have more severe mis-segmentation than LViT.”
Self-citationResidual Attention Network for Image Classification
Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., & Tang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.683 · Cited by 4,073
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 5×
“to refine the pseudo label progressively using the Exponential Moving Average (EMA)[8].”
MethodInfluentialWang, H., Cao, P., Wang, J., & Zaiane, O. R. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20144 · Cited by 1,111
Uncertainty-Aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation
Yu, L., Wang, S., Li, X., Fu, C.-W., & Heng, P.-A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_67 · Cited by 1,061
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
Kim, W., Son, B., & Kim, I. (2021) · arXiv · doi:10.48550/arxiv.2102.03334 · Cited by 951 · cited 1×
“[26] proposed a more parameter-efficient architecture, i.”
Semi-supervised Medical Image Segmentation through Dual-task Consistency
Luo, X., Chen, J., Song, T., & Wang, G. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17066 · Cited by 684
Hofmanninger, J., Prayer, F., Pan, J., Röhrich, S., Prosch, H., & Langs, G. (2020) · European Radiology Experimental · doi:10.1186/s41747-020-00173-2 · Cited by 595
Huang, S.-C., Shen, L., Lungren, M. P., & Yeung, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00391 · Cited by 489
GroupViT: Semantic Segmentation Emerges from Text Supervision
Xu, J., De Mello, S., Liu, S., Byeon, W., Breuel, T., Kautz, J., & Wang, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01760 · Cited by 456
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation
Yang, Z., Wang, J., Tang, Y., Chen, K., Zhao, H., & Torr, P. H. S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01762 · Cited by 406
Mutual consistency learning for semi-supervised medical image segmentation
Wu, Y., Ge, Z., Zhang, D., Xu, M., Zhang, L., Xia, Y., & Cai, J. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102530 · Cited by 394
MosMedData: data set of 1110 chest CT scans performed during the COVID-19 epidemic
Morozov, S. P., Andreychenko, A. E., Blokhin, I. A., Gelezhe, P. B., Gonchar, A. P., Nikolaev, A. E., Pavlov, N. A., Chernina, V. Y., & Gombolevskiy, V. A. (2020) · Digital Diagnostics · doi:10.17816/dd46826 · Cited by 391
Med3D: Transfer Learning for 3D Medical Image Analysis
Chen, S., Ma, K., & Zheng, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.00625 · Cited by 317
Semi-Supervised Medical Image Classification With Relation-Driven Self-Ensembling Model
Liu, Q., Yu, L., Luo, L., Dou, Q., & Heng, P. A. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2995518 · Cited by 295
Xia, Y., Yang, D., Yu, Z., Liu, F., Cai, J., Yu, L., Zhu, Z., Xu, D., Yuille, A., & Roth, H. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101766 · Cited by 278
Boundary-Aware Transformers for Skin Lesion Segmentation
Wang, J., Wei, L., Wang, L., Zhou, Q., Zhu, L., & Qin, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87193-2_20 · Cited by 278
Vision-Language Transformer and Query Generation for Referring Segmentation
Ding, H., Liu, C., Wang, S., & Jiang, X. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01601 · Cited by 266
Contrastive Learning of Medical Visual Representations from Paired Images and Text
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., & Langlotz, C. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00747 · Cited by 265
Self-citationTransfer learning for medical images analyses: A survey
Yu, X., Wang, J., Hong, Q.-Q., Teku, R., Wang, S.-H., & Zhang, Y.-D. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.08.159 · Cited by 232
Self-citationChaitanya, K., Erdil, E., Karani, N., & Konukoglu, E. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102792 · Cited by 223
TGANet: Text-Guided Attention for Improved Polyp Segmentation
Tomar, N. K., Jha, D., Bagci, U., & Ali, S. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-16437-8_15 · Cited by 195
VLT: Vision-Language Transformer and Query Generation for Referring Segmentation
Ding, H., Liu, C., Wang, S., & Jiang, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3217852 · Cited by 160
Discriminative Localization in CNNs for Weakly-Supervised Segmentation of Pulmonary Nodules
Feng, X., Yang, J., Laine, A. F., & Angelini, E. D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-66179-7_65 · Cited by 155
Osegnet: Operational Segmentation Network for Covid-19 Detection Using Chest X-Ray Images
Degerli, A., Serkan Kiranyaz, B., Chowdhury, M. E. H., & Gabbouj, M. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897412 · Cited by 111
GT U-Net: A U-Net Like Group Transformer Network for Tooth Root Segmentation
Li, Y., Wang, S., Wang, J., Zeng, G., Liu, W., Zhang, Q., Jin, Q., & Wang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87589-3_40 · Cited by 93
Self-citationTFCNs: A CNN-Transformer Hybrid Network for Medical Image Segmentation
Li, Z., Li, D., Xu, C., Wang, W., Hong, Q., Li, Q., & Tian, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15937-4_65 · Cited by 92
Self-citationLi, Y., Luo, L., Lin, H., Chen, H., & Heng, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87196-3_19 · Cited by 77
Self-citationLi, Y., Zeng, G., Zhang, Y., Wang, J., Jin, Q., Sun, L., Zhang, Q., Lian, Q., Qian, G., Xia, N., et al. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3129245 · Cited by 54
Self-citationDCAU-Net: dense convolutional attention U-Net for segmentation of intracranial aneurysm images
Yuan, W., Peng, Y., Guo, Y., Ren, Y., & Xue, Q. (2022) · Visual Computing for Industry Biomedicine and Art · doi:10.1186/s42492-022-00105-4 · Cited by 48
Huang, C., Lan, Y., Xu, G., Zhai, X., Wu, J., Lin, F., Zeng, N., Hong, Q., Ng, E. Y. K., Peng, Y., et al. (2020) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2020.2973971 · Cited by 41
Self-citationImproving Pneumonia Localization via Cross-Attention on Medical Images and Reports
Bhalodia, R., Hatamizadeh, A., Tam, L., Xu, Z., Wang, X., Turkbey, E. B., & Xu, D. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87196-3_53 · Cited by 37
Zhu, Z., Jin, D., Yan, K., Ho, T., Ye, X., Guo, D., Chao, C.-H., Xiao, J. Z., Yuille, A., & Lü, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59728-3_73 · Cited by 32
Self-citationScaling Up Multi-domain Semantic Segmentation with Sentence Embeddings
Yin, W., Liu, Y., Shen, C., Sun, B., & van den Hengel, A. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02060-4 · Cited by 21
Coarse-to-Fine Covid-19 Segmentation via Vision-Language Alignment
Shan, D., Li, Z., Chen, W., Li, Q., Tian, J., & Hong, Q. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096683 · Cited by 18
Self-citationMRDFF: A deep forest based framework for CT whole heart segmentation
Xu, F. Y., Lin, L., Li, Z., et al. (2022) · Methods · doi:10.1016/j.ymeth.2022.10.005 · Cited by 17
Self-citationSelf-supervised Tumor Segmentation through Layer DecompositionPreprint
Zhang, X., Xie, W., Huang, C., Wang, Y., Zhang, Y., Chen, X., & Tian, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03230 · Cited by 3
Self-citationA Multi-Resolution Deep Forest Framework with Hybrid Feature Fusion for CT Whole Heart Segmentation
Xu, F., Lin, L., Li, D., et al. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669868 · Cited by 2
Cited by
A scoping review on multimodal deep learning in biomedical images and texts
Sun, Z., Lin, M., Zhu, Q., Xie, Q., Wang, F., Lu, Z., & Peng, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104482 · Cited by 69
“Li et al. [68] proposed LViT, which used medical text annotations to improve the quality of image data and guide the generation of pseudo labels, leading to better segmentation performance.”
BackgroundLarge Language Models in Critical Care Medicine: Scoping Review
Shi, T., Ma, J., Yu, Z., Xu, H., Yang, R., Xiong, M., Xiao, M., Li, Y., Zhao, H., & Kong, G. (2025) · JMIR Medical Informatics · doi:10.2196/76326 · Cited by 21
Self-citation
Metadata
- Authors
- Li, Z., Li, Y., Li, Q., Wang, P., Guo, D., Lu, L., Jin, D., Zhang, Y., & Hong, Q.
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2024-01
- Volume
- 43
- Issue
- 1
- Pages
- 96-107
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W4382998948
- Cited by
- 374
- References
- 51
- FWCI
- 33.73
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.