The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.
Transformer-Based Visual Segmentation: A Survey
Open accessAbstract
Visual segmentation seeks to partition images, video frames, or point clouds into multiple segments or groups. This technique has numerous real-world applications, such as autonomous driving, image editing, robot sensing, and medical analysis. Over the past decade, deep learning-based methods have made remarkable strides in this area. Recently, transformers, a type of neural network based on self-attention originally designed for natural language processing, have considerably surpassed previous convolutional or recurrent approaches in various vision processing tasks. Specifically, vision transformers offer robust, unified, and even simpler solutions for various segmentation tasks. This survey provides a thorough overview of transformer-based visual segmentation, summarizing recent advancements. We first review the background, encompassing problem definitions, datasets, and prior convolutional methods. Next, we summarize a meta-architecture that unifies all recent transformer-based approaches. Based on this meta-architecture, we examine various method designs, including modifications to the meta-architecture and associated applications. We also present several specific subfields, including 3D point cloud segmentation, foundation model tuning, domain-aware segmentation, efficient segmentation, and medical segmentation. Additionally, we compile and re-evaluate the reviewed methods on several well-established datasets. Finally, we identify open challenges in this field and propose directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Self-citationFeature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Research works printed but not linked: 265
- 10.1023/A:1011174803800 · DOI · Paper page
- 10.1109/34.868688 · DOI · Paper page
- Stella, Multiclass spectral clustering, 313
- 10.5244/C.22.54 · DOI · Paper page
- 10.1007/BF00133570 · DOI · Paper page
- 10.1007/s11263-015-0816-y · DOI · Paper page
- Simonyan, Very deep convolutional networks for large-scale image recognition, 1
- 10.1109/CVPR.2015.7298965 · DOI · Paper page
- 10.1109/TPAMI.2017.2699184 · DOI · Paper page
- 10.1109/CVPR.2017.660 · DOI · Paper page
- 10.1109/CVPR.2018.00254 · DOI · Paper page
- 10.48550/ARXIV.1706.03762 · DOI · Paper page
- 10.1007/978-3-030-01240-3_17 · DOI · Paper page
- 10.1109/CVPR42600.2020.01009 · DOI · Paper page
- 10.1109/ICCV.2019.00356 · DOI · Paper page
- 10.1109/CVPR46437.2021.00542 · DOI · Paper page
- 10.1109/ICCV48922.2021.01595 · DOI · Paper page
- 10.1109/CVPR46437.2021.00738 · DOI · Paper page
- 10.1109/tpami.2023.3243465 · DOI · Paper page
- 10.1109/tpami.2023.3275156 · DOI · Paper page
- 10.1016/j.neucom.2019.11.118 · DOI · Paper page
- 10.1109/TPAMI.2022.3225573 · DOI · Paper page
- 10.1109/CVPR52688.2022.02036 · DOI
- 10.1007/978-3-319-10602-1_48 · DOI · Paper page
- 10.1109/CVPR52729.2023.02259 · DOI · Paper page
- 10.1109/CVPR46437.2021.00412 · DOI
- 10.1109/ICCV.2019.00529 · DOI · Paper page
- 10.1007/s11263-022-01629-1 · DOI
- 10.1109/CVPR42600.2020.00988 · DOI · Paper page
- Weber, STEP: Segmenting and tracking every pixel
- 10.1109/ICCV51070.2023.01850 · DOI
- 10.1109/ICCV51070.2023.00254 · DOI
- 10.1109/CVPR.2018.00199 · DOI · Paper page
- 10.1109/TIP.2019.2962685 · DOI
- 10.1109/CVPR.2019.00909 · DOI · Paper page
- 10.1109/TIP.2018.2878975 · DOI
- 10.1609/aaai.v34i07.6805 · DOI · Paper page
- 10.1109/TIP.2021.3099366 · DOI
- 10.48550/arXiv.1909.11065 · arXiv · doi:10.48550/arxiv.1909.11065 · Paper page
- Zhang, Dual graph convolutional network for semantic segmentation, 254
- 10.1007/978-3-030-58452-8_45 · DOI · Paper page
- 10.1109/CVPR42600.2020.00982 · DOI · Paper page
- 10.1109/ICCV.2019.00692 · DOI · Paper page
- 10.1007/978-3-030-58520-4_26 · DOI · Paper page
- 10.1109/ICCV48922.2021.01556 · DOI
- 10.1109/CVPR.2019.00326 · DOI · Paper page
- 10.1007/978-3-030-58452-8_17 · DOI · Paper page
- 10.1109/CVPR.2019.00904 · DOI · Paper page
- 10.1109/CVPR42600.2020.01024 · DOI
- 10.1109/CVPR42600.2020.01249 · DOI · Paper page
- 10.1109/ICCV.2019.00215 · DOI · Paper page
- Wang, SOLOv2: Dynamic and fast instance segmentation
- 10.1109/CVPR.2019.00902 · DOI · Paper page
- 10.1109/CVPR46437.2021.00028 · DOI · Paper page
- 10.1007/978-3-030-58548-8_7 · DOI · Paper page
- 10.1109/ICCV.2017.477 · DOI · Paper page
- 10.1007/978-3-319-49409-8_69 · DOI · Paper page
- 10.1109/CVPR.2017.441 · DOI · Paper page
- 10.1109/CVPR42600.2020.00976 · DOI · Paper page
- 10.1609/aaai.v35i2.16225 · DOI
- 10.1109/TPAMI.2022.3211612 · DOI
- 10.1109/CVPR46437.2021.00399 · DOI
- 10.1109/CVPR.2016.85 · DOI · Paper page
- Qi, PointNet: Deep learning on point sets for 3D classification and segmentation, 652
- Qi, PointNet: Deep hierarchical feature learning on point sets in a metric space, 5099
- Yang, Learning object bounding boxes for 3D instance segmentation on point clouds, 6737
- 10.1109/CVPR.2019.00407 · DOI · Paper page
- 10.1109/CVPR.2018.00272 · DOI · Paper page
- 10.1109/CVPR42600.2020.00492 · DOI · Paper page
- 10.1109/ICCV.2019.00166 · DOI · Paper page
- 10.1109/CVPR46437.2021.01236 · DOI · Paper page
- 10.1109/CVPR46437.2021.01299 · DOI
- 10.1609/aaai.v36i3.20197 · DOI
- 10.1109/CVPR46437.2021.01289 · DOI
- 10.1109/CVPR46437.2021.00548 · DOI
- 10.1109/CVPR46437.2021.00981 · DOI · Paper page
- Chen, A2-Nets: Double attention networks, 350
- Ren, Faster R-CNN: Towards real-time object detection with region proposal networks, 91
- 10.1109/TPAMI.2020.3032166 · DOI · Paper page
- 10.1109/CVPR.2019.00720 · DOI · Paper page
- Xie, SegFormer: Simple and efficient design for semantic segmentation with transformers, 12077
- 10.1109/CVPR52733.2024.02640 · DOI
- 10.1109/CVPR52729.2023.01797 · DOI
- 10.1007/978-3-031-19812-0_34 · DOI
- 10.1109/iccv51070.2023.00240 · DOI
- Touvron, Training data-efficient image transformers & distillation through attention, 10347
- 10.1109/CVPR52688.2022.00476 · DOI · Paper page
- 10.1109/CVPR52688.2022.00714 · DOI · Paper page
- Ali, XCiT: Cross-covariance image transformers, 20014
- 10.1109/ICCV48922.2021.00041 · DOI · Paper page
- 10.1007/978-3-030-58604-1_20 · DOI · Paper page
- 10.1109/ICCV48922.2021.00983 · DOI · Paper page
- 10.1109/ICCV48922.2021.00009 · DOI · Paper page
- Xu, ViTAE: Vision transformer advanced by exploring intrinsic inductive Bias, 28522
- 10.1109/CVPR52688.2022.01167 · DOI · Paper page
- Han, On the connection between local attention and dynamic depth-wise convolution, 1
- Guo, SegNeXt: Rethinking convolutional attention design for semantic segmentation, 1140
- 10.1109/CVPR52688.2022.01055 · DOI · Paper page
- 10.5555/3524938.3525087 · doi:10.5555/3524938.3525087
- Bao, BEiT: BERT pre-training of image transformers, 14668
- Gandelsman, Test-time training with masked autoencoders
- Gao, ConvMAE: Masked convolution meets masked autoencoders, 35632
- 10.1109/cvpr52729.2023.02240 · DOI · Paper page
- 10.1109/ICCV48922.2021.00683 · DOI · Paper page
- Zhang, K-Net: Towards unified image segmentation, 10326
- Cheng, Per-pixel classification is not all you need for semantic segmentation, 17864
- 10.1109/CVPR52688.2022.00134 · DOI · Paper page
- 10.1109/CVPR46437.2021.00863 · DOI · Paper page
- 10.1109/TPAMI.2022.3223955 · DOI · Paper page
- 10.1109/CVPR52688.2022.00290 · DOI
- Hwang, Video instance segmentation using inter-frame communication transformers, 13352
- 10.1007/978-3-031-19815-1_32 · DOI
- 10.1109/CVPR52688.2022.01828 · DOI
- 10.1109/CVPR52688.2022.01354 · DOI
- 10.1609/aaai.v36i3.20158 · DOI · Paper page
- Liu, DAB-DETR: Dynamic anchor boxes are better queries for DETR, 1
- 10.1109/CVPR52729.2023.00297 · DOI · Paper page
- Wang, Learning equivariant segmentation with instance-unique querying
- 10.1109/CVPR52729.2023.01887 · DOI · Paper page
- Huang, MinVIS: A minimal video instance segmentation framework without video-based training, 31265
- 10.1007/978-3-031-19815-1_34 · DOI
- 10.1007/978-3-031-19812-0_42 · DOI
- 10.1109/CVPR52688.2022.00168 · DOI
- 10.1007/978-3-031-19836-6_31 · DOI
- 10.1007/978-3-031-19812-0_30 · DOI
- 10.1109/ICCV48922.2021.01601 · DOI · Paper page
- 10.1109/CVPR52688.2022.01762 · DOI · Paper page
- 10.1109/CVPR52688.2022.01761 · DOI · Paper page
- 10.1109/CVPR52688.2022.01139 · DOI · Paper page
- 10.1109/CVPR52688.2022.00493 · DOI · Paper page
- 10.1109/CVPR52688.2022.00491 · DOI
- 10.1109/tip.2024.3371348 · DOI
- 10.1109/CVPR52688.2022.00492 · DOI · Paper page
- Zhang, Few-shot segmentation via cycle-consistent transformer, 21984
- Yang, Associating objects with transformers for video object segmentation, 2491
- 10.1109/CVPR52688.2022.01140 · DOI
- 10.1007/978-3-031-19815-1_36 · DOI
- 10.1109/tcsvt.2023.3252807 · DOI
- 10.1007/978-3-031-19818-2_35 · DOI
- 10.1109/CVPR52688.2022.01170 · DOI · Paper page
- Chen, CycleMLP: A MLP-like architecture for dense prediction, 1
- 10.1109/CVPR52688.2022.00090 · DOI
- 10.1007/s11263-023-01852-4 · DOI · Paper page
- Tian, Designing BERT for convolutional networks: Sparse and hierarchical masked modeling, 1
- 10.1109/cvpr52729.2023.00264 · DOI
- 10.1109/TPAMI.2022.3217852 · DOI · Paper page
- 10.1609/aaai.v36i3.20227 · DOI
- 10.1109/ICCV48922.2021.00717 · DOI · Paper page
- 10.1109/CVPR52688.2022.00259 · DOI
- 10.1007/978-3-031-19818-2_17 · DOI
- 10.1109/CVPR52688.2022.00439 · DOI · Paper page
- 10.1109/CVPR52688.2022.00102 · DOI
- 10.1109/ICCV48922.2021.00360 · DOI · Paper page
- 10.1109/CVPR52688.2022.00529 · DOI
- Zheng, End-to-end object detection with adaptive clustering transformer, 226
- Roh, Sparse DETR: Efficient end-to-end object detection with learnable sparsity, 1
- Heo, VITA: Video instance segmentation via object token association, 23109
- 10.1109/CVPR52729.2023.01451 · DOI · Paper page
- 10.1007/978-3-031-20059-5_17 · DOI
- 10.1109/wacv57701.2024.00030 · DOI
- Li, Tube-link: A flexible cross tube baseline for universal video segmentation, 13877
- 10.1109/CVPR52729.2023.01733 · DOI
- 10.1007/978-3-031-20077-9_6 · DOI
- 10.1109/CVPR52733.2024.01266 · DOI
- 10.1109/CVPR52729.2023.01868 · DOI
- 10.1109/CVPR52733.2024.01247 · DOI
- 10.1109/TMM.2022.3163578 · DOI
- 10.1109/CVPR42600.2020.01005 · DOI · Paper page
- 10.1109/CVPR52688.2022.00494 · DOI
- 10.1109/ICCV48922.2021.00180 · DOI · Paper page
- 10.1007/s40747-024-01539-4 · DOI
- 10.1109/TIP.2022.3155958 · DOI
- 10.1109/tpami.2024.3421340 · DOI
- 10.1109/CVPR52688.2022.00831 · DOI · Paper page
- 10.1109/CVPR52688.2022.01871 · DOI · Paper page
- 10.1109/icra48891.2023.10160590 · DOI · Paper page
- 10.1609/aaai.v37i2.25335 · DOI
- 10.1609/aaai.v37i2.25329 · DOI
- 10.1109/CVPR52688.2022.01631 · DOI · Paper page
- Zhang, Tip-Adapter: Training-free CLIP-adapter for better vision-language modeling, 493
- 10.1007/978-3-031-19833-5_23 · DOI · Paper page
- Chen, Vision transformer adapter for dense predictions, 1
- 10.1109/CVPR52688.2022.01755 · DOI · Paper page
- 10.1109/CVPR52688.2022.00695 · DOI · Paper page
- 10.1109/CVPR52729.2023.00292 · DOI · Paper page
- 10.1109/CVPR46437.2021.01416 · DOI · Paper page
- Gu, Open-vocabulary object detection via vision and language knowledge distillation, 1
- 10.1007/978-3-031-20077-9_7 · DOI · Paper page
- 10.1109/CVPR52729.2023.01081 · DOI
- Kuo, F-VLM: Open-vocabulary object detection upon frozen vision and language models, 1
- 10.1007/978-3-031-20080-9_30 · DOI
- 10.1007/978-3-031-20059-5_31 · DOI · Paper page
- 10.1109/ICCV51070.2023.02005 · DOI
- 10.1109/CVPR52729.2023.01863 · DOI
- 10.1109/ICCV48922.2021.01060 · DOI
- 10.1109/CVPR52729.2023.00289 · DOI · Paper page
- 10.1109/CVPR52688.2022.00902 · DOI · Paper page
- 10.1109/CVPR52729.2023.00288 · DOI · Paper page
- 10.1109/CVPR42600.2020.01242 · DOI
- 10.1109/CVPR42600.2020.00414 · DOI · Paper page
200 of 265 shown
Cited by
The Segment Anything Model (SAM) for remote sensing applications: From zero to one shot
Osco, L. P., Wu, Q., de Lemos, E. L., Gonçalves, W. N., Ramos, A. P. M., Li, J., & Marcato, J. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103540 · Cited by 338
Rethinking Mobile Block for Efficient Attention-based Models
Zhang, J., Li, X., Li, J., Liu, L., Xue, Z., Zhang, B., Jiang, Z., Huang, T., Wang, Y., & Wang, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00134 · Cited by 313
Self-citation
Metadata
- Authors
- Li, X., Ding, H., Yuan, H., Zhang, W., Pang, J., Cheng, G., Chen, K., Liu, Z., & Loy, C. C.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-12
- Volume
- 46
- Issue
- 12
- Pages
- 10138-10163
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4401070841
- Cited by
- 236
- References
- 88
- FWCI
- 30.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Funders
- Alan Turing Institute (SDCfP2\100009)
- Industry Alignment Fund Industry Collaboration Projects
- Funding Initiative and Singapore MOE AcRF Tier 1 (RG16/21)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Transformer-Based Visual Segmentation: A Survey. https://app.cochat.ai/papers/10.1109/tpami.2024.3434373
- The paper
- Li, X., Ding, H., Yuan, H., Zhang, W., Pang, J., Cheng, G., Chen, K., Liu, Z., & Loy, C. C. (2024). Transformer-Based Visual Segmentation: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2024.3434373
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.