Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 236

The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.

Transformer-Based Visual Segmentation: A Survey

Open access
Li, Xiangtai, Ding, Henghui, Yuan, Haobo, Zhang, Wenwei, Pang, Jiangmiao, Cheng, Guangliang, Chen, Kai, Liu, Ziwei, Loy, Chen Change · 2024 · IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract

Visual segmentation seeks to partition images, video frames, or point clouds into multiple segments or groups. This technique has numerous real-world applications, such as autonomous driving, image editing, robot sensing, and medical analysis. Over the past decade, deep learning-based methods have made remarkable strides in this area. Recently, transformers, a type of neural network based on self-attention originally designed for natural language processing, have considerably surpassed previous convolutional or recurrent approaches in various vision processing tasks. Specifically, vision transformers offer robust, unified, and even simpler solutions for various segmentation tasks. This survey provides a thorough overview of transformer-based visual segmentation, summarizing recent advancements. We first review the background, encompassing problem definitions, datasets, and prior convolutional methods. Next, we summarize a meta-architecture that unifies all recent transformer-based approaches. Based on this meta-architecture, we examine various method designs, including modifications to the meta-architecture and associated applications. We also present several specific subfields, including 3D point cloud segmentation, foundation model tuning, domain-aware segmentation, efficient segmentation, and medical segmentation. Additionally, we compile and re-evaluate the reviewed methods on several well-established datasets. Finally, we identify open challenges in this field and propose directions for future research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2024.3434373

Cites

88 of 88 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 88 references

Research works printed but not linked: 265

  1. 10.1023/A:1011174803800 · DOI · Paper page
  2. 10.1109/34.868688 · DOI · Paper page
  3. Stella, Multiclass spectral clustering, 313
  4. 10.5244/C.22.54 · DOI · Paper page
  5. 10.1007/BF00133570 · DOI · Paper page
  6. 10.1007/s11263-015-0816-y · DOI · Paper page
  7. Simonyan, Very deep convolutional networks for large-scale image recognition, 1
  8. 10.1109/CVPR.2015.7298965 · DOI · Paper page
  9. 10.1109/TPAMI.2017.2699184 · DOI · Paper page
  10. 10.1109/CVPR.2017.660 · DOI · Paper page
  11. 10.1109/CVPR.2018.00254 · DOI · Paper page
  12. 10.48550/ARXIV.1706.03762 · DOI · Paper page
  13. 10.1007/978-3-030-01240-3_17 · DOI · Paper page
  14. 10.1109/CVPR42600.2020.01009 · DOI · Paper page
  15. 10.1109/ICCV.2019.00356 · DOI · Paper page
  16. 10.1109/CVPR46437.2021.00542 · DOI · Paper page
  17. 10.1109/ICCV48922.2021.01595 · DOI · Paper page
  18. 10.1109/CVPR46437.2021.00738 · DOI · Paper page
  19. 10.1109/tpami.2023.3243465 · DOI · Paper page
  20. 10.1109/tpami.2023.3275156 · DOI · Paper page
  21. 10.1016/j.neucom.2019.11.118 · DOI · Paper page
  22. 10.1109/TPAMI.2022.3225573 · DOI · Paper page
  23. 10.1109/CVPR52688.2022.02036 · DOI
  24. 10.1007/978-3-319-10602-1_48 · DOI · Paper page
  25. 10.1109/CVPR52729.2023.02259 · DOI · Paper page
  26. 10.1109/CVPR46437.2021.00412 · DOI
  27. 10.1109/ICCV.2019.00529 · DOI · Paper page
  28. 10.1007/s11263-022-01629-1 · DOI
  29. 10.1109/CVPR42600.2020.00988 · DOI · Paper page
  30. Weber, STEP: Segmenting and tracking every pixel
  31. 10.1109/ICCV51070.2023.01850 · DOI
  32. 10.1109/ICCV51070.2023.00254 · DOI
  33. 10.1109/CVPR.2018.00199 · DOI · Paper page
  34. 10.1109/TIP.2019.2962685 · DOI
  35. 10.1109/CVPR.2019.00909 · DOI · Paper page
  36. 10.1109/TIP.2018.2878975 · DOI
  37. 10.1609/aaai.v34i07.6805 · DOI · Paper page
  38. 10.1109/TIP.2021.3099366 · DOI
  39. 10.48550/arXiv.1909.11065 · arXiv · doi:10.48550/arxiv.1909.11065 · Paper page
  40. Zhang, Dual graph convolutional network for semantic segmentation, 254
  41. 10.1007/978-3-030-58452-8_45 · DOI · Paper page
  42. 10.1109/CVPR42600.2020.00982 · DOI · Paper page
  43. 10.1109/ICCV.2019.00692 · DOI · Paper page
  44. 10.1007/978-3-030-58520-4_26 · DOI · Paper page
  45. 10.1109/ICCV48922.2021.01556 · DOI
  46. 10.1109/CVPR.2019.00326 · DOI · Paper page
  47. 10.1007/978-3-030-58452-8_17 · DOI · Paper page
  48. 10.1109/CVPR.2019.00904 · DOI · Paper page
  49. 10.1109/CVPR42600.2020.01024 · DOI
  50. 10.1109/CVPR42600.2020.01249 · DOI · Paper page
  51. 10.1109/ICCV.2019.00215 · DOI · Paper page
  52. Wang, SOLOv2: Dynamic and fast instance segmentation
  53. 10.1109/CVPR.2019.00902 · DOI · Paper page
  54. 10.1109/CVPR46437.2021.00028 · DOI · Paper page
  55. 10.1007/978-3-030-58548-8_7 · DOI · Paper page
  56. 10.1109/ICCV.2017.477 · DOI · Paper page
  57. 10.1007/978-3-319-49409-8_69 · DOI · Paper page
  58. 10.1109/CVPR.2017.441 · DOI · Paper page
  59. 10.1109/CVPR42600.2020.00976 · DOI · Paper page
  60. 10.1609/aaai.v35i2.16225 · DOI
  61. 10.1109/TPAMI.2022.3211612 · DOI
  62. 10.1109/CVPR46437.2021.00399 · DOI
  63. 10.1109/CVPR.2016.85 · DOI · Paper page
  64. Qi, PointNet: Deep learning on point sets for 3D classification and segmentation, 652
  65. Qi, PointNet: Deep hierarchical feature learning on point sets in a metric space, 5099
  66. Yang, Learning object bounding boxes for 3D instance segmentation on point clouds, 6737
  67. 10.1109/CVPR.2019.00407 · DOI · Paper page
  68. 10.1109/CVPR.2018.00272 · DOI · Paper page
  69. 10.1109/CVPR42600.2020.00492 · DOI · Paper page
  70. 10.1109/ICCV.2019.00166 · DOI · Paper page
  71. 10.1109/CVPR46437.2021.01236 · DOI · Paper page
  72. 10.1109/CVPR46437.2021.01299 · DOI
  73. 10.1609/aaai.v36i3.20197 · DOI
  74. 10.1109/CVPR46437.2021.01289 · DOI
  75. 10.1109/CVPR46437.2021.00548 · DOI
  76. 10.1109/CVPR46437.2021.00981 · DOI · Paper page
  77. Chen, A2-Nets: Double attention networks, 350
  78. Ren, Faster R-CNN: Towards real-time object detection with region proposal networks, 91
  79. 10.1109/TPAMI.2020.3032166 · DOI · Paper page
  80. 10.1109/CVPR.2019.00720 · DOI · Paper page
  81. Xie, SegFormer: Simple and efficient design for semantic segmentation with transformers, 12077
  82. 10.1109/CVPR52733.2024.02640 · DOI
  83. 10.1109/CVPR52729.2023.01797 · DOI
  84. 10.1007/978-3-031-19812-0_34 · DOI
  85. 10.1109/iccv51070.2023.00240 · DOI
  86. Touvron, Training data-efficient image transformers & distillation through attention, 10347
  87. 10.1109/CVPR52688.2022.00476 · DOI · Paper page
  88. 10.1109/CVPR52688.2022.00714 · DOI · Paper page
  89. Ali, XCiT: Cross-covariance image transformers, 20014
  90. 10.1109/ICCV48922.2021.00041 · DOI · Paper page
  91. 10.1007/978-3-030-58604-1_20 · DOI · Paper page
  92. 10.1109/ICCV48922.2021.00983 · DOI · Paper page
  93. 10.1109/ICCV48922.2021.00009 · DOI · Paper page
  94. Xu, ViTAE: Vision transformer advanced by exploring intrinsic inductive Bias, 28522
  95. 10.1109/CVPR52688.2022.01167 · DOI · Paper page
  96. Han, On the connection between local attention and dynamic depth-wise convolution, 1
  97. Guo, SegNeXt: Rethinking convolutional attention design for semantic segmentation, 1140
  98. 10.1109/CVPR52688.2022.01055 · DOI · Paper page
  99. 10.5555/3524938.3525087 · doi:10.5555/3524938.3525087
  100. Bao, BEiT: BERT pre-training of image transformers, 14668
  101. Gandelsman, Test-time training with masked autoencoders
  102. Gao, ConvMAE: Masked convolution meets masked autoencoders, 35632
  103. 10.1109/cvpr52729.2023.02240 · DOI · Paper page
  104. 10.1109/ICCV48922.2021.00683 · DOI · Paper page
  105. Zhang, K-Net: Towards unified image segmentation, 10326
  106. Cheng, Per-pixel classification is not all you need for semantic segmentation, 17864
  107. 10.1109/CVPR52688.2022.00134 · DOI · Paper page
  108. 10.1109/CVPR46437.2021.00863 · DOI · Paper page
  109. 10.1109/TPAMI.2022.3223955 · DOI · Paper page
  110. 10.1109/CVPR52688.2022.00290 · DOI
  111. Hwang, Video instance segmentation using inter-frame communication transformers, 13352
  112. 10.1007/978-3-031-19815-1_32 · DOI
  113. 10.1109/CVPR52688.2022.01828 · DOI
  114. 10.1109/CVPR52688.2022.01354 · DOI
  115. 10.1609/aaai.v36i3.20158 · DOI · Paper page
  116. Liu, DAB-DETR: Dynamic anchor boxes are better queries for DETR, 1
  117. 10.1109/CVPR52729.2023.00297 · DOI · Paper page
  118. Wang, Learning equivariant segmentation with instance-unique querying
  119. 10.1109/CVPR52729.2023.01887 · DOI · Paper page
  120. Huang, MinVIS: A minimal video instance segmentation framework without video-based training, 31265
  121. 10.1007/978-3-031-19815-1_34 · DOI
  122. 10.1007/978-3-031-19812-0_42 · DOI
  123. 10.1109/CVPR52688.2022.00168 · DOI
  124. 10.1007/978-3-031-19836-6_31 · DOI
  125. 10.1007/978-3-031-19812-0_30 · DOI
  126. 10.1109/ICCV48922.2021.01601 · DOI · Paper page
  127. 10.1109/CVPR52688.2022.01762 · DOI · Paper page
  128. 10.1109/CVPR52688.2022.01761 · DOI · Paper page
  129. 10.1109/CVPR52688.2022.01139 · DOI · Paper page
  130. 10.1109/CVPR52688.2022.00493 · DOI · Paper page
  131. 10.1109/CVPR52688.2022.00491 · DOI
  132. 10.1109/tip.2024.3371348 · DOI
  133. 10.1109/CVPR52688.2022.00492 · DOI · Paper page
  134. Zhang, Few-shot segmentation via cycle-consistent transformer, 21984
  135. Yang, Associating objects with transformers for video object segmentation, 2491
  136. 10.1109/CVPR52688.2022.01140 · DOI
  137. 10.1007/978-3-031-19815-1_36 · DOI
  138. 10.1109/tcsvt.2023.3252807 · DOI
  139. 10.1007/978-3-031-19818-2_35 · DOI
  140. 10.1109/CVPR52688.2022.01170 · DOI · Paper page
  141. Chen, CycleMLP: A MLP-like architecture for dense prediction, 1
  142. 10.1109/CVPR52688.2022.00090 · DOI
  143. 10.1007/s11263-023-01852-4 · DOI · Paper page
  144. Tian, Designing BERT for convolutional networks: Sparse and hierarchical masked modeling, 1
  145. 10.1109/cvpr52729.2023.00264 · DOI
  146. 10.1109/TPAMI.2022.3217852 · DOI · Paper page
  147. 10.1609/aaai.v36i3.20227 · DOI
  148. 10.1109/ICCV48922.2021.00717 · DOI · Paper page
  149. 10.1109/CVPR52688.2022.00259 · DOI
  150. 10.1007/978-3-031-19818-2_17 · DOI
  151. 10.1109/CVPR52688.2022.00439 · DOI · Paper page
  152. 10.1109/CVPR52688.2022.00102 · DOI
  153. 10.1109/ICCV48922.2021.00360 · DOI · Paper page
  154. 10.1109/CVPR52688.2022.00529 · DOI
  155. Zheng, End-to-end object detection with adaptive clustering transformer, 226
  156. Roh, Sparse DETR: Efficient end-to-end object detection with learnable sparsity, 1
  157. Heo, VITA: Video instance segmentation via object token association, 23109
  158. 10.1109/CVPR52729.2023.01451 · DOI · Paper page
  159. 10.1007/978-3-031-20059-5_17 · DOI
  160. 10.1109/wacv57701.2024.00030 · DOI
  161. Li, Tube-link: A flexible cross tube baseline for universal video segmentation, 13877
  162. 10.1109/CVPR52729.2023.01733 · DOI
  163. 10.1007/978-3-031-20077-9_6 · DOI
  164. 10.1109/CVPR52733.2024.01266 · DOI
  165. 10.1109/CVPR52729.2023.01868 · DOI
  166. 10.1109/CVPR52733.2024.01247 · DOI
  167. 10.1109/TMM.2022.3163578 · DOI
  168. 10.1109/CVPR42600.2020.01005 · DOI · Paper page
  169. 10.1109/CVPR52688.2022.00494 · DOI
  170. 10.1109/ICCV48922.2021.00180 · DOI · Paper page
  171. 10.1007/s40747-024-01539-4 · DOI
  172. 10.1109/TIP.2022.3155958 · DOI
  173. 10.1109/tpami.2024.3421340 · DOI
  174. 10.1109/CVPR52688.2022.00831 · DOI · Paper page
  175. 10.1109/CVPR52688.2022.01871 · DOI · Paper page
  176. 10.1109/icra48891.2023.10160590 · DOI · Paper page
  177. 10.1609/aaai.v37i2.25335 · DOI
  178. 10.1609/aaai.v37i2.25329 · DOI
  179. 10.1109/CVPR52688.2022.01631 · DOI · Paper page
  180. Zhang, Tip-Adapter: Training-free CLIP-adapter for better vision-language modeling, 493
  181. 10.1007/978-3-031-19833-5_23 · DOI · Paper page
  182. Chen, Vision transformer adapter for dense predictions, 1
  183. 10.1109/CVPR52688.2022.01755 · DOI · Paper page
  184. 10.1109/CVPR52688.2022.00695 · DOI · Paper page
  185. 10.1109/CVPR52729.2023.00292 · DOI · Paper page
  186. 10.1109/CVPR46437.2021.01416 · DOI · Paper page
  187. Gu, Open-vocabulary object detection via vision and language knowledge distillation, 1
  188. 10.1007/978-3-031-20077-9_7 · DOI · Paper page
  189. 10.1109/CVPR52729.2023.01081 · DOI
  190. Kuo, F-VLM: Open-vocabulary object detection upon frozen vision and language models, 1
  191. 10.1007/978-3-031-20080-9_30 · DOI
  192. 10.1007/978-3-031-20059-5_31 · DOI · Paper page
  193. 10.1109/ICCV51070.2023.02005 · DOI
  194. 10.1109/CVPR52729.2023.01863 · DOI
  195. 10.1109/ICCV48922.2021.01060 · DOI
  196. 10.1109/CVPR52729.2023.00289 · DOI · Paper page
  197. 10.1109/CVPR52688.2022.00902 · DOI · Paper page
  198. 10.1109/CVPR52729.2023.00288 · DOI · Paper page
  199. 10.1109/CVPR42600.2020.01242 · DOI
  200. 10.1109/CVPR42600.2020.00414 · DOI · Paper page

200 of 265 shown

Cited by

See all 236

Metadata

Authors
Li, X., Ding, H., Yuan, H., Zhang, W., Pang, J., Cheng, G., Chen, K., Liu, Z., & Loy, C. C.
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Published
2024-12
Volume
46
Issue
12
Pages
10138-10163
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
OpenAlex
W4401070841
Cited by
236
References
88
FWCI
30.37
Type
journal article
Language
en
Open access
hybrid
Funders
  • Alan Turing Institute (SDCfP2\100009)
  • Industry Alignment Fund Industry Collaboration Projects
  • Funding Initiative and Singapore MOE AcRF Tier 1 (RG16/21)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Transformer-Based Visual Segmentation: A Survey. https://app.cochat.ai/papers/10.1109/tpami.2024.3434373
The paper
Li, X., Ding, H., Yuan, H., Zhang, W., Pang, J., Cheng, G., Chen, K., Liu, Z., & Loy, C. C. (2024). Transformer-Based Visual Segmentation: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2024.3434373

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.