Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 236

The DOI matches its Crossref record; 92 of the 92 references it cites were found in a registry or an index.

Transformer-Based Visual Segmentation: A Survey

Open access
Li, Xiangtai, Ding, Henghui, Yuan, Haobo, Zhang, Wenwei, Pang, Jiangmiao, Cheng, Guangliang, Chen, Kai, Liu, Ziwei, Loy, Chen Change · 2024 · IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract

Visual segmentation seeks to partition images, video frames, or point clouds into multiple segments or groups. This technique has numerous real-world applications, such as autonomous driving, image editing, robot sensing, and medical analysis. Over the past decade, deep learning-based methods have made remarkable strides in this area. Recently, transformers, a type of neural network based on self-attention originally designed for natural language processing, have considerably surpassed previous convolutional or recurrent approaches in various vision processing tasks. Specifically, vision transformers offer robust, unified, and even simpler solutions for various segmentation tasks. This survey provides a thorough overview of transformer-based visual segmentation, summarizing recent advancements. We first review the background, encompassing problem definitions, datasets, and prior convolutional methods. Next, we summarize a meta-architecture that unifies all recent transformer-based approaches. Based on this meta-architecture, we examine various method designs, including modifications to the meta-architecture and associated applications. We also present several specific subfields, including 3D point cloud segmentation, foundation model tuning, domain-aware segmentation, efficient segmentation, and medical segmentation. Additionally, we compile and re-evaluate the reviewed methods on several well-established datasets. Finally, we identify open challenges in this field and propose directions for future research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2024.3434373

Cites

92 of 92 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 263

  1. 10.1023/A:1011174803800 · DOI · Paper page
  2. 10.1109/34.868688 · DOI · Paper page
  3. Stella, Multiclass spectral clustering, 313
  4. 10.5244/C.22.54 · DOI · Paper page
  5. 10.1007/BF00133570 · DOI · Paper page
  6. 10.1007/s11263-015-0816-y · DOI · Paper page
  7. Simonyan, Very deep convolutional networks for large-scale image recognition, 1
  8. 10.1109/CVPR.2015.7298965 · DOI · Paper page
  9. 10.1109/CVPR.2017.660 · DOI · Paper page
  10. 10.1109/CVPR.2018.00254 · DOI · Paper page
  11. 10.48550/ARXIV.1706.03762 · DOI · Paper page
  12. 10.1007/978-3-030-01240-3_17 · DOI · Paper page
  13. 10.1109/CVPR42600.2020.01009 · DOI · Paper page
  14. 10.1109/ICCV.2019.00356 · DOI · Paper page
  15. 10.1109/CVPR46437.2021.00542 · DOI · Paper page
  16. 10.1109/ICCV48922.2021.01595 · DOI · Paper page
  17. 10.1109/CVPR46437.2021.00738 · DOI · Paper page
  18. 10.1109/tpami.2023.3243465 · DOI · Paper page
  19. 10.1109/tpami.2023.3275156 · DOI · Paper page
  20. 10.1016/j.neucom.2019.11.118 · DOI · Paper page
  21. 10.1109/TPAMI.2022.3225573 · DOI · Paper page
  22. 10.1109/CVPR52688.2022.02036 · DOI
  23. 10.1007/978-3-319-10602-1_48 · DOI · Paper page
  24. 10.1109/CVPR52729.2023.02259 · DOI · Paper page
  25. 10.1109/CVPR46437.2021.00412 · DOI
  26. 10.1109/ICCV.2019.00529 · DOI · Paper page
  27. 10.1007/s11263-022-01629-1 · DOI
  28. 10.1109/CVPR42600.2020.00988 · DOI · Paper page
  29. Weber, STEP: Segmenting and tracking every pixel
  30. 10.1109/ICCV51070.2023.01850 · DOI
  31. 10.1109/ICCV51070.2023.00254 · DOI
  32. 10.1109/CVPR.2018.00199 · DOI · Paper page
  33. 10.1109/TIP.2019.2962685 · DOI
  34. 10.1109/CVPR.2019.00909 · DOI · Paper page
  35. 10.1109/TIP.2018.2878975 · DOI
  36. 10.1609/aaai.v34i07.6805 · DOI · Paper page
  37. 10.1109/TIP.2021.3099366 · DOI
  38. 10.48550/arXiv.1909.11065 · arXiv · doi:10.48550/arxiv.1909.11065 · Paper page
  39. Zhang, Dual graph convolutional network for semantic segmentation, 254
  40. 10.1007/978-3-030-58452-8_45 · DOI · Paper page
  41. 10.1109/CVPR42600.2020.00982 · DOI · Paper page
  42. 10.1109/ICCV.2019.00692 · DOI · Paper page
  43. 10.1007/978-3-030-58520-4_26 · DOI · Paper page
  44. 10.1109/ICCV48922.2021.01556 · DOI
  45. 10.1109/CVPR.2019.00326 · DOI · Paper page
  46. 10.1007/978-3-030-58452-8_17 · DOI · Paper page
  47. 10.1109/CVPR.2019.00904 · DOI · Paper page
  48. 10.1109/CVPR42600.2020.01024 · DOI
  49. 10.1109/ICCV.2019.00215 · DOI · Paper page
  50. Wang, SOLOv2: Dynamic and fast instance segmentation
  51. 10.1109/CVPR.2019.00902 · DOI · Paper page
  52. 10.1109/CVPR46437.2021.00028 · DOI · Paper page
  53. 10.1007/978-3-030-58548-8_7 · DOI · Paper page
  54. 10.1109/ICCV.2017.477 · DOI · Paper page
  55. 10.1007/978-3-319-49409-8_69 · DOI · Paper page
  56. 10.1109/CVPR.2017.441 · DOI · Paper page
  57. 10.1109/CVPR42600.2020.00976 · DOI · Paper page
  58. 10.1609/aaai.v35i2.16225 · DOI
  59. 10.1109/TPAMI.2022.3211612 · DOI
  60. 10.1109/CVPR46437.2021.00399 · DOI
  61. 10.1109/CVPR.2016.85 · DOI · Paper page
  62. Qi, PointNet: Deep learning on point sets for 3D classification and segmentation, 652
  63. Qi, PointNet: Deep hierarchical feature learning on point sets in a metric space, 5099
  64. Yang, Learning object bounding boxes for 3D instance segmentation on point clouds, 6737
  65. 10.1109/CVPR.2019.00407 · DOI · Paper page
  66. 10.1109/CVPR.2018.00272 · DOI · Paper page
  67. 10.1109/CVPR42600.2020.00492 · DOI · Paper page
  68. 10.1109/ICCV.2019.00166 · DOI · Paper page
  69. 10.1109/CVPR46437.2021.01236 · DOI · Paper page
  70. 10.1109/CVPR46437.2021.01299 · DOI
  71. 10.1609/aaai.v36i3.20197 · DOI
  72. 10.1109/CVPR46437.2021.01289 · DOI
  73. 10.1109/CVPR46437.2021.00548 · DOI
  74. 10.1109/CVPR46437.2021.00981 · DOI · Paper page
  75. Chen, A2-Nets: Double attention networks, 350
  76. Ren, Faster R-CNN: Towards real-time object detection with region proposal networks, 91
  77. 10.1109/TPAMI.2020.3032166 · DOI · Paper page
  78. 10.1109/CVPR.2019.00720 · DOI · Paper page
  79. Xie, SegFormer: Simple and efficient design for semantic segmentation with transformers, 12077
  80. 10.1109/CVPR52733.2024.02640 · DOI
  81. 10.1109/CVPR52729.2023.01797 · DOI
  82. 10.1007/978-3-031-19812-0_34 · DOI
  83. 10.1109/iccv51070.2023.00240 · DOI
  84. Touvron, Training data-efficient image transformers & distillation through attention, 10347
  85. 10.1109/CVPR52688.2022.00476 · DOI · Paper page
  86. 10.1109/CVPR52688.2022.00714 · DOI · Paper page
  87. Ali, XCiT: Cross-covariance image transformers, 20014
  88. 10.1109/ICCV48922.2021.00041 · DOI · Paper page
  89. 10.1007/978-3-030-58604-1_20 · DOI · Paper page
  90. 10.1109/ICCV48922.2021.00983 · DOI · Paper page
  91. 10.1109/ICCV48922.2021.00009 · DOI · Paper page
  92. Xu, ViTAE: Vision transformer advanced by exploring intrinsic inductive Bias, 28522
  93. 10.1109/CVPR52688.2022.01167 · DOI · Paper page
  94. Han, On the connection between local attention and dynamic depth-wise convolution, 1
  95. Guo, SegNeXt: Rethinking convolutional attention design for semantic segmentation, 1140
  96. 10.1109/CVPR52688.2022.01055 · DOI · Paper page
  97. 10.5555/3524938.3525087 · doi:10.5555/3524938.3525087
  98. Bao, BEiT: BERT pre-training of image transformers, 14668
  99. Gandelsman, Test-time training with masked autoencoders
  100. Gao, ConvMAE: Masked convolution meets masked autoencoders, 35632
  101. 10.1109/cvpr52729.2023.02240 · DOI · Paper page
  102. 10.1109/ICCV48922.2021.00683 · DOI · Paper page
  103. Zhang, K-Net: Towards unified image segmentation, 10326
  104. Cheng, Per-pixel classification is not all you need for semantic segmentation, 17864
  105. 10.1109/CVPR52688.2022.00134 · DOI · Paper page
  106. 10.1109/CVPR46437.2021.00863 · DOI · Paper page
  107. 10.1109/TPAMI.2022.3223955 · DOI · Paper page
  108. 10.1109/CVPR52688.2022.00290 · DOI
  109. Hwang, Video instance segmentation using inter-frame communication transformers, 13352
  110. 10.1007/978-3-031-19815-1_32 · DOI
  111. 10.1109/CVPR52688.2022.01828 · DOI
  112. 10.1109/CVPR52688.2022.01354 · DOI
  113. 10.1609/aaai.v36i3.20158 · DOI · Paper page
  114. Liu, DAB-DETR: Dynamic anchor boxes are better queries for DETR, 1
  115. 10.1109/CVPR52729.2023.00297 · DOI · Paper page
  116. Wang, Learning equivariant segmentation with instance-unique querying
  117. 10.1109/CVPR52729.2023.01887 · DOI · Paper page
  118. Huang, MinVIS: A minimal video instance segmentation framework without video-based training, 31265
  119. 10.1007/978-3-031-19815-1_34 · DOI
  120. 10.1007/978-3-031-19812-0_42 · DOI
  121. 10.1109/CVPR52688.2022.00168 · DOI
  122. 10.1007/978-3-031-19836-6_31 · DOI
  123. 10.1007/978-3-031-19812-0_30 · DOI
  124. 10.1109/ICCV48922.2021.01601 · DOI · Paper page
  125. 10.1109/CVPR52688.2022.01762 · DOI · Paper page
  126. 10.1109/CVPR52688.2022.01761 · DOI · Paper page
  127. 10.1109/CVPR52688.2022.01139 · DOI · Paper page
  128. 10.1109/CVPR52688.2022.00493 · DOI · Paper page
  129. 10.1109/CVPR52688.2022.00491 · DOI
  130. 10.1109/tip.2024.3371348 · DOI
  131. 10.1109/CVPR52688.2022.00492 · DOI · Paper page
  132. Zhang, Few-shot segmentation via cycle-consistent transformer, 21984
  133. Yang, Associating objects with transformers for video object segmentation, 2491
  134. 10.1109/CVPR52688.2022.01140 · DOI
  135. 10.1007/978-3-031-19815-1_36 · DOI
  136. 10.1109/tcsvt.2023.3252807 · DOI
  137. 10.1007/978-3-031-19818-2_35 · DOI
  138. 10.1109/CVPR52688.2022.01170 · DOI · Paper page
  139. Chen, CycleMLP: A MLP-like architecture for dense prediction, 1
  140. 10.1109/CVPR52688.2022.00090 · DOI
  141. 10.1007/s11263-023-01852-4 · DOI · Paper page
  142. Tian, Designing BERT for convolutional networks: Sparse and hierarchical masked modeling, 1
  143. 10.1109/cvpr52729.2023.00264 · DOI
  144. 10.1109/TPAMI.2022.3217852 · DOI · Paper page
  145. 10.1609/aaai.v36i3.20227 · DOI
  146. 10.1109/ICCV48922.2021.00717 · DOI · Paper page
  147. 10.1109/CVPR52688.2022.00259 · DOI
  148. 10.1007/978-3-031-19818-2_17 · DOI
  149. 10.1109/CVPR52688.2022.00439 · DOI · Paper page
  150. 10.1109/CVPR52688.2022.00102 · DOI
  151. 10.1109/ICCV48922.2021.00360 · DOI · Paper page
  152. 10.1109/CVPR52688.2022.00529 · DOI
  153. Zheng, End-to-end object detection with adaptive clustering transformer, 226
  154. Roh, Sparse DETR: Efficient end-to-end object detection with learnable sparsity, 1
  155. Heo, VITA: Video instance segmentation via object token association, 23109
  156. 10.1109/CVPR52729.2023.01451 · DOI · Paper page
  157. 10.1007/978-3-031-20059-5_17 · DOI
  158. 10.1109/wacv57701.2024.00030 · DOI
  159. Li, Tube-link: A flexible cross tube baseline for universal video segmentation, 13877
  160. 10.1109/CVPR52729.2023.01733 · DOI
  161. 10.1007/978-3-031-20077-9_6 · DOI
  162. 10.1109/CVPR52733.2024.01266 · DOI
  163. 10.1109/CVPR52729.2023.01868 · DOI
  164. 10.1109/CVPR52733.2024.01247 · DOI
  165. 10.1109/TMM.2022.3163578 · DOI
  166. 10.1109/CVPR42600.2020.01005 · DOI · Paper page
  167. 10.1109/CVPR52688.2022.00494 · DOI
  168. 10.1109/ICCV48922.2021.00180 · DOI · Paper page
  169. 10.1007/s40747-024-01539-4 · DOI
  170. 10.1109/TIP.2022.3155958 · DOI
  171. 10.1109/tpami.2024.3421340 · DOI
  172. 10.1109/CVPR52688.2022.00831 · DOI · Paper page
  173. 10.1109/CVPR52688.2022.01871 · DOI · Paper page
  174. 10.1109/icra48891.2023.10160590 · DOI · Paper page
  175. 10.1609/aaai.v37i2.25335 · DOI
  176. 10.1609/aaai.v37i2.25329 · DOI
  177. 10.1109/CVPR52688.2022.01631 · DOI · Paper page
  178. Zhang, Tip-Adapter: Training-free CLIP-adapter for better vision-language modeling, 493
  179. 10.1007/978-3-031-19833-5_23 · DOI · Paper page
  180. Chen, Vision transformer adapter for dense predictions, 1
  181. 10.1109/CVPR52688.2022.01755 · DOI · Paper page
  182. 10.1109/CVPR52688.2022.00695 · DOI · Paper page
  183. 10.1109/CVPR52729.2023.00292 · DOI · Paper page
  184. 10.1109/CVPR46437.2021.01416 · DOI · Paper page
  185. Gu, Open-vocabulary object detection via vision and language knowledge distillation, 1
  186. 10.1007/978-3-031-20077-9_7 · DOI · Paper page
  187. 10.1109/CVPR52729.2023.01081 · DOI
  188. Kuo, F-VLM: Open-vocabulary object detection upon frozen vision and language models, 1
  189. 10.1007/978-3-031-20080-9_30 · DOI
  190. 10.1007/978-3-031-20059-5_31 · DOI · Paper page
  191. 10.1109/ICCV51070.2023.02005 · DOI
  192. 10.1109/CVPR52729.2023.01863 · DOI
  193. 10.1109/ICCV48922.2021.01060 · DOI
  194. 10.1109/CVPR52729.2023.00289 · DOI · Paper page
  195. 10.1109/CVPR52688.2022.00902 · DOI · Paper page
  196. 10.1109/CVPR52729.2023.00288 · DOI · Paper page
  197. 10.1109/CVPR42600.2020.01242 · DOI
  198. 10.1109/CVPR42600.2020.00414 · DOI · Paper page
  199. 10.1109/CVPR52688.2022.00969 · DOI · Paper page
  200. 10.1007/978-3-031-20056-4_22 · DOI · Paper page

200 of 263 shown

Cited by

See all 236

Metadata

Authors
Li, X., Ding, H., Yuan, H., Zhang, W., Pang, J., Cheng, G., Chen, K., Liu, Z., & Loy, C. C.
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Published
2024-12
Volume
46
Issue
12
Pages
10138-10163
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
OpenAlex
W4401070841
Cited by
236
References
92
FWCI
30.37
Type
journal article
Language
en
Open access
hybrid
Funders
  • Alan Turing Institute (SDCfP2\100009)
  • Industry Alignment Fund Industry Collaboration Projects
  • Funding Initiative and Singapore MOE AcRF Tier 1 (RG16/21)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.