The DOI matches its Crossref record; 92 of the 92 references it cites were found in a registry or an index.
Transformer-Based Visual Segmentation: A Survey
Open accessAbstract
Visual segmentation seeks to partition images, video frames, or point clouds into multiple segments or groups. This technique has numerous real-world applications, such as autonomous driving, image editing, robot sensing, and medical analysis. Over the past decade, deep learning-based methods have made remarkable strides in this area. Recently, transformers, a type of neural network based on self-attention originally designed for natural language processing, have considerably surpassed previous convolutional or recurrent approaches in various vision processing tasks. Specifically, vision transformers offer robust, unified, and even simpler solutions for various segmentation tasks. This survey provides a thorough overview of transformer-based visual segmentation, summarizing recent advancements. We first review the background, encompassing problem definitions, datasets, and prior convolutional methods. Next, we summarize a meta-architecture that unifies all recent transformer-based approaches. Based on this meta-architecture, we examine various method designs, including modifications to the meta-architecture and associated applications. We also present several specific subfields, including 3D point cloud segmentation, foundation model tuning, domain-aware segmentation, efficient segmentation, and medical segmentation. Additionally, we compile and re-evaluate the reviewed methods on several well-established datasets. Finally, we identify open challenges in this field and propose directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Self-citationFeature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · cited 1×
“• Neck: Feature pyramid network (FPN) has been shown effective in object detection and instance segmentation [116], [117], [118] for scale variation modeling.”
BackgroundHe, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · arXiv · doi:10.48550/arxiv.1802.02611 · Cited by 20,063
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
The Hungarian method for the assignment problem
Kuhn, H. W. (1955) · Naval Research Logistics Quarterly · doi:10.1002/nav.3800020109 · Cited by 13,120
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
Milletari, F., Navab, N., & Ahmadi, S.-A. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.79 · Cited by 11,931
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 4×
“Motivated by MIM [24], MIC [293] proposes a masked image consistency to learn spatial context relations of the target domain as additional clues.”
BackgroundMethodInfluentialEmerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 3×
“The following works focus on improving the MIM framework [153], [154] or replacing the backbone of ViTs with CNN architecture [155], [216].”
BackgroundPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 1×
“Pyramid ViT [135] is the first work to build multiscale features for detection and segmentation tasks.”
BackgroundSelf-citationHan (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206 · cited 2×
“TransUNet [345] merges transformer and U-Net, where the transformer encodes tokenized image patches to build the global context.”
BackgroundRethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 2×
“In semantic segmentation, methods such as Segformer [123] and SETR [124] employ a fully connected layer and predict each pixel as previous FCN-based methods [9], [62], [125].”
MethodRethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.15840 · Cited by 4,004
Image segmentation using deep learning: A survey
Minaee, S., Boykov, Y. Y., Porikli, F., Plaza, A. J., Kehtarnavaz, N., & Terzopoulos, D. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3059968 · Cited by 3,796
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
UNETR: Transformers for 3D Medical Image Segmentation
Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H. R., & Xu, D. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00181 · Cited by 3,586 · cited 1×
“UNETR [348] focuses on 3D input medical images and designs a similar U-Net-like architecture.”
BackgroundUNETR: Transformers for 3D Medical Image Segmentation
Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H., & Xu, D. (2021) · arXiv · doi:10.48550/arxiv.2103.10504 · Cited by 3,580
Masked-attention Mask Transformer for Universal Image Segmentation
Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., & Girdhar, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00135 · Cited by 3,565
BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation
Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., & Sang, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_20 · Cited by 2,968
Deformable ConvNets V2: More Deformable, Better Results
Zhu, X., Hu, H., Lin, S., & Dai, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00953 · Cited by 2,943
Swin Transformer V2: Scaling Up Capacity and Resolution
Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.09883 · Cited by 2,679
Self-citationYOLACT: Real-Time Instance Segmentation
Bolya, D., Zhou, C., Xiao, F., & Lee, Y. J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00925 · Cited by 2,536 · cited 1×
“The designs of mask heads are various, including FCN heads [76], [80], diverse mask encodings [81], and dynamic kernels [77], [82].”
BackgroundPre-Trained Image Processing Transformer
Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01212 · Cited by 2,308 · cited 1×
“Because of a rapid upsurge in transformer-based vision models, there are several surveys on vision transformer [31], [32], [33].”
BackgroundSelf-citationLiu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2021) · arXiv · doi:10.48550/arxiv.2106.13230 · Cited by 2,294
Self-citationLiu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00320 · Cited by 2,294
Self-citationRandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds
Hu, Q., Yang, B., Xie, L., Rosa, S., Guo, Y., Wang, Z., Trigoni, N., & Markham, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01112 · Cited by 2,275
Guo, M.-H., Cai, J.-X., Liu, Z.-N., Mu, T.-J., Martin, R. R., & Hu, S.-M. (2021) · Computational Visual Media · doi:10.1007/s41095-021-0229-5 · Cited by 2,100
Self-citationMLP-Mixer: An all-MLP Architecture for Vision
Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.01601 · Cited by 2,087
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977
Semantic Understanding of Scenes Through the ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., & Torralba, A. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1140-0 · Cited by 1,831
Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network
Peng, C., Zhang, X., Yu, G., Luo, G., & Sun, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.189 · Cited by 1,750 · cited 1×
“These methods can be divided into the following aspects, including better encoder-decoder frameworks [58], [59], larger kernels [60], [61], multiscale pooling [11], [62], multiscale feature fusion [12], [63], [64], [65], non-local modeling [18], [66], [67], efficient modeling [68], [69], [70], and…”
MethodICNet for Real-Time Semantic Segmentation on High-Resolution Images
Zhao, H., Qi, X., Shen, X., Shi, J., & Jia, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_25 · Cited by 1,709
Is Space-Time Attention All You Need for Video Understanding?
Bertasius, G., Wang, H., & Torresani, L. (2021) · arXiv · doi:10.48550/arxiv.2102.05095 · Cited by 1,663
An Empirical Study of Training Self-Supervised Vision Transformers
Chen, X., Xie, S., & He, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00950 · Cited by 1,648
Sparse R-CNN: End-to-End Object Detection with Learnable Proposals
Sun, P., Zhang, R., Jiang, Y., Kong, T., Xu, C., Zhan, W., Tomizuka, M., Li, L., Yuan, Z., Wang, C., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01422 · Cited by 1,617
Hybrid Task Cascade for Instance Segmentation
Chen, K., Pang, J., Wang, J., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Shi, J., Ouyang, W., et al. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00511 · Cited by 1,587 · cited 2×
“The designs of mask heads are various, including FCN heads [76], [80], diverse mask encodings [81], and dynamic kernels [77], [82].”
BackgroundMethodSelf-citationLarge-Scale Point Cloud Semantic Segmentation with Superpoint Graphs
Landrieu, L., & Simonovsky, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00479 · Cited by 1,543
The Role of Context for Object Detection and Semantic Segmentation in the Wild
Mottaghi, R., Chen, X., Liu, X., Cho, N.-G., Lee, S.-W., Fidler, S., Urtasun, R., & Yuille, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.119 · Cited by 1,438 · cited 2×
“Pascal Context [42] 4,998 / 5,105 SS mIoU PASCAL Context dataset is an extension of PASCAL VOC containing 400+ classes (usually 59 most frequently).”
BackgroundThe Mapillary Vistas Dataset for Semantic Understanding of Street Scenes
Neuhold, G., Ollmann, T., Bulo, S. R., & Kontschieder, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.534 · Cited by 1,434 · cited 1×
“For PS, depending on the class definition, each class may have a different number of masks.”
BackgroundLin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422
Multiscale vision transformers
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00675 · Cited by 1,407 · cited 1×
“MViT-V1 [131] introduces the multiscale feature representation and pooling strategies to reduce the computation cost in MHSA.”
MethodObject-Contextual Representations for Semantic Segmentation
Yuan, Y., Chen, X., & Wang, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_11 · Cited by 1,285
DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution
Qiao, S., Chen, L.-C., & Yuille, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01008 · Cited by 1,245 · cited 3×
“Several works [83], [119] design stronger FPNs via cross-scale modeling using dilation or deformable convolution.”
BackgroundMethodDINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., & Shum, H.-Y. (2022) · arXiv · Cited by 1,154 · cited 2×
“The subfields include point cloud segmentation, domain-aware segmentation, label and model efficient segmentation, class agnostic segmentation, tracking, and medical segmentation.”
BackgroundMethodMobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
Mehta, S., & Rastegari, M. (2021) · arXiv · doi:10.48550/arxiv.2110.02178 · Cited by 1,110 · cited 2×
“In particular, Mobile-ViT [323] introduces the first transformer backbone for mobile devices.”
BackgroundDN-DETR: Accelerate DETR Training by Introducing Query DeNoising
Li, F., Zhang, H., Liu, S., Guo, J., Ni, L. M., & Zhang, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01325 · Cited by 1,102
TrackFormer: Multi-Object Tracking with Transformers
Meinhardt, T., Kirillov, A., Leal-Taixe, L., & Feichtenhofer, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00864 · Cited by 1,081 · cited 1×
“TrackFormer [185] is the first to treat multi-object tracking as a set prediction problem by performing joint detection and tracking-by-attention.”
Background(2016) · doi:10.1007/978-3-319-46475-6_5 · Cited by 1,081
Conditional DETR for Fast Training Convergence
Meng, D., Chen, X., Fan, Z., Zeng, G., Li, H., Yuan, Y., Sun, L., & Wang, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00363 · Cited by 1,055
CMT: Convolutional Neural Networks Meet Vision Transformers
Guo, J., Han, K., Wu, H., Tang, Y., Chen, X., Wang, Y., & Xu, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01186 · Cited by 1,005
CMT: Convolutional Neural Networks Meet Vision Transformers
Guo, J., Han, K., Wu, H., Tang, Y., Chen, X., Wang, Y., & Xu, C. (2021) · arXiv · doi:10.48550/arxiv.2107.06263 · Cited by 1,005
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.12602 · Cited by 909 · cited 1×
“For video representation learning, most current works [219], [220], [221] verify such representation learning on action or motion learning, such as action recognition.”
BackgroundTwins: Revisiting the Design of Spatial Attention in Vision Transformers
Chu, X., Tian, Z., Wang, Y., Zhang, B., Ren, H., Wei, X., Xia, H., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2104.13840 · Cited by 817
MOTR: End-to-End Multiple-Object Tracking with Transformer
Zeng, F., Dong, B., Zhang, Y., Wang, T., Zhang, X., & Wei, Y. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19812-0_38 · Cited by 752
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Objects365: A Large-Scale, High-Quality Dataset for Object Detection
Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., & Sun, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00852 · Cited by 673 · cited 1×
“However, due to the extra datasets [352] they used, we do not list them here.”
BackgroundMOTS: Multi-Object Tracking and Segmentation
Voigtlaender, P., Krause, M., Osep, A., Luiten, J., Sekar, B. B. G., Geiger, A., & Leibe, B. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00813 · Cited by 658
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation
Cheng, B., Collins, M. D., Zhu, Y., Liu, T., Huang, T. S., Adam, H., & Chen, L.-C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01249 · Cited by 635
Masked Autoencoders for Point Cloud Self-supervised Learning
Pang, Y., Wang, W., Tay, F. E. H., Liu, W., Tian, Y., & Yuan, L. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20086-1_35 · Cited by 605
Masked Feature Prediction for Self-Supervised Visual Pre-Training
Wei, C., Fan, H., Xie, S., Wu, C.-Y., Yuille, A., & Feichtenhofer, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01426 · Cited by 587
Detecting Twenty-Thousand Classes Using Image-Level Supervision
Zhou, X., Girdhar, R., Joulin, A., Krähenbühl, P., & Misra, I. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20077-9_21 · Cited by 564
Dynamic DETR: End-to-End Object Detection with Dynamic Attention
Dai, X., Chen, Y., Yang, J., Zhang, P., Yuan, L., & Zhang, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00298 · Cited by 456 · cited 1×
“From the feature re-weighting view, Dynamic-DETR [234] introduces dynamic attention to both the encoder and decoder parts of DETR using RoI-wise dynamic convolution.”
MethodXMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model
Cheng, H. K., & Schwing, A. G. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19815-1_37 · Cited by 449
Edgenext: Efficiently amalgamated CNN-transformer architecture for mobile vision applications
Maaz, M., Shaker, A., Cholakkal, H., Khan, S., Zamir, S. W., Anwer, R. M., & Shahbaz Khan, F. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-25082-8_1 · Cited by 407
Language-driven Semantic Segmentation
Li, B., Weinberger, K. Q., Belongie, S., Koltun, V., & Ranftl, R. (2022) · arXiv · doi:10.48550/arxiv.2201.03546 · Cited by 316 · cited 2×
“Besidestheseopen vocabularydetectionsettings,recentworks[281],[282]per-formopenvocabularysegmentation.”
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark
Xu, N., Yang, L., Fan, Y., Yue, D., Liang, Y., Yang, J., & Huang, T. (2018) · arXiv · doi:10.48550/arxiv.1809.03327 · Cited by 301
Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals
Van Gansbeke, W., Vandenhende, S., Georgoulis, S., & Van Gool, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00990 · Cited by 231 · cited 1×
“Video K-Net [172] achieves the best results on the KITTI-STEP dataset.”
BackgroundLite DETR : An Interleaved Multi-Scale Encoder for Efficient DETR
Li, F., Zeng, A., Liu, S., Zhang, H., Li, H., Zhang, L., & Ni, L. M. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01780 · Cited by 154
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
Lu, J., Clark, C., Zellers, R., Mottaghi, R., & Kembhavi, A. (2022) · arXiv · doi:10.48550/arxiv.2206.08916 · Cited by 145
SOLQ: Segmenting Objects by Learning Queries
Dong, B., Zeng, F., Wang, T., Zhang, X., & Wei, Y. (2021) · arXiv · doi:10.48550/arxiv.2106.02351 · Cited by 83
DN-DETR: Accelerate DETR Training by Introducing Query DeNoising
Li, F., Zhang, H., Liu, S., Guo, J., Ni, L. M., & Zhang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.01305 · Cited by 57
The 2017 DAVIS Challenge on Video Object Segmentation
Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., & Van Gool, L. (2017) · arXiv · doi:10.48550/arxiv.1704.00675 · Cited by 13
Research works printed but not linked: 263
- 10.1023/A:1011174803800 · DOI · Paper page
- 10.1109/34.868688 · DOI · Paper page
- Stella, Multiclass spectral clustering, 313
- 10.5244/C.22.54 · DOI · Paper page
- 10.1007/BF00133570 · DOI · Paper page
- 10.1007/s11263-015-0816-y · DOI · Paper page
- Simonyan, Very deep convolutional networks for large-scale image recognition, 1
- 10.1109/CVPR.2015.7298965 · DOI · Paper page
- 10.1109/CVPR.2017.660 · DOI · Paper page
- 10.1109/CVPR.2018.00254 · DOI · Paper page
- 10.48550/ARXIV.1706.03762 · DOI · Paper page
- 10.1007/978-3-030-01240-3_17 · DOI · Paper page
- 10.1109/CVPR42600.2020.01009 · DOI · Paper page
- 10.1109/ICCV.2019.00356 · DOI · Paper page
- 10.1109/CVPR46437.2021.00542 · DOI · Paper page
- 10.1109/ICCV48922.2021.01595 · DOI · Paper page
- 10.1109/CVPR46437.2021.00738 · DOI · Paper page
- 10.1109/tpami.2023.3243465 · DOI · Paper page
- 10.1109/tpami.2023.3275156 · DOI · Paper page
- 10.1016/j.neucom.2019.11.118 · DOI · Paper page
- 10.1109/TPAMI.2022.3225573 · DOI · Paper page
- 10.1109/CVPR52688.2022.02036 · DOI
- 10.1007/978-3-319-10602-1_48 · DOI · Paper page
- 10.1109/CVPR52729.2023.02259 · DOI · Paper page
- 10.1109/CVPR46437.2021.00412 · DOI
- 10.1109/ICCV.2019.00529 · DOI · Paper page
- 10.1007/s11263-022-01629-1 · DOI
- 10.1109/CVPR42600.2020.00988 · DOI · Paper page
- Weber, STEP: Segmenting and tracking every pixel
- 10.1109/ICCV51070.2023.01850 · DOI
- 10.1109/ICCV51070.2023.00254 · DOI
- 10.1109/CVPR.2018.00199 · DOI · Paper page
- 10.1109/TIP.2019.2962685 · DOI
- 10.1109/CVPR.2019.00909 · DOI · Paper page
- 10.1109/TIP.2018.2878975 · DOI
- 10.1609/aaai.v34i07.6805 · DOI · Paper page
- 10.1109/TIP.2021.3099366 · DOI
- 10.48550/arXiv.1909.11065 · arXiv · doi:10.48550/arxiv.1909.11065 · Paper page
- Zhang, Dual graph convolutional network for semantic segmentation, 254
- 10.1007/978-3-030-58452-8_45 · DOI · Paper page
- 10.1109/CVPR42600.2020.00982 · DOI · Paper page
- 10.1109/ICCV.2019.00692 · DOI · Paper page
- 10.1007/978-3-030-58520-4_26 · DOI · Paper page
- 10.1109/ICCV48922.2021.01556 · DOI
- 10.1109/CVPR.2019.00326 · DOI · Paper page
- 10.1007/978-3-030-58452-8_17 · DOI · Paper page
- 10.1109/CVPR.2019.00904 · DOI · Paper page
- 10.1109/CVPR42600.2020.01024 · DOI
- 10.1109/ICCV.2019.00215 · DOI · Paper page
- Wang, SOLOv2: Dynamic and fast instance segmentation
- 10.1109/CVPR.2019.00902 · DOI · Paper page
- 10.1109/CVPR46437.2021.00028 · DOI · Paper page
- 10.1007/978-3-030-58548-8_7 · DOI · Paper page
- 10.1109/ICCV.2017.477 · DOI · Paper page
- 10.1007/978-3-319-49409-8_69 · DOI · Paper page
- 10.1109/CVPR.2017.441 · DOI · Paper page
- 10.1109/CVPR42600.2020.00976 · DOI · Paper page
- 10.1609/aaai.v35i2.16225 · DOI
- 10.1109/TPAMI.2022.3211612 · DOI
- 10.1109/CVPR46437.2021.00399 · DOI
- 10.1109/CVPR.2016.85 · DOI · Paper page
- Qi, PointNet: Deep learning on point sets for 3D classification and segmentation, 652
- Qi, PointNet: Deep hierarchical feature learning on point sets in a metric space, 5099
- Yang, Learning object bounding boxes for 3D instance segmentation on point clouds, 6737
- 10.1109/CVPR.2019.00407 · DOI · Paper page
- 10.1109/CVPR.2018.00272 · DOI · Paper page
- 10.1109/CVPR42600.2020.00492 · DOI · Paper page
- 10.1109/ICCV.2019.00166 · DOI · Paper page
- 10.1109/CVPR46437.2021.01236 · DOI · Paper page
- 10.1109/CVPR46437.2021.01299 · DOI
- 10.1609/aaai.v36i3.20197 · DOI
- 10.1109/CVPR46437.2021.01289 · DOI
- 10.1109/CVPR46437.2021.00548 · DOI
- 10.1109/CVPR46437.2021.00981 · DOI · Paper page
- Chen, A2-Nets: Double attention networks, 350
- Ren, Faster R-CNN: Towards real-time object detection with region proposal networks, 91
- 10.1109/TPAMI.2020.3032166 · DOI · Paper page
- 10.1109/CVPR.2019.00720 · DOI · Paper page
- Xie, SegFormer: Simple and efficient design for semantic segmentation with transformers, 12077
- 10.1109/CVPR52733.2024.02640 · DOI
- 10.1109/CVPR52729.2023.01797 · DOI
- 10.1007/978-3-031-19812-0_34 · DOI
- 10.1109/iccv51070.2023.00240 · DOI
- Touvron, Training data-efficient image transformers & distillation through attention, 10347
- 10.1109/CVPR52688.2022.00476 · DOI · Paper page
- 10.1109/CVPR52688.2022.00714 · DOI · Paper page
- Ali, XCiT: Cross-covariance image transformers, 20014
- 10.1109/ICCV48922.2021.00041 · DOI · Paper page
- 10.1007/978-3-030-58604-1_20 · DOI · Paper page
- 10.1109/ICCV48922.2021.00983 · DOI · Paper page
- 10.1109/ICCV48922.2021.00009 · DOI · Paper page
- Xu, ViTAE: Vision transformer advanced by exploring intrinsic inductive Bias, 28522
- 10.1109/CVPR52688.2022.01167 · DOI · Paper page
- Han, On the connection between local attention and dynamic depth-wise convolution, 1
- Guo, SegNeXt: Rethinking convolutional attention design for semantic segmentation, 1140
- 10.1109/CVPR52688.2022.01055 · DOI · Paper page
- 10.5555/3524938.3525087 · doi:10.5555/3524938.3525087
- Bao, BEiT: BERT pre-training of image transformers, 14668
- Gandelsman, Test-time training with masked autoencoders
- Gao, ConvMAE: Masked convolution meets masked autoencoders, 35632
- 10.1109/cvpr52729.2023.02240 · DOI · Paper page
- 10.1109/ICCV48922.2021.00683 · DOI · Paper page
- Zhang, K-Net: Towards unified image segmentation, 10326
- Cheng, Per-pixel classification is not all you need for semantic segmentation, 17864
- 10.1109/CVPR52688.2022.00134 · DOI · Paper page
- 10.1109/CVPR46437.2021.00863 · DOI · Paper page
- 10.1109/TPAMI.2022.3223955 · DOI · Paper page
- 10.1109/CVPR52688.2022.00290 · DOI
- Hwang, Video instance segmentation using inter-frame communication transformers, 13352
- 10.1007/978-3-031-19815-1_32 · DOI
- 10.1109/CVPR52688.2022.01828 · DOI
- 10.1109/CVPR52688.2022.01354 · DOI
- 10.1609/aaai.v36i3.20158 · DOI · Paper page
- Liu, DAB-DETR: Dynamic anchor boxes are better queries for DETR, 1
- 10.1109/CVPR52729.2023.00297 · DOI · Paper page
- Wang, Learning equivariant segmentation with instance-unique querying
- 10.1109/CVPR52729.2023.01887 · DOI · Paper page
- Huang, MinVIS: A minimal video instance segmentation framework without video-based training, 31265
- 10.1007/978-3-031-19815-1_34 · DOI
- 10.1007/978-3-031-19812-0_42 · DOI
- 10.1109/CVPR52688.2022.00168 · DOI
- 10.1007/978-3-031-19836-6_31 · DOI
- 10.1007/978-3-031-19812-0_30 · DOI
- 10.1109/ICCV48922.2021.01601 · DOI · Paper page
- 10.1109/CVPR52688.2022.01762 · DOI · Paper page
- 10.1109/CVPR52688.2022.01761 · DOI · Paper page
- 10.1109/CVPR52688.2022.01139 · DOI · Paper page
- 10.1109/CVPR52688.2022.00493 · DOI · Paper page
- 10.1109/CVPR52688.2022.00491 · DOI
- 10.1109/tip.2024.3371348 · DOI
- 10.1109/CVPR52688.2022.00492 · DOI · Paper page
- Zhang, Few-shot segmentation via cycle-consistent transformer, 21984
- Yang, Associating objects with transformers for video object segmentation, 2491
- 10.1109/CVPR52688.2022.01140 · DOI
- 10.1007/978-3-031-19815-1_36 · DOI
- 10.1109/tcsvt.2023.3252807 · DOI
- 10.1007/978-3-031-19818-2_35 · DOI
- 10.1109/CVPR52688.2022.01170 · DOI · Paper page
- Chen, CycleMLP: A MLP-like architecture for dense prediction, 1
- 10.1109/CVPR52688.2022.00090 · DOI
- 10.1007/s11263-023-01852-4 · DOI · Paper page
- Tian, Designing BERT for convolutional networks: Sparse and hierarchical masked modeling, 1
- 10.1109/cvpr52729.2023.00264 · DOI
- 10.1109/TPAMI.2022.3217852 · DOI · Paper page
- 10.1609/aaai.v36i3.20227 · DOI
- 10.1109/ICCV48922.2021.00717 · DOI · Paper page
- 10.1109/CVPR52688.2022.00259 · DOI
- 10.1007/978-3-031-19818-2_17 · DOI
- 10.1109/CVPR52688.2022.00439 · DOI · Paper page
- 10.1109/CVPR52688.2022.00102 · DOI
- 10.1109/ICCV48922.2021.00360 · DOI · Paper page
- 10.1109/CVPR52688.2022.00529 · DOI
- Zheng, End-to-end object detection with adaptive clustering transformer, 226
- Roh, Sparse DETR: Efficient end-to-end object detection with learnable sparsity, 1
- Heo, VITA: Video instance segmentation via object token association, 23109
- 10.1109/CVPR52729.2023.01451 · DOI · Paper page
- 10.1007/978-3-031-20059-5_17 · DOI
- 10.1109/wacv57701.2024.00030 · DOI
- Li, Tube-link: A flexible cross tube baseline for universal video segmentation, 13877
- 10.1109/CVPR52729.2023.01733 · DOI
- 10.1007/978-3-031-20077-9_6 · DOI
- 10.1109/CVPR52733.2024.01266 · DOI
- 10.1109/CVPR52729.2023.01868 · DOI
- 10.1109/CVPR52733.2024.01247 · DOI
- 10.1109/TMM.2022.3163578 · DOI
- 10.1109/CVPR42600.2020.01005 · DOI · Paper page
- 10.1109/CVPR52688.2022.00494 · DOI
- 10.1109/ICCV48922.2021.00180 · DOI · Paper page
- 10.1007/s40747-024-01539-4 · DOI
- 10.1109/TIP.2022.3155958 · DOI
- 10.1109/tpami.2024.3421340 · DOI
- 10.1109/CVPR52688.2022.00831 · DOI · Paper page
- 10.1109/CVPR52688.2022.01871 · DOI · Paper page
- 10.1109/icra48891.2023.10160590 · DOI · Paper page
- 10.1609/aaai.v37i2.25335 · DOI
- 10.1609/aaai.v37i2.25329 · DOI
- 10.1109/CVPR52688.2022.01631 · DOI · Paper page
- Zhang, Tip-Adapter: Training-free CLIP-adapter for better vision-language modeling, 493
- 10.1007/978-3-031-19833-5_23 · DOI · Paper page
- Chen, Vision transformer adapter for dense predictions, 1
- 10.1109/CVPR52688.2022.01755 · DOI · Paper page
- 10.1109/CVPR52688.2022.00695 · DOI · Paper page
- 10.1109/CVPR52729.2023.00292 · DOI · Paper page
- 10.1109/CVPR46437.2021.01416 · DOI · Paper page
- Gu, Open-vocabulary object detection via vision and language knowledge distillation, 1
- 10.1007/978-3-031-20077-9_7 · DOI · Paper page
- 10.1109/CVPR52729.2023.01081 · DOI
- Kuo, F-VLM: Open-vocabulary object detection upon frozen vision and language models, 1
- 10.1007/978-3-031-20080-9_30 · DOI
- 10.1007/978-3-031-20059-5_31 · DOI · Paper page
- 10.1109/ICCV51070.2023.02005 · DOI
- 10.1109/CVPR52729.2023.01863 · DOI
- 10.1109/ICCV48922.2021.01060 · DOI
- 10.1109/CVPR52729.2023.00289 · DOI · Paper page
- 10.1109/CVPR52688.2022.00902 · DOI · Paper page
- 10.1109/CVPR52729.2023.00288 · DOI · Paper page
- 10.1109/CVPR42600.2020.01242 · DOI
- 10.1109/CVPR42600.2020.00414 · DOI · Paper page
- 10.1109/CVPR52688.2022.00969 · DOI · Paper page
- 10.1007/978-3-031-20056-4_22 · DOI · Paper page
200 of 263 shown
Cited by
The Segment Anything Model (SAM) for remote sensing applications: From zero to one shot
Osco, L. P., Wu, Q., de Lemos, E. L., Gonçalves, W. N., Ramos, A. P. M., Li, J., & Marcato, J. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103540 · Cited by 338
Rethinking Mobile Block for Efficient Attention-based Models
Zhang, J., Li, X., Li, J., Liu, L., Xue, Z., Zhang, B., Jiang, Z., Huang, T., Wang, Y., & Wang, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00134 · Cited by 313
Self-citation
Metadata
- Authors
- Li, X., Ding, H., Yuan, H., Zhang, W., Pang, J., Cheng, G., Chen, K., Liu, Z., & Loy, C. C.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-12
- Volume
- 46
- Issue
- 12
- Pages
- 10138-10163
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4401070841
- Cited by
- 236
- References
- 92
- FWCI
- 30.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Funders
- Alan Turing Institute (SDCfP2\100009)
- Industry Alignment Fund Industry Collaboration Projects
- Funding Initiative and Singapore MOE AcRF Tier 1 (RG16/21)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.