The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Multi-class Token Transformer for Weakly Supervised Semantic Segmentation
Open accessAbstract
This paper proposes a new transformer-based framework to learn class-specific object localization maps as pseudo labels for weakly supervised semantic segmentation (WSSS). Inspired by the fact that the attended regions of the one-class token in the standard vision transformer can be leveraged to form a class-agnostic localization map, we investigate if the transformer model can also effectively capture class-specific attention for more discriminative object localization by learning multiple class tokens within the transformer. To this end, we propose a Multi-class Token Transformer, termed as MCTformer, which uses multiple class tokens to learn interactions between the class tokens and the patch tokens. The proposed MCTformer can successfully produce class-discriminative object localization maps from the class-to-patch attentions corresponding to different class tokens. We also propose to use a patch-level pairwise affinity, which is extracted from the patch-to-patch transformer attention, to further refine the localization maps. Moreover, the proposed framework is shown to fully complement the Class Activation Mapping (CAM) method, leading to remarkably superior WSSS results on the PASCAL VOC and MS COCO datasets. These results underline the importance of the class token for WSSS.11https://github.com/xulianuwa/MCTformer
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 3×
“[3] adapted self-supervised methods to ViT and observed that the attentions of the class token on patches contain information about the semantic layout of the scenes.”
BackgroundMethodInfluentialSemantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
Vision Transformers for Dense Prediction
Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01196 · Cited by 2,373 · cited 1×
“…to model long-range dependencies of long sequences in the field of NLP. Recently, transformer models have been adapted to accommodate a wide variety of vision tasks [18], such as image classification [10], saliency detection [27] and semantic segmentation [30], achieving promising performances.”
BackgroundSemantic contours from inverse detectors
Hariharan, B., Arbelaez, P., Bourdev, L., Maji, S., & Malik, J. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126343 · Cited by 1,624
Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
Wu, Z., Shen, C., & van den Hengel, A. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.01.006 · Cited by 1,589
Ahn, J., & Kwak, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00523 · Cited by 872
Wei, Y., Feng, J., Liang, X., Cheng, M.-M., Zhao, Y., & Yan, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.687 · Cited by 863
Twins: Revisiting the Design of Spatial Attention in Vision Transformers
Chu, X., Tian, Z., Wang, Y., Zhang, B., Ren, H., Wei, X., Xia, H., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2104.13840 · Cited by 817
Self-Supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation
Wang, Y., Zhang, J., Kan, M., Shan, S., & Chen, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01229 · Cited by 770
Singh, K. K., & Lee, Y. J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.381 · Cited by 724 · cited 1×
“A few methods developed heuristic strategies, such as “Hide & Seek” [31] and Erasing [40], applied either on images [24, 49] or feature maps [16,21] to drive the network to learn novel object patterns.”
MethodSeed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation
Kolesnikov, A., & Lampert, C. H. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_42 · Cited by 719
Wei, Y., Xiao, H., Shi, H., Jie, Z., Feng, J., & Huang, T. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00759 · Cited by 627
Adversarial Complementary Learning for Weakly Supervised Object Localization
Zhang, X., Wei, Y., Feng, J., Yang, Y., & Huang, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00144 · Cited by 592
Liu, N., Zhang, N., Wan, K., Shao, L., & Han, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00468 · Cited by 549
Tell Me Where to Look: Guided Attention Inference Network
Li, K., Wu, Z., Peng, K.-C., Ernst, J., & Fu, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00960 · Cited by 535
Conditional Positional Encodings for Vision Transformers
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489 · cited 1×
“Although the class token has been removed in a number of recent transformer methods [7, 8, 29], this work will underline its importance for weakly supervised semantic segmentation.”
MethodCausal Intervention for Weakly-Supervised Semantic Segmentation
Zhang, D., Zhang, H., Tang, J., Hua, X., & Sun, Q. (2020) · arXiv · doi:10.48550/arxiv.2009.12547 · Cited by 353
Weakly-Supervised Semantic Segmentation via Sub-Category Exploration
Chang, Y.-T., Wang, Q., Hung, W.-C., Piramuthu, R., Tsai, Y.-H., & Yang, M.-H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00901 · Cited by 318
Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation
Sun, G., Wang, W., Dai, J., & Van Gool, L. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58536-5_21 · Cited by 288
Single-Stage Semantic Segmentation From Image Labels
Araslanov, N., & Roth, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00431 · Cited by 275
Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation
Lee, J., Kim, E., & Yoon, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00406 · Cited by 268
Lee, S., Lee, M., Lee, J., & Shim, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00545 · Cited by 263
On Regularized Losses for Weakly-supervised CNN Segmentation
Tang, M., Perazzi, F., Djelouah, A., Ayed, I. B., Schroers, C., & Boykov, Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_31 · Cited by 247
TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization
Gao, W., Wan, F., Pan, X., Peng, Z., Tian, Q., Han, Z., Zhou, B., & Ye, Q. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00288 · Cited by 219
Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation
Yao, Y., Chen, T., Xie, G.-S., Zhang, C., Shen, F., Wu, Q., Tang, Z., & Zhang, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00265 · Cited by 211
CIAN: Cross-Image Affinity Net for Weakly Supervised Semantic Segmentation
Fan, J., Zhang, Z., Tan, T., Song, C., & Xiao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6705 · Cited by 207
Learning Integral Objects With Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation
Fan, J., Zhang, Z., Song, C., & Tan, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00434 · Cited by 207
Kweon, H., Yoon, S.-H., Kim, H., Park, D.-H., & Yoon, K. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00691 · Cited by 207
Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach
Zhang, B., Xiao, J., Wei, Y., Sun, M., & Huang, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6971 · Cited by 204
Complementary Patch for Weakly Supervised Semantic Segmentation
Zhang, F., Gu, C., Zhang, C., & Dai, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00715 · Cited by 204
Embedded Discriminative Attention Mechanism for Weakly Supervised Semantic Segmentation
Wu, T., Huang, J., Gao, G., Wei, X., Wei, X. K., Luo, X., & Liu, C. H. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01649 · Cited by 201
Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation
Su, Y., Sun, R., Lin, G., & Wu, Q. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00692 · Cited by 177
Weakly Supervised Semantic Segmentation with Boundary Exploration
Chen, L., Wu, W., Fu, C., Han, X., & Zhang, Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58574-7_21 · Cited by 175
Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentation
Xu, L., Ouyang, W., Bennamoun, M., Boussaid, F., Sohel, F., & Xu, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00690 · Cited by 175
Self-citationScalable Vision Transformers with Hierarchical Pooling
Pan, Z., Zhuang, B., Liu, J., He, H., & Cai, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00043 · Cited by 175
Sun, K., Shi, H., Zhang, Z. J., & Huang, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00719 · Cited by 158
Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation
Li, X., Zhou, T., Li, J., Zhou, Y., & Zhang, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16294 · Cited by 142
Weakly-Supervised Semantic Segmentation by Iterative Affinity Learning
Wang, X., Liu, S., Ma, H., & Yang, M.-H. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01293-3 · Cited by 105
Zhang, T., Lin, G., Liu, W., Cai, J., & Kot, A. C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58542-6_40 · Cited by 63
Learning Saliency-Free Model with Generic Features for Weakly-Supervised Semantic Segmentation
Luo, W., & Yang, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6842 · Cited by 17
Cited by
Efficient Frequency Domain-based Transformers for High-Quality Image Deblurring
Kong, L., Dong, J., Ge, J., Li, M., & Pan, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00570 · Cited by 398
Metadata
- Authors
- Xu, L., Ouyang, W., Bennamoun, M., Boussaid, F., & Xu, D.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 4300-4309
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4312509967
- Cited by
- 305
- References
- 48
- FWCI
- 26.27
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.