Token Contrast for Weakly-Supervised Semantic Segmentation
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Weakly-Supervised Semantic Segmentation (WSSS) using image-level labels typically utilizes Class Activation Map (CAM) to generate the pseudo labels. Limited by the local structure perception of CNN, CAM usually cannot identify the integral object regions. Though the recent Vision Transformer (ViT) can remedy this flaw, we observe it also brings the over-smoothing issue, i.e., the final patch tokens incline to be uniform. In this work, we propose Token Contrast (ToCo) to address this issue and further explore the virtue of ViT for WSSS. Firstly, motivated by the observation that intermediate layers in ViT can still retain semantic diversity, we designed a Patch Token Contrast module (PTC). PTC supervises the final patch tokens with the pseudo token relations derived from intermediate layers, allowing them to align the semantic regions and thus yield more accurate CAM. Secondly, to further differentiate the low-confidence regions in CAM, we devised a Class Token Contrast module (CTC) inspired by the fact that class tokens in ViT can capture high-level semantics. CTC facilitates the representation consistency between uncertain local regions and global objects by contrasting their class tokens. Experiments on the PASCAL VOC and MS COCO datasets show the proposed ToCo can remarkably surpass other single-stage competitors and achieve comparable performance with state-of-the-art multi-stage methods. Code is available at https://github.com/rulixiang/ToCo.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · cited 1×
“In the inference stage, following the common practice in semantic segmentation [8], we use multi-scale”
MethodThe pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 2×
“CAM [53] is proposed to identify the activated regions when a classification network predicts an image.”
BackgroundRepresentation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 5×
“Inspired by the property that class tokens in ViT can aggregate the highlevel semantics [6, 15], we design a Class Token Contrast (CTC) module to facilitate the representation consistency between the local non-salient regions and the global object, which can further enforce more object regions to be activated in CAM.”
BackgroundMethodInfluentialSemantic contours from inverse detectors
Hariharan, B., Arbelaez, P., Bourdev, L., Maji, S., & Malik, J. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126343 · Cited by 1,624 · cited 1×
“Following common practice, VOC 2012 dataset is further augmented with the SBD dataset [17].”
MethodWhat’s the Point: Semantic Segmentation with Point Supervision
Bearman, A., Russakovsky, O., Ferrari, V., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_34 · Cited by 887 · cited 1×
“To reduce the expensive annotation costs of deep semantic segmentation models, weakly-supervised semantic segmentation (WSSS) is proposed to predict pixel-level predictions with only weak and cheap annotations, such as imagelevel labels [2], points [4], scribbles [51] and bounding”
MethodAhn, J., & Kwak, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00523 · Cited by 872
Wei, Y., Feng, J., Liang, X., Cheng, M.-M., Zhao, Y., & Yan, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.687 · Cited by 863
Self-Supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation
Wang, Y., Zhang, J., Kan, M., Shan, S., & Chen, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01229 · Cited by 770
Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations
Ahn, J., Cho, S., & Kwak, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00231 · Cited by 613 · cited 2×
“However, CAM is usually flawed since it typically only identifies the most discriminative semantic regions, severely weakening the final performance of semantic segmentation [1, 19, 43].”
BackgroundMethodWeakly-Supervised Semantic Segmentation via Sub-Category Exploration
Chang, Y.-T., Wang, Q., Hung, W.-C., Piramuthu, R., Tsai, Y.-H., & Yang, M.-H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00901 · Cited by 318
Multi-class Token Transformer for Weakly Supervised Semantic Segmentation
Xu, L., Ouyang, W., Bennamoun, M., Boussaid, F., & Xu, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00427 · Cited by 305 · cited 5×
“1 – MCTformer [47] CVPR’2022 I WR38 71.”
BackgroundInfluentialMining Cross-Image Semantics for Weakly Supervised Semantic Segmentation
Sun, G., Wang, W., Dai, J., & Van Gool, L. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58536-5_21 · Cited by 288
Ru, L., Zhan, Y., Yu, B., & Du, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01634 · Cited by 281
Self-citationSingle-Stage Semantic Segmentation From Image Labels
Araslanov, N., & Roth, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00431 · Cited by 275
Lee, S., Lee, M., Lee, J., & Shim, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00545 · Cited by 263 · cited 1×
“[24, 49] utilize extra saliency maps as supervision to suppress the background regions and mine the nonsalient objects.”
MethodIntegral Object Mining via Online Attention Accumulation
Jiang, P.-T., Hou, Q., Cao, Y., Cheng, M.-M., Wei, Y., & Xiong, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00216 · Cited by 254 · cited 1×
“To address this drawback, recent works proposed various training schemes, such as erasing [44], online attention accumulation [18] and cross-image semantic mining [38].”
On Regularized Losses for Weakly-supervised CNN Segmentation
Tang, M., Perazzi, F., Djelouah, A., Ayed, I. B., Schroers, C., & Boykov, Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_31 · Cited by 247
Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation
Chen, Z., Wang, T., Wu, X., Hua, X.-S., Zhang, H., & Sun, Q. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00104 · Cited by 235
BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentation
Lee, J., Yi, J., Shin, C., & Yoon, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00267 · Cited by 220 · cited 1×
“To reduce the expensive annotation costs of deep semantic segmentation models, weakly-supervised semantic segmentation (WSSS) is proposed to predict pixel-level predictions with only weak and cheap annotations, such as imagelevel labels [2], points [4], scribbles [51] and bounding boxes [23].”
TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization
Gao, W., Wan, F., Pan, X., Peng, Z., Tian, Q., Han, Z., Zhou, B., & Ye, Q. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00288 · Cited by 219
Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation
Yao, Y., Chen, T., Xie, G.-S., Zhang, C., Shen, F., Wu, Q., Tang, Z., & Zhang, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00265 · Cited by 211 · cited 1×
“[24, 49] utilize extra saliency maps as supervision to suppress the background regions and mine the non-salient objects.”
MethodRegional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentation
Zhou, T., Zhang, M., Zhao, F., & Li, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00426 · Cited by 206 · cited 1×
“[9,13,54] contrast the pixel and prototype representations to encourage the integral activation of the object regions.”
BackgroundReliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach
Zhang, B., Xiao, J., Wei, Y., Sun, M., & Huang, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6971 · Cited by 204
Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentation
Chen, Q., Yang, L., Lai, J., & Xie, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00425 · Cited by 200 · cited 1×
“[9,13,54] contrast the pixel and prototype representations to encourage the integral activation of the object regions.”
BackgroundWeakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast
Du, Y., Fu, Z., Liu, Q., & Wang, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00428 · Cited by 195 · cited 1×
“[9,13,54] contrast the pixel and prototype representations to encourage the integral activation of the object regions.”
BackgroundJiang, P.-T., Yang, Y., Hou, Q., & Wei, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01638 · Cited by 192 · cited 1×
“However, CAM is usually flawed since it typically only identifies the most discriminative semantic regions, severely weakening the final performance of semantic segmentation [1, 19, 43].”
BackgroundCLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation
Xie, J., Hou, X., Ye, K., & Shen, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00444 · Cited by 191
Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation
Su, Y., Sun, R., Lin, G., & Wu, Q. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00692 · Cited by 179 · cited 1×
“[22, 37, 46] counter the problem of semantic co-occurrence via distilling knowledge from CLIP [30], decoupling object context [37] and comparing the out-ofdistribution images [22], respectively.”
Weakly Supervised Semantic Segmentation using Out-of-Distribution Data
Lee, J., Oh, S. J., Yun, S., Choe, J., Kim, E., & Yoon, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01639 · Cited by 162 · cited 1×
“[22, 37, 46] counter the problem of semantic co-occurrence via distilling knowledge from CLIP [30], decoupling object context [37] and comparing the out-ofdistribution images [22], respectively.”
DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction
Xu, Y., Yang, Y., & Zhang, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25411 · Cited by 106
Rossetti, S., Zappia, D., Sanzari, M., Schaerf, M., & Pirri, F. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20056-4_26 · Cited by 106
Pan, J., Zhu, P., Zhang, K., Cao, B., Wang, Y., Zhang, D., Han, J., & Hu, Q. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01590-z · Cited by 104 · cited 4×
“Compared to ToCo, the previous methods, AFA [34] and SLRNet [28] only achieve 83.”
MethodInfluentialWeakly-Supervised Semantic Segmentation with Visual Words Learning and Hybrid Pooling
Ru, L., Du, B., Zhan, Y., & Wu, C. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01586-9 · Cited by 71 · cited 5×
“It’s noted that the segmentation loss also improves the CAM’s quality, since we use PAR [33] to refine the pseudo labels which enforce the pseudo labels to align better with the lowlevel object boundaries.”
BackgroundMethodInfluentialSelf-citationChen, T., Zhang, Z., Cheng, Y., Awadallah, A. H., & Wang, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01171 · Cited by 70 · cited 2×
“However, minimizing the original cosine similarity cannot ensure the diversity [10, 16], since a token pair with minus cosine similarity ( e.g ., − 1 ) could be highly correlated.”
BackgroundMethodZhang, X., Peng, Z., Zhu, P., Zhang, T., Li, C., Zhou, H., & Jiao, L. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475675 · Cited by 68
Vision Transformers with Patch Diversification
Gong, C., Wang, D., Li, M., Chandra, V., & Liu, Q. (2021) · arXiv · doi:10.48550/arxiv.2104.12753 · Cited by 41
Learning to Learn Better for Video Object Segmentation
Lan, M., Zhang, J., Zhang, L., & Tao, D. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25203 · Cited by 27
Dynamic feature regularized loss for weakly supervised semantic segmentation
Zhang, B., Xiao, J., & Zhao, Y. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111540 · Cited by 24
Metadata
- Authors
- Ru, L., Zheng, H., Zhan, Y., & Du, B.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 3093-3102
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386066092
- Cited by
- 213
- References
- 46
- FWCI
- 24.22
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.