The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers
Open accessAbstract
Weakly-supervised semantic segmentation (WSSS) with image-level labels is an important and challenging task. Due to the high training efficiency, end-to-end solutions for WSSS have received increasing attention from the community. However, current methods are mainly based on convolutional neural networks and fail to explore the global information properly, thus usually resulting in incomplete object regions. In this paper, to address the aforementioned problem, we introduce Transformers, which naturally integrate global information, to generate more integral initial pseudo labels for end-to-end WSSS. Motivated by the inherent consistency between the self-attention in Transformers and the semantic affinity, we propose an Affinity from Attention (AFA) module to learn semantic affinity from the multi-head self-attention (MHSA) in Transformers. The learned affinity is then leveraged to refine the initial pseudo labels for segmentation. In addition, to efficiently derive reliable affinity labels for supervising AFA and ensure the local consistency of pseudo labels, we devise a Pixel-Adaptive Refinement module that incorporates low-level image appearance information to refine the pseudo labels. We perform extensive experiments and our method achieves 66.0% and 38.9% mIoU on the PASCAL VOC 2012 and MS COCO 2014 datasets, respectively, significantly outperforming recent end-to-end methods and several multi-stage competitors. Code is available at https://github.com/rulixiang/afa.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1710.10903 · Cited by 12,107
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428
Veličković, P., Casanova, A., Liò, P., Cucurull, G., Romero, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.17863/cam.48429 · Cited by 8,657
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 8×
“We use the MiT-B1 proposed in SegFormer [49] as the backbone, which is a more friendly backbone for image segmentation tasks than the vanilla ViT [11].”
BackgroundMethodInfluentialEfficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Krähenbühl, P., & Koltun, V. (2012) · arXiv · doi:10.48550/arxiv.1210.5644 · Cited by 2,908
ViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767
Vision Transformers for Dense Prediction
Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01196 · Cited by 2,373 · cited 1×
“Later variants show ViT also benefits downstream vision tasks, such as semantic segmentation [49, 9, 58], depth estimation [33], and video understanding [5].”
BackgroundSemantic contours from inverse detectors
Hariharan, B., Arbelaez, P., Bourdev, L., Maji, S., & Malik, J. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126343 · Cited by 1,624
Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
Wu, Z., Shen, C., & van den Hengel, A. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.01.006 · Cited by 1,589
ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation
Lin, D., Dai, J., Jia, J., He, K., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.344 · Cited by 1,129
Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation
Papandreou, G., Chen, L.-C., Murphy, K. P., & Yuille, A. L. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.203 · Cited by 1,005
Ahn, J., & Kwak, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00523 · Cited by 872
Wei, Y., Feng, J., Liang, X., Cheng, M.-M., Zhao, Y., & Yan, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.687 · Cited by 863
Self-Supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation
Wang, Y., Zhang, J., Kan, M., Shan, S., & Chen, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01229 · Cited by 770
Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation
Kolesnikov, A., & Lampert, C. H. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_42 · Cited by 719
From image-level to pixel-level labeling with Convolutional Networks
Pinheiro, P. O., & Collobert, R. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298780 · Cited by 673
Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations
Ahn, J., Cho, S., & Kwak, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00231 · Cited by 613
Causal Intervention for Weakly-Supervised Semantic Segmentation
Zhang, D., Zhang, H., Tang, J., Hua, X., & Sun, Q. (2020) · arXiv · doi:10.48550/arxiv.2009.12547 · Cited by 353
Pixel-Adaptive Convolutional Neural Networks
Su, H., Jampani, V., Sun, D., Gallo, O., Learned-Miller, E., & Kautz, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01142 · Cited by 348
Weakly-Supervised Semantic Segmentation via Sub-Category Exploration
Chang, Y.-T., Wang, Q., Hung, W.-C., Piramuthu, R., Tsai, Y.-H., & Yang, M.-H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00901 · Cited by 318
Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation
Sun, G., Wang, W., Dai, J., & Van Gool, L. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58536-5_21 · Cited by 288
Single-Stage Semantic Segmentation From Image Labels
Araslanov, N., & Roth, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00431 · Cited by 275
Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation
Lee, J., Kim, E., & Yoon, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00406 · Cited by 268
Lee, S., Lee, M., Lee, J., & Shim, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00545 · Cited by 263
Integral Object Mining via Online Attention Accumulation
Jiang, P.-T., Hou, Q., Cao, Y., Cheng, M.-M., Wei, Y., & Xiong, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00216 · Cited by 254
On Regularized Losses for Weakly-supervised CNN Segmentation
Tang, M., Perazzi, F., Djelouah, A., Ayed, I. B., Schroers, C., & Boykov, Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_31 · Cited by 247
BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentation
Lee, J., Yi, J., Shin, C., & Yoon, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00267 · Cited by 220
Learning Random-Walk Label Propagation for Weakly-Supervised Semantic Segmentation
Vernaza, P., & Chandraker, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.315 · Cited by 219
TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization
Gao, W., Wan, F., Pan, X., Peng, Z., Tian, Q., Han, Z., Zhou, B., & Ye, Q. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00288 · Cited by 219
Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation
Yao, Y., Chen, T., Xie, G.-S., Zhang, C., Shen, F., Wu, Q., Tang, Z., & Zhang, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00265 · Cited by 211
Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach
Zhang, B., Xiao, J., Wei, Y., Sun, M., & Huang, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6971 · Cited by 204
Adaptive Affinity Fields for Semantic Segmentation
Ke, T.-W., Hwang, J.-J., Liu, Z., & Yu, S. X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_36 · Cited by 197
Affinity Attention Graph Neural Network for Weakly Supervised Semantic Segmentation
Zhang, B., Xiao, J., Jiao, J., Wei, Y., & Zhao, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3083269 · Cited by 180
Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentation
Xu, L., Ouyang, W., Bennamoun, M., Boussaid, F., Sohel, F., & Xu, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00690 · Cited by 175
Embedded Discriminative Attention Mechanism for Weakly Supervised Semantic Segmentation
Wu, T., Huang, J., Gao, G., Wei, X., Wei, X. K., Luo, X., & Liu, C. H. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01649 · Cited by 169
Per-Pixel Classification is Not All You Need for Semantic Segmentation
Cheng, B., Schwing, A. G., & Kirillov, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06278 · Cited by 164
Complementary Patch for Weakly Supervised Semantic Segmentation
Zhang, F., Gu, C., Zhang, C., & Dai, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00715 · Cited by 161
Kweon, H., Yoon, S.-H., Kim, H., Park, D.-H., & Yoon, K. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00691 · Cited by 157
Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation
Su, Y., Sun, R., Lin, G., & Wu, Q. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00692 · Cited by 150
Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation
Li, X., Zhou, T., Li, J., Zhou, Y., & Zhang, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16294 · Cited by 142
Sun, K., Shi, H., Zhang, Z. J., & Huang, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00719 · Cited by 135
Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation
Kim, B., Han, S., & Kim, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16269 · Cited by 123
Combining Bottom-Up, Top-Down, and Smoothness Cues for Weakly Supervised Image Segmentation
Roy, A., & Todorović, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.770 · Cited by 111
Weakly-Supervised Semantic Segmentation with Visual Words Learning and Hybrid Pooling
Ru, L., Du, B., Zhan, Y., & Wu, C. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01586-9 · Cited by 66
Self-citationZhang, X., Peng, Z., Zhu, P., Zhang, T., Li, C., Zhou, H., & Jiao, L. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475675 · Cited by 60
Dynamic feature regularized loss for weakly supervised semantic segmentation
Zhang, B., Xiao, J., & Zhao, Y. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111540 · Cited by 24
Learning Visual Words for Weakly-Supervised Semantic Segmentation
Ru, L., Du, B., & Wu, C. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/136 · Cited by 23
Self-citationTowards Single Stage Weakly Supervised Semantic SegmentationPreprint
Akiva, P., & Dana, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10309 · Cited by 13
Cited by
Centralized Feature Pyramid for Object Detection
Quan, Y., Zhang, D., Zhang, L., & Tang, J. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3297408 · Cited by 342
MRFTrans: Multimodal representation fusion transformer for monocular 3D semantic scene completion
Xu, R., Zhang, J., Sun, J., Wang, C., Wu, Y., Xu, S., Meng, W., & Zhang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102493 · Cited by 24
Metadata
- Authors
- Ru, L., Zhan, Y., Yu, B., & Du, B.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 16825-16834
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4313153210
- Cited by
- 281
- References
- 58
- FWCI
- 23.47
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.