The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Decoupling Zero-Shot Semantic Segmentation
Open accessAbstract
Zero-shot semantic segmentation (ZS3) aims to segment the novel categories that have not been seen in the training. Existing works formulate ZS3 as a pixel-level zeroshot classification problem, and transfer semantic knowledge from seen classes to unseen ones with the help of language models pre-trained only with texts. While simple, the pixel-level ZS3 formulation shows the limited capability to integrate vision-language models that are often pre-trained with image-text pairs and currently demonstrate great potential for vision tasks. Inspired by the observation that humans often perform segment-level semantic labeling, we propose to decouple the ZS3 into two sub-tasks: 1) a classagnostic grouping task to group the pixels into segments. 2) a zero-shot classification task on segments. The former task does not involve category information and can be directly transferred to group pixels for unseen classes. The latter task performs at segment-level and provides a natural way to leverage large-scale vision-language models pre-trained with image-text pairs (e.g. CLIP) for ZS3. Based on the decoupling formulation, we propose a simple and effective zero-shot semantic segmentation model, called ZegFormer, which outperforms the previous methods on ZS3 standard benchmarks by large margins, e.g., 22 points on the PAS-CAL VOC and 3 points on the COCO-Stuff in terms of mIoU for unseen classes. Code will be released at https://github.com/dingjiansw101/ZegFormer.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
Normalized cuts and image segmentation
Shi, J., & Malik, J. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.868688 · Cited by 15,502
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1×
“The key idea in zero-shot learning is to transfer kowledge from seen classes to unseen classes via semantic representation, such as the semantic attributes [2,24,28,30], concept ontology [16,37,39,47,48] and semantic word vectors [17, 41, 60].”
BackgroundV-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
Milletari, F., Navab, N., & Ahmadi, S.-A. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.79 · Cited by 11,931
Recognition-by-components: A theory of human image understanding.
Biederman, I. (1987) · Psychological Review · doi:10.1037/0033-295x.94.2.115 · Cited by 5,621
Contour Detection and Hierarchical Image Segmentation
Arbeláez, P., Maire, M., Fowlkes, C., & Malik, J. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.161 · Cited by 5,454
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090
Scene Parsing through ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.544 · Cited by 3,326
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,308
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977
Region competition: unifying snakes, region growing, and Bayes/MDL for multiband image segmentation
Zhu, S. C., & Yuille, A. (1996) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.537343 · Cited by 1,913
Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly
Xian, Y., Lampert, C. H., Schiele, B., & Akata, Z. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2857768 · Cited by 1,678
Attribute-Based Classification for Zero-Shot Visual Object Categorization
Lampert, C. H., Nickisch, H., & Harmeling, S. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.140 · Cited by 1,627
Object-Contextual Representations for Semantic Segmentation
Yuan, Y., Chen, X., & Wang, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_11 · Cited by 1,285
COCO-Stuff: Thing and Stuff Classes in Context
Caesar, H., Uijlings, J., & Ferrari, V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00132 · Cited by 1,277
Multiscale Combinatorial Grouping
Arbelaez, P., Pont-Tuset, J., Barron, J., Marques, F., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.49 · Cited by 1,155
FastText.zip: Compressing text classification models
Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., & Mikolov, T. (2016) · arXiv · doi:10.48550/arxiv.1612.03651 · Cited by 887
Zero-Shot Learning by Convex Combination of Semantic Embeddings
Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., Corrado, G. S., & Dean, J. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.5650 · Cited by 710
Learning to Segment Object Candidates
Pinheiro, P. O., Collobert, R., & Dollár, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06204 · Cited by 648
Evaluation of Output Embeddings for Fine-grained Image Classification
Akata, Z., Reed, S., Walter, D., Lee, H., & Schiele, B. (2015) · Max Planck Digital Library · Cited by 567
Image segmentation by data-driven markov chain monte carlo
Tu, Z., & Zhu, S.-C. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.1000239 · Cited by 565
Open-Vocabulary Object Detection Using Captions
Zareian, A., Rosa, K. D., Hu, D. H., & Chang, S.-F. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01416 · Cited by 425
Zero-Shot Learning via Joint Latent Similarity Embedding
Zhang, Z., & Saligrama, V. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.649 · Cited by 336
Evaluating knowledge transfer and zero-shot learning in a large-scale setting
Rohrbach, M., Stark, M., & Schiele, B. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995627 · Cited by 303
Learning to Segment Every Thing
Hu, R., Dollar, P., He, K., Darrell, T., & Girshick, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00445 · Cited by 281
What helps where – and why? Semantic relatedness for knowledge transfer
Rohrbach, M., Stark, M., Szarvas, G., Gurevych, I., & Schiele, B. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540121 · Cited by 273
Semantic Projection Network for Zero- and Few-Label Semantic Segmentation
Xian, Y., Choudhury, S., He, Y., Schiele, B., & Akata, Z. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00845 · Cited by 240
K-Net: Towards Unified Image Segmentation
Zhang, W., Pang, J., Chen, K., & Loy, C. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14855 · Cited by 196
COSTA: Co-Occurrence Statistics for Zero-Shot Classification
Mensink, T., Gavves, E., & Snoek, C. G. M. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.313 · Cited by 193
Zero Shot Recognition with Unreliable Attributes
Jayaraman, D., & Grauman, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.4327 · Cited by 166
Efficient Training of Visual Transformers with Small DatasetsPreprint
Liu, Y., Sangineto, E., Bi, W., Sebe, N., Lepri, B., & Nadai, M. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03746 · Cited by 145
Supervised Evaluation of Image Segmentation and Object Proposal Techniques
Pont-Tuset, J., & Marqués, F. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2481406 · Cited by 141
Context-aware Feature Generation For Zero-shot Semantic Segmentation
Gu, Z., Zhou, S., Niu, L., Zhao, Z., & Zhang, L. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413593 · Cited by 128
Semantic Label Sharing for Learning with Many Categories
Fergus, R., Bernal, H. J., Weiss, Y., & Torralba, A. B. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15549-9_55 · Cited by 117
Zhao, H., Puig, X., Zhou, B., Fidler, S., & Torralba, A. B. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.221 · Cited by 112
Online incremental attribute-based zero-shot learning
Kankuekul, P., Kawewong, A., Tangruamsub, S., & Hasegawa, O. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248112 · Cited by 100
Al-Halah, Z., Tapaswi, M., & Stiefelhagen, R. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.643 · Cited by 98
Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation
Baek, D., Oh, Y., & Ham, B. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00940 · Cited by 75
Open World Entity Segmentation
Qi, L., Kuen, J., Wang, Y., Gu, J., Zhao, H., Torr, P. H. S., Lin, Z., & Jia, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3227513 · Cited by 74
Zero-Shot Instance Segmentation
Zheng, Y., Wu, J., Qin, Y., Zhang, F., & Cui, L. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00262 · Cited by 64
A Closer Look at Self-training for Zero-Label Semantic Segmentation
Pastore, G., Cermelli, F., Xian, Y., Mancini, M., Akata, Z., & Barbara Caputo, B. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00303 · Cited by 62
Consistent Structural Relation Learning for Zero-Shot Segmentation.
Li, P. P., Wei, Y., & Yang, Y. (2020) · UTS ePRESS (University of Technology Sydney) · Cited by 51
Cheng, J., Nandi, S., Natarajan, P., & AbdAlmageed, W. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00942 · Cited by 50
Recent Advances in Zero-shot RecognitionPreprint
Fu, Y., Xiang, T., Jiang, Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.04837 · Cited by 32
End-to-End Joint Semantic Segmentation of Actors and Actions in Video
Ji, J., Buch, S., Soto, Á., & Niebles, J. C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01225-0_43 · Cited by 32
Zero-Shot Detection via Vision and Language Knowledge Distillation.Preprint
Gu, X., Lin, T.-Y., Kuo, W., & Cui, Y. (2021) · arXiv (Cornell University) · Cited by 29
From Pixel to Patch: Synthesize Context-Aware Features for Zero-Shot Semantic Segmentation
Gu, Z., Zhou, S., Niu, L., Zhao, Z., & Zhang, L. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3145962 · Cited by 21
Cap2Seg: Inferring Semantic and Spatial Context from Captions for Zero-Shot Image Segmentation
Tian, G., Wang, S., Feng, J., Zhou, L., & Mu, Y. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413990 · Cited by 13
Conterfactual Generative Zero-Shot Semantic SegmentationPreprint
Shen, F., Liu, J., & Hu, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06360 · Cited by 12
Cited by
Vision-Language Models for Vision Tasks: A Survey
Zhang, J., Huang, J., Jin, S., & Lu, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3369699 · Cited by 1,051
MaPLe: Multi-modal Prompt Learning
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., & Khan, F. S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01832 · Cited by 925
Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., & Marculescu, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00682 · Cited by 464
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., & De Mello, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00289 · Cited by 359
Side Adapter Network for Open-Vocabulary Semantic Segmentation
Xu, M., Zhang, Z., Wei, F., Hu, H., & Bai, X. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00288 · Cited by 316
Metadata
- Authors
- Ding, J., Xue, N., Xia, G.-S., & Dai, D.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 11573-11582
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4312935996
- Cited by
- 296
- References
- 61
- FWCI
- 35.95
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.