PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering
Open accessAbstract
We present a new framework for semantic segmentation without annotations via clustering. Off-the-shelf clustering methods are limited to curated, single-label, and objectcentric images yet real-world data are dominantly uncurated, multi-label, and scene-centric. We extend clustering from images to pixels and assign separate cluster membership to different instances within each image. However, solely relying on pixel-wise feature similarity fails to learn high-level semantic concepts and overfits to lowlevel visual cues. We propose a method to incorporate geometric consistency as an inductive bias to learn invariance and equivariance for photometric and geometric variations. With our novel learning objective, our framework can learn high-level semantic concepts. Our method, PiCIE (Pixel-level feature Clustering using Invariance and Equivariance), is the first method capable of segmenting both things and stuff categories without any hyperparameter tuning or task-specific pre-processing. Our method largely outperforms existing baselines on COCO [31] and Cityscapes [8] with +17.5 Acc. and +4.5 mIoU. We show that PiCIE gives a better initialization for standard supervised training. The code is available at https://github.com/janghyuncho/PiCIE.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some USS methods utilize supervised prior knowledge, making it difficult to evaluate these methods.
“2) Due to the lack of clear problem definition and standardized evaluation, some methods utilize supervised prior knowledge, e.g., supervised pre-trained network weights [46], supervised edge detection [30], and supervised saliency detection [31], [47], [48], making it difficult to evaluate these methods.”
From Large-Scale Unsupervised Semantic Segmentation (2023) · cites this paper as [46]
The citation to PiCIE is attached specifically to the example of supervised edge detection. In S3. Analysis (p. 14), the paper says it lacks supervision for precise boundaries and discusses using an edge detector only as possible future work, outside the project’s scope. It does not report using supervised edge detection. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · see citing works
Self-citationThe pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · see citing works
Momentum Contrast for Unsupervised Visual Representation LearningPreprint
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. B. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.05722 · Cited by 16,250 · see citing works
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works
The Hungarian method for the assignment problem
Kuhn, H. W. (1955) · Naval Research Logistics Quarterly · doi:10.1002/nav.3800020109 · Cited by 13,120 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
Fully-Convolutional Siamese Networks for Object Tracking
Bertinetto, L., Valmadre, J., Henriques, J. F., Vedaldi, A., & Torr, P. H. S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-48881-3_56 · Cited by 4,546 · see citing works
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228 · see citing works
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Krähenbühl, P., & Koltun, V. (2012) · arXiv · doi:10.48550/arxiv.1210.5644 · Cited by 2,908 · see citing works
Deep Clustering for Unsupervised Learning of Visual Features
Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_9 · Cited by 2,674 · see citing works
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · see citing works
Conditional Image Generation with PixelCNN DecodersPreprint
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05328 · Cited by 1,928 · see citing works
Conditional Image Generation with PixelCNN DecodersPreprint
van den Oord, A., Kalchbrenner, N., Vinyals, O., et al. (2016) · arXiv (Cornell University) · Cited by 1,586 · see citing works
Big Self-Supervised Models are Strong Semi-Supervised LearnersPreprint
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., & Hinton, G. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10029 · Cited by 1,480 · see citing works
Panoptic Feature Pyramid Networks
Kirillov, A., Girshick, R., He, K., & Dollár, P. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00656 · Cited by 1,460 · see citing works
LVIS: A Dataset for Large Vocabulary Instance Segmentation
Gupta, A., Dollar, P., & Girshick, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00550 · Cited by 1,274 · see citing works
Sculley, D. (2010) · International World Wide Web Conference (WWW) · doi:10.1145/1772690.1772862 · Cited by 1,178 · see citing works
Improved Deep Embedded Clustering with Local Structure Preservation
Guo, X., Gao, L., Liu, X., & Yin, J. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/243 · Cited by 1,002 · see citing works
Joint Unsupervised Learning of Deep Representations and Image Clusters
Yang, J., Parikh, D., & Batra, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.556 · Cited by 879 · see citing works
SCAN: Learning to Classify Images Without Labels
Gansbeke, W. V., Vandenhende, S., Georgoulis, S., Proesmans, M., & Gool, L. V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58607-2_16 · Cited by 561 · see citing works
Local Aggregation for Unsupervised Learning of Visual Embeddings
Zhuang, C., Zhai, A., & Yamins, D. L. K. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00610 · Cited by 422 · see citing works
AD-Cluster: Augmented Discriminative Clustering for Domain Adaptive Person Re-Identification
Zhai, Y., Lu, S., Ye, Q., Shan, X., Chen, J., Ji, R., & Tian, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00904 · Cited by 341 · see citing works
Learning to Discover Novel Visual Categories via Deep Transfer Clustering
Han, K., Vedaldi, A., & Zisserman, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00849 · Cited by 314 · see citing works
Unsupervised Pre-Training of Image Features on Non-Curated Data
Caron, M., Bojanowski, P., Mairal, J., & Joulin, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00305 · Cited by 258 · see citing works
PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition
Su, K., Liu, X., & Shlizerman, E. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00965 · Cited by 244 · see citing works
Online Deep Clustering for Unsupervised Representation Learning
Zhan, X., Xie, J., Liu, Z., Ong, Y.-S., & Loy, C. C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00672 · Cited by 204 · see citing works
Unsupervised Discovery of Object Landmarks as Structural Representations
Zhang, Y., Guo, Y., Jin, Y., Luo, Y., He, Z., & Lee, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00285 · Cited by 192 · see citing works
Self-Supervised Convolutional Subspace Clustering Network
Zhang, J., Li, C.-G., You, C., Qi, X., Zhang, H., Guo, J. H., & Lin, Z. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00562 · Cited by 182 · see citing works
Deep Clustering by Gaussian Mixture Variational Autoencoders With Graph Embedding
Yang, L., Cheung, N. M., Li, J., & Fang, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00654 · Cited by 156 · see citing works
Unsupervised Part-Based Disentangling of Object Shape and Appearance
Lorenz, D., Bereska, L. F., Milbich, T., & Ommer, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01121 · Cited by 145 · see citing works
Unsupervised Learning of Object Landmarks by Factorized Spatial Embeddings
Thewlis, J., Bilen, H., & Vedaldi, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.348 · Cited by 143 · see citing works
ClusterFit: Improving Generalization of Visual Representations
Yan, X., Misra, I., Gupta, A. K., Ghadiyaram, D., & Mahajan, D. K. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00654 · Cited by 106 · see citing works
Autoregressive Unsupervised Image Segmentation
Ouali, Y., Hudelot, C., & Tami, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58571-6_9 · Cited by 75 · see citing works
Self-Supervised Learning of Geometrically Stable Features Through Probabilistic Introspection
Novotný, D., Albanie, S., Larlus, D., & Vedaldi, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00383 · Cited by 65 · see citing works
Deep Image Clustering with Category-Style Representation
Zhao, J., Lu, D., Ma, K., Zhang, Y., & Zheng, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58568-6_4 · Cited by 42 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [6][6] Xu Ji, Jo˜ ao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference on Computer Vision , pages 9865-9874, 2019. 4321, 4322 -
- [7][7] Jeff Johnson, Matthijs Douze, and Herv´ e J´ egou. Billionscale similarity search with gpus. arXiv preprint arXiv:1702.08734 , 2017. 4322 - · arXiv
Cited by
Sorted by citations
Large-Scale Unsupervised Semantic Segmentation
Gao, S., Li, Z.-Y., Yang, M.-H., Cheng, M.-M., Han, J., & Torr, P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3218275 · Cited by 88 · see citing works
- Authors
- Hyun Cho, J., Mall, U., Bala, K., & Hariharan, B.
- Affiliations
- Cornell University (US)
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 16789-16799
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3148215308
- Cited by
- 159
- References
- 39
- Citations by year
- 2026: 8 · 2025: 32 · 2024: 52 · 2023: 46 · 2022: 18 · 2021: 3
- FWCI
- 9.67
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Keywords
- unsupervised semantic segmentation, clustering, invariance, equivariance, geometric consistency, inductive bias, multi-label segmentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
