CoChat
Sign Up
Cited by 126Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMedical Image Segmentation Techniques

SegSort: Segmentation by Discriminative Sorting of Segments

Open access
Hwang, Jyh-Jing, Yu, Stella, Shi, Jianbo, Collins, Maxwell, Yang, Tien-Ju, Zhang, Xiao, Chen, Liang-Chieh·2019·2019 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv.2019.00743
Open this paper in CoChat — free accountOpen PDF

Abstract

Almost all existing deep learning approaches for semantic segmentation tackle this task as a pixel-wise classification problem. Yet humans understand a scene not in terms of pixels, but by decomposing it into perceptual groups and structures that are the basic building blocks of recognition. This motivates us to propose an end-to-end pixel-wise metric learning approach that mimics this process. In our approach, the optimal visual representation determines the right segmentation within individual images and associates segments with the same semantic classes across images. The core visual learning problem is therefore to maximize the similarity within segments and minimize the similarity between segments. Given a model trained this way, inference is performed consistently by extracting pixel-wise embeddings and clustering, with the semantic label determined by the majority vote of its nearest neighbors from an annotated set. As a result, we present the SegSort, as a first attempt using deep learning for unsupervised semantic segmentation, achieving 76% performance of its supervised counterpart. When supervision is available, SegSort shows consistent improvements over conventional approaches based on pixel-wise softmax training. Additionally, our approach produces more precise boundaries and consistent region predictions. The proposed SegSort further produces an interpretable result, as each choice of label can be easily understood from the retrieved nearest segments.

Integrity signals

Registry

Crossref · 10.1109/iccv.2019.00743

Cites

90 of 90 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Some USS methods utilize supervised prior knowledge, making it difficult to evaluate these methods.

    “2) Due to the lack of clear problem definition and standardized evaluation, some methods utilize supervised prior knowledge, e.g., supervised pre-trained network weights [46], supervised edge detection [30], and supervised saliency detection [31], [47], [48], making it difficult to evaluate these methods.”

    From Large-Scale Unsupervised Semantic Segmentation (2023) · cites this paper as [30]

  • Existing unsupervised segmentation methods mainly deal with small data with limited number of categories and small diversity, and are not suitable for the LUSS task.

    “Due to insufficient data, a few unsupervised segmentation methods [28], [29], [30], [31] mainly deal with small data with limited number of categories and small diversity, thus are not suitable for the LUSS task.”

    From Large-Scale Unsupervised Semantic Segmentation (2023) · cites this paper as [30]

  • Existing USS methods focus on small datasets and a few easy categories, so the advantages of unsupervised learning of rich representations from large-scale data are not explored.

    “1) Existing methods focus on small datasets [28], [29], [30], [31] and a few (e.g., 20+) easy categories [28] (e.g., sky and ground). Because of the insufficient data, the advantages of unsupervised learning of rich representations from large-scale data are not explored.”

    From Large-Scale Unsupervised Semantic Segmentation (2023) · cites this paper as [30]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Hwang, J.-J., Yu, S., Shi, J., Collins, M., Yang, T.-J., Zhang, X., & Chen, L.-C.
Affiliations
  • International Computer Science Institute (US)
  • University of Pennsylvania (US)
  • Google (United States) (US)
  • Massachusetts Institute of Technology (US)
ORCID
Venue
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2019-10
Pages
7333-7343
DOI
10.1109/iccv.2019.00743
OpenAlex
W2982106100
Cited by
124
References
87
Citations by year
2026: 2 · 2025: 16 · 2024: 27 · 2023: 25 · 2022: 29 · 2021: 20 · 2020: 5
FWCI
4.40
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1651389)
  • Defense Advanced Research Projects Agency
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
Keywords
unsupervised semantic segmentation, metric learning, segment clustering, nearest neighbor classification, interpretable segmentation, visual representation learning, boundary precision

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review