Cited by 34Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsMedical Image Segmentation Techniques

Mix-and-Match Tuning for Self-Supervised Semantic Segmentation

Open access
Zhan, Xiaohang, Liu, Ziwei, Luo, Ping, Tang, Xiaoou, Loy, Chen·2018·Proceedings of the AAAI Conference on Artificial Intelligence

Abstract

Deep convolutional networks for semantic image segmentation typically require large-scale labeled data, e.g., ImageNet and MS COCO, for network pre-training. To reduce annotation efforts, self-supervised semantic segmentation is recently proposed to pre-train a network without any human-provided labels. The key of this new form of learning is to design a proxy task (e.g., image colorization), from which a discriminative loss can be formulated on unlabeled data. Many proxy tasks, however, lack the critical supervision signals that could induce discriminative representation for the target image segmentation task. Thus self-supervision’s performance is still far from that of supervised pre-training. In this study, we overcome this limitation by incorporating a "mix-and-match" (M&M) tuning stage in the self-supervision pipeline. The proposed approach is readily pluggable to many self-supervision methods and does not use more annotated samples than the original process. Yet, it is capable of boosting the performance of target image segmentation task to surpass fully-supervised pre-trained counterpart. The improvement is made possible by better harnessing the limited pixel-wise annotations in the target dataset. Specifically, we first introduce the "mix" stage, which sparsely samples and mixes patches from the target set to reflect rich and diverse local patch statistics of target images. A ‘match’ stage then forms a class-wise connected graph, which can be used to derive a strong triplet-based discriminative loss for finetuning the network. Our paradigm follows the standard practice in existing self-supervised studies and no extra data or label is required. With the proposed M&M approach, for the first time, a self-supervision method can achieve comparable or even better performance compared to its ImageNet pretrained counterpart on both PASCAL VOC2012 dataset and CityScapes dataset.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v32i1.12331

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing USS methods focus on small datasets and a few easy categories, so the advantages of unsupervised learning of rich representations from large-scale data are not explored.

    “1) Existing methods focus on small datasets [28], [29], [30], [31] and a few (e.g., 20+) easy categories [28] (e.g., sky and ground). Because of the insufficient data, the advantages of unsupervised learning of rich representations from large-scale data are not explored.”

    From Large-Scale Unsupervised Semantic Segmentation (2023) · cites this paper as [29]

    The Introduction (p. 1) says pixel-wise annotations for fine-tuning are available in only thousands of images, which supports a narrower point about limited labeled data. But the Experiments section (p. 5) says the self-supervised methods used 1.3 million unlabeled ImageNet images for training, contrary to the claim that large-scale data was not explored. The Introduction (p. 1) attributes their performance gap mainly to a mismatch between the proxy and target tasks, not insufficient data. (LLM, conf=0.87)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured
  • Existing unsupervised segmentation methods mainly deal with small data with limited number of categories and small diversity, and are not suitable for the LUSS task.

    “Due to insufficient data, a few unsupervised segmentation methods [28], [29], [30], [31] mainly deal with small data with limited number of categories and small diversity, thus are not suitable for the LUSS task.”

    From Large-Scale Unsupervised Semantic Segmentation (2023) · cites this paper as [29]

    The Introduction (p. 1) discusses limited pixel-wise annotations for fine-tuning, and the Experiments (p. 5) report results on PASCAL VOC 2012. Neither those passages nor the passages found by searching for dataset size, category limits, diversity, and LUSS say that unsupervised segmentation methods are confined to small, low-diversity datasets or are unsuitable for LUSS. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Zhan, X., Liu, Z., Luo, P., Tang, X., & Loy, C.
Affiliations
Chinese University of Hong Kong (HK)
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Published
2018-04-27
Volume
32
Issue
1
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
OpenAlex
W2962902289
Cited by
34
References
42
Citations by year
2025: 1 · 2024: 3 · 2023: 5 · 2022: 4 · 2021: 6 · 2020: 10 · 2019: 3 · 2018: 2
FWCI
2.14
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Chinese University of Hong Kong
  • SenseTime Group
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
Keywords
proxy tasks, triplet loss, unlabeled data, pre-training, cityscape, Pascal VOC 2012

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review