CoChat
Sign Up
PreprintCited by 83Advanced Neural Network ApplicationsVisual Attention and Saliency DetectionDomain Adaptation and Few-Shot Learning

SOLQ: Segmenting Objects by Learning Queries

Open access
Dong, Bin, Zeng, Fangao, Wang, Tiancai, Zhang, Xiangyu, Wei, Yichen·2021·arXiv
DOI 10.48550/arxiv.2106.02351arXiv 2106.02351
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we propose an end-to-end framework for instance segmentation. Based on the recently introduced DETR [1], our method, termed SOLQ, segments objects by learning unified queries. In SOLQ, each query represents one object and has multiple representations: class, location and mask. The object queries learned perform classification, box regression and mask encoding simultaneously in an unified vector form. During training phase, the mask vectors encoded are supervised by the compression coding of raw spatial masks. In inference time, mask vectors produced can be directly transformed to spatial masks by the inverse process of compression coding. Experimental results show that SOLQ can achieve state-of-the-art performance, surpassing most of existing approaches. Moreover, the joint learning of unified query representation can greatly improve the detection performance of DETR. We hope our SOLQ can serve as a strong baseline for the Transformer-based instance segmentation. Code is available at https://github.com/megvii-research/SOLQ.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2106.02351

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 54 · Cited by 83

Research works printed but not linked: 43

  1. [2][2] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proc. IEEE Int. Conf. Comp. Vis. , 2017. -
  2. [3][3] Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask scoring r-cnn. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2019. -
  3. [4][4] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2019. -
  4. [5][5] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2018. -
  5. [6][6] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Proc. Advances in Neural Inf. Process. Syst. , 2015. -
  6. [8][8] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In Proc. IEEE Int. Conf. Comp. Vis. , 2019. -
  7. [9][9] Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. Proc. Eur. Conf. Comp. Vis. , 2020. -
  8. [10][10] Youngwan Lee and Jongyoul Park. Centermask: Real-time anchor-free instance segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2020. -
  9. [11][11] Sida Peng, Wen Jiang, Huaijin Pi, Xiuli Li, Hujun Bao, and Xiaowei Zhou. Deep snake for real-time instance segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2020. -
  10. [13][13] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proc. IEEE Int. Conf. Comp. Vis. , 2019. -
  11. [14][14] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc Van Gool. Dynamic filter networks. In Proc. Advances in Neural Inf. Process. Syst. , 2016. -
  12. [15][15] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations. In Proc. Eur. Conf. Comp. Vis. , 2020. -
  13. [16][16] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. Solov2: Dynamic, faster and stronger. Proc. Advances in Neural Inf. Process. Syst. , 2020. -
  14. [18][18] Lingling Tao, Fatih Porikli, and René Vidal. Sparse dictionaries for semantic segmentation. In Proc. Eur. Conf. Comp. Vis. , 2014. -
  15. [19][19] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proc. Eur. Conf. Comp. Vis. , 2014. -
  16. [20][20] Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei. Fully convolutional instance-aware semantic segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2017. -
  17. [21][21] Rufeng Zhang, Zhi Tian, Chunhua Shen, Mingyu You, and Youliang Yan. Mask encoding for single shot instance segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2020. -
  18. [22][22] Hao Chen, Kunyang Sun, Zhi Tian, Chunhua Shen, Yongming Huang, and Youliang Yan. Blendmask: Top-down meets bottom-up for instance segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2020. -
  19. [23][23] Xinlei Chen, Ross Girshick, Kaiming He, and Piotr Dollár. Tensormask: A foundation for dense object segmentation. In Proc. IEEE Int. Conf. Comp. Vis. , 2019. -
  20. [24][24] Yuxin Fang, Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances as queries. Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2021. -
  21. [25][25] Naiyu Gao, Yanhu Shan, Yupei Wang, Xin Zhao, Yinan Yu, Ming Yang, and Kaiqi Huang. Ssap: Single-shot instance segmentation with affinity pyramid. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2019. -
  22. [26][26] Shu Liu, Jiaya Jia, Sanja Fidler, and Raquel Urtasun. Sgn: Sequential grouping networks for instance segmentation. In Proc. IEEE Int. Conf. Comp. Vis. , 2017. -
  23. [27][27] Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with a discriminative loss function. arXiv:1708.02551 , 2017. - · arXiv
  24. [28][28] Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. arXiv:1611.05424 , 2016. - · arXiv
  25. [29][29] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo. Polarmask: Single shot instance segmentation with polar representation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2020. -
  26. [30][30] Jie Hu, Liujuan Cao, Yao Lu, ShengChuan Zhang, Yan Wang, Ke Li, Feiyue Huang, Ling Shao, and Rongrong Ji. Istr: End-to-end instance segmentation with transformers. arXiv preprint arXiv:2105.00637 , 2021. - · arXiv
  27. [31][31] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2021. -
  28. [32][32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Advances in Neural Inf. Process. Syst. , 2017. -
  29. [36][36] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. arXiv:2012.00364 , 2020. - · arXiv
  30. [38][38] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv:2012.12877 , 2020. - · arXiv
  31. [41][41] David L Donoho. Compressed sensing. Proc. IEEE. Trans. Inf. The. , 2006. -
  32. [42][42] Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Proc. Chem. Int. Lab. Syst. , 1987. -
  33. [43][43] Nasir Ahmed, T\_ Natarajan, and Kamisetty R Rao. Discrete cosine transform. Proc. IEEE. Trans. Comp. , 1974. -
  34. [44][44] Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren. Learning in the frequency domain. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2020. -
  35. [45][45] Xing Shen, Jirui Yang, Chunbo Wei, Bing Deng, Jianqiang Huang, Xiansheng Hua, Xiaoliang Cheng, and Kewei Liang. Dct-mask: Discrete cosine transform mask representation for instance segmentation. arXiv:2011.09876 , 2020. - · arXiv
  36. [46][46] Shao-Yuan Lo and Hsueh-Ming Hang. Exploring semantic segmentation on the dct representation. In ACM Int. Conf. Mul. Med. Asia . 2019. -
  37. [48][48] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proc. IEEE Int. Conf. Comp. Vis. , 2017. -
  38. [49][49] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2019. -
  39. [51][51] Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2010. -
  40. [53][53] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2016. -
  41. [55][55] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov. Boundary iou: Improving object-centric image segmentation evaluation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2021. -
  42. [56][56] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2020. -
  43. [57][57] Tianheng Cheng, Xinggang Wang, Lichao Huang, and Wenyu Liu. Boundary-preserving mask r-cnn. In Proc. Eur. Conf. Comp. Vis. , 2020.

Authors
Dong, B., Zeng, F., Wang, T., Zhang, X., & Wei, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
34
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2106.02351
arXiv
2106.02351
OpenAlex
W3168783492
Cited by
83
References
54
Citations by year
2025: 11 · 2024: 25 · 2023: 32 · 2022: 13 · 2021: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
instance segmentation, object queries, DETR, transformer-based segmentation, object detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review