CoChat
Sign Up
PreprintCited by 2,679Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningVisual Attention and Saliency Detection

Swin Transformer V2: Scaling Up Capacity and Resolution

Open access
Liu, Ze, Hu, Han, Lin, Yutong, Yao, Zhuliang, Xie, Zhenda, Wei, Yixuan, Ning, Jia, Cao, Yue +4 more·2021·arXiv
DOI 10.48550/arxiv.2111.09883arXiv 2111.09883
Open this paper in CoChat — free accountOpen PDF

Abstract

Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536$\times$1,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time. Code is available at \url{https://github.com/microsoft/Swin-Transformer}.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2111.09883

Cites

65 of 65 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • DINO reduces model size to 1/15 compared to SwinV2-G.

    “we reduce the model size to 1/15 compared to SwinV2-G [22].”

    From DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection (2022) · cites this paper as [22]

  • Compared with previous SOTA models, DINO uses a much smaller model size (1/15 parameters compared with SwinV2-G), backbone pre-training data size (1/60 images compared with Florence), and detection pretraining data size (1/5 images compared with Florence), while achieving better results.

    “Compared with the previous SOTA models, we use a much smaller model size (1 / 15 parameters compared with SwinV2-G [22]), backbone pre-training data size (1 / 60 images compared with Florence), and detection pretraining data size (1 / 5 images compared with Florence), while achieving better results.”

    From DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection (2022) · cites this paper as [22]

  • Without using any other tricks, DINO achieves 63.1 on val2017 and 63.2 on test-dev without TTA, outperforming previous state-of-the-art 63.1 achieved by SwinV2 with a much neater solution.

    “Without using any other tricks, we achieve the result of 63 . 1 on val2017 and 63 . 2 on test-dev without test time augmentation (TTA) (see Appendix A), outperforming the previous state-of-the-art result 63 . 1 achieved by SwinV2 [22] with a much neater solution.”

    From DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection (2022) · cites this paper as [22]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

5 of the 70 references couldn’t be matched to a known paper.

Research works printed but not linked: 15

  1. Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S. Davis. Soft-nms -improving object detection with one line of code. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017. 9, 10
  2. Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delv- ing into high quality object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 6154-6162, 2018. 9
  3. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European Confer- ence on Computer Vision, pages 213-229. Springer, 2020. 9
  4. Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaox- iao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4974- 4983, 2019. 9, 10
  5. Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection tool- box and benchmark. arXiv preprint arXiv:1906.07155, 2019. 9 · arXiv
  6. Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmenta- tion with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020. 9, 10
  7. Hao-Shu Fang, Jianhua Sun, Runzhong Wang, Minghao Gou, Yong-Lu Li, and Cewu Lu. Instaboost: Boosting instance segmentation via probability map guided copy- pasting. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 682-691, 2019. 9
  8. Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7036- 7045, 2019. 9
  9. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Gir- shick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017. 9
  10. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kil- ian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pages 646-661. Springer, 2016. 9, 10
  11. Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2019. 9, 10
  12. Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chen- feng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. arXiv preprint arXiv:2011.12450, 2020. 9 · arXiv
  13. Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regular- ization strategy to train strong classifiers with localizable fea- tures. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 6023-6032, 2019. 9
  14. Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. arXiv preprint arXiv:1710.09412, 2017. 9 · arXiv
  15. Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13001-13008, 2020. 9, 10

Cited by

Sorted by citations

Related

Authors
Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., Wei, F., & Guo, B.
Affiliations
  • University of Science and Technology of China (CN)
  • Microsoft Research Asia (China) (CN)
  • Xi'an Jiaotong University (CN)
  • Tsinghua University (CN)
  • Huazhong University of Science and Technology (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
11999-12009
ISSN
1063-6919 (ISSN-L)
DOI
10.48550/arxiv.2111.09883
arXiv
2111.09883
OpenAlex
W4312349930
Cited by
2,679
References
70
Citations by year
2026: 483 · 2025: 858 · 2024: 757 · 2023: 485 · 2022: 94 · 2021: 1 · 2012: 1
FWCI
198.93
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Microsoft
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords
Swin Transformer, large vision models, model scaling, high-resolution image classification, object detection, semantic segmentation, video action recognition, position bias, training stability, GPU memory efficiency, self-supervised pre-training

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review