CoChat
Sign Up
PreprintCited by 224Adversarial Robustness in Machine LearningAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; 5 of the 5 references it cites were found in a registry or an index.

Towards Robust Vision Transformer

Open access
Mao, Xiaofeng, Qi, Gege, Chen, Yuefeng, Li, Xiaodan, Duan, Ranjie, Ye, Shaokai, He, Yuan, Xue, Hui·2021·arXiv
DOI 10.48550/arxiv.2105.07926arXiv 2105.07926
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advances on Vision Transformer (ViT) and its improved variants have shown that self-attention-based networks surpass traditional Convolutional Neural Networks (CNNs) in most vision tasks. However, existing ViTs focus on the standard accuracy and computation cost, lacking the investigation of the intrinsic influence on model robustness and generalization. In this work, we conduct systematic evaluation on components of ViTs in terms of their impact on robustness to adversarial examples, common corruptions and distribution shifts. We find some components can be harmful to robustness. By using and combining robust components as building blocks of ViTs, we propose Robust Vision Transformer (RVT), which is a new vision transformer and has superior performance with strong robustness. We further propose two new plug-and-play techniques called position-aware attention scaling and patch-wise augmentation to augment our RVT, which we abbreviate as RVT*. The experimental results on ImageNet and six robustness benchmarks show the advanced robustness and generalization ability of RVT compared with previous ViTs and state-of-the-art CNNs. Furthermore, RVT-S* also achieves Top-1 rank on multiple robustness leaderboards including ImageNet-C and ImageNet-Sketch. The code will be available at \url{https://github.com/alibaba/easyrobust}.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2105.07926

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Adding Convs to Transformer's MLP block improves accuracy and robustness.

    “It also provides an explanation of how adding Convs to Transformer's MLP block improves accuracy and robustness (Yuan et al., 2021; Guo et al., 2021; Mao et al., 2021).”

    From How Do Vision Transformers Work? (2022) · cites this paper as Mao et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. [1][1] Yutong Bai, Jieru Mei, Alan L Yuille, and Cihang Xie. Are transformers more robust than cnns? Advances in Neural Information Processing Systems , 34, 2021. 8 -
  2. [3][3] Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and Huaxia Xia. Do we really need explicit position encodings for vision transformers? arXiv preprint arXiv:2102.10882 , 2021. 3, 4, 12 - · arXiv
  3. [4][4] Nadav Cohen and Amnon Shashua. Inductive bias of deep convolutional networks through pooling geometry. In Proceedings of the International Conference on Learning Representations , 2017. 3 -
  4. [5][5] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the Computer Vision and Pattern Recognition Workshops , pages 702-703, 2020. 6, 8 -
  5. [6][6] St´ ephane d'Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103.10697 , 2021. 4, 7 - · arXiv
  6. [7][7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the Computer Vision and Pattern Recognition , pages 248-255. Ieee, 2009. 6 -
  7. [8][8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2019. 1 -
  8. [9][9] Minjing Dong, Yanxi Li, Yunhe Wang, and Chang Xu. Adversarially robust neural architectures. arXiv preprint arXiv:2009.00902 , 2020. 2 - · arXiv
  9. [11][11] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations , 2015. 1, 2, 5, 6, 7 -
  10. [12][12] Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herv´ e J´ egou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint arXiv:2104.01136 , 2021. 2, 4 - · arXiv
  11. [13][13] Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu, and Dahua Lin. When nas meets robustness: In search of robust architectures against adversarial attacks. In Proceedings of the Computer Vision and Pattern Recognition , pages 631-640, 2020. 2 -
  12. [14][14] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.00112 , 2021. 2, 7 - · arXiv
  13. [16][16] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006.16241 , 2020. 2, 6, 7 - · arXiv
  14. [17][17] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In Proceedings of the International Conference on Learning Representations , 2019. 2, 6 -
  15. [18][18] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In Proceedings of the International Conference on Learning Representation , 2020. 2, 6, 7 -
  16. [19][19] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.07174 , 2019. 6 - · arXiv
  17. [20][20] Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. arXiv preprint arXiv:2103.16302 , 2021. 1, 3, 7 - · arXiv
  18. [21][21] Guolin Ke, Di He, and Tie-Yan Liu. Rethinking the positional encoding in language pre-training. arXiv preprint arXiv:2006.15595 , 2020. 5 - · arXiv
  19. [22][22] Yingwei Li, Qihang Yu, Mingxing Tan, Jieru Mei, Peng Tang, Wei Shen, Alan Yuille, and Cihang Xie. Shape-texture debiased neural network training. In Proceedings of the International Conference on Learning Representations , 2021. 2, 7 -
  20. [23][23] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2104.05707 , 2021. 1, 4 - · arXiv
  21. [24][24] Ji Lin, Chuang Gan, and Song Han. Defensive quantization: When efficiency meets robustness. In Proceedings of the International Conference on Learning Representations , 2019. 2 -
  22. [26][26] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Rtowards deep learning models resistant to adversarial attacks. In Proceedings of the International Conference on Learning Representations , 2018. 2, 6, 7 -
  23. [27][27] Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk. On the robustness of vision transformers to adversarial examples. arXiv preprint arXiv:2104.02610 , 2021. 2 - · arXiv
  24. [28][28] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in Neural Information Processing Systems , 2019. 4 -
  25. [29][29] Sayak Paul and Pin-Yu Chen. Vision transformers are robust learners. arXiv preprint arXiv:2105.07581 , 2021. 2 - · arXiv
  26. [31][31] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Doll´ ar. Designing network design spaces. In Proceedings of the Computer Vision and Pattern Recognition , pages 10428-10436, 2020. 7 -
  27. [32][32] Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. Kornia: an open source differentiable computer vision library for pytorch. In Proceedings of the Winter Conference on Applications of Computer Vision , pages 3674-3683, 2020. 8 -
  28. [33][33] Evgenia Rusak, Lukas Schott, Roland S Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions. In Proceedings of the European Conference on Computer Vision , pages 53-69. Springer, 2020. 2, 7 -
  29. [34][34] Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, and Cho-Jui Hsieh. On the adversarial robustness of visual transformers. arXiv preprint arXiv:2103.15670 , 2021. 2, 7 - · arXiv
  30. [35][35] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Selfattention with relative position representations. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2018. 3, 12 -
  31. [36][36] Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin-Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy?a comprehensive study on the robustness of 18 deep image classification models. In Proceedings of the European Conference on Computer Vision , pages 631-648, 2018. 2 -
  32. [37][37] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the Computer Vision and Pattern Recognition , pages 2818-2826, 2016. 7 -
  33. [38][38] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Proceedings of the International Conference on Learning Representations , 2014. 2 -
  34. [39][39] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning , pages 6105-6114. PMLR, 2019. 7 -
  35. [40][40] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv´ e J´ egou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877 , 2020. 1, 2, 6, 7 - · arXiv
  36. [41][41] Cristina Vasconcelos, Hugo Larochelle, Vincent Dumoulin, Nicolas Le Roux, and Ross Goroshin. An effective antialiasing approach for residual networks. arXiv preprint arXiv:2011.10675 , 2020. 2 - · arXiv
  37. [42][42] Haohan Wang, Songwei Ge, Eric P Xing, and Zachary C Lipton. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems , 2019. 2, 6 -
  38. [44][44] Boxi Wu, Jinghui Chen, Deng Cai, Xiaofei He, and Quanquan Gu. Do wider neural networks really help adversarial robustness? arXiv preprint arXiv:2010.01279 , 2020. 2 - · arXiv
  39. [45][45] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.15808 , 2021. 1 - · arXiv
  40. [46][46] Cihang Xie, Mingxing Tan, Boqing Gong, Alan Yuille, and Quoc V Le. Smooth adversarial training. arXiv preprint arXiv:2006.14536 , 2020. 2 - · arXiv
  41. [47][47] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the Computer Vision and Pattern Recognition , pages 10687-10698, 2020. 2 -
  42. [48][48] Saining Xie, Ross Girshick, Piotr Doll´ ar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the Computer Vision and Pattern Recognition , pages 1492-1500, 2017. 7 -
  43. [49][49] Shaokai Ye, Kaidi Xu, Sijia Liu, Hao Cheng, Jan-Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma, Yanzhi Wang, and Xue Lin. Adversarial robustness vs. model compression, or both? In Proceedings of the International Conference on Computer Vision , pages 111-120, 2019. 2 -
  44. [50][50] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. arXiv preprint arXiv:2103.11816 , 2021. 1, 2, 4 - · arXiv
  45. [51][51] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokensto-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986 , 2021. 2, 7 - · arXiv
  46. [52][52] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Proceedings of the International Conference on Learning Representations , 2018. 6 -
  47. [53][53] Richard Zhang. Making convolutional networks shiftinvariant again. In Proceedings of International Conference on Machine Learning , pages 7324-7334. PMLR, 2019. 2, 7

Cited by

Sorted by citations

Metadata

Authors
Mao, X., Qi, G., Chen, Y., Li, X., Duan, R., Ye, S., He, Y., & Xue, H.
Affiliations
  • Alibaba Group (China) (CN)
  • Swinburne University of Technology (AU)
  • École Polytechnique Fédérale de Lausanne (CH)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
12032-12041
ISSN
1063-6919 (ISSN-L)
DOI
10.48550/arxiv.2105.07926
arXiv
2105.07926
OpenAlex
W4312957757
Cited by
224
References
5
Citations by year
2026: 28 · 2025: 57 · 2024: 68 · 2023: 49 · 2022: 17 · 2021: 5
FWCI
26.40
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
vision transformer, model robustness, adversarial examples, image corruptions, distribution shift, self-attention, generalization, robustness analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review