CoChat
Sign Up
Cited by 1,259Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningVisual Attention and Saliency Detection

Going deeper with Image Transformers

Open access
Touvron, Hugo, Cord, Matthieu, Sablayrolles, Alexandre, Synnaeve, Gabriel, Jegou, Herve·2021·2021 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv48922.2021.00010
Open this paper in CoChat — free accountOpen PDF

Abstract

Transformers have been recently adapted for large scale image classification, achieving high scores shaking up the long supremacy of convolutional neural networks. However the optimization of vision transformers has been little studied so far. In this work, we build and optimize deeper transformer networks for image classification. In particular, we investigate the interplay of architecture and optimization of such dedicated transformers. We make two architecture changes that significantly improve the accuracy of deep transformers. This leads us to produce models whose performance does not saturate early with more depth, for in-stance we obtain 86.5% top-1 accuracy on Imagenet when training with no external data, we thus attain the current sate of the art with less floating-point operations and parameters. Our best model establishes the new state of the art on Imagenet with Reassessed labels and Imagenet-V2 / match frequency, in the setting with no additional training data. We share our code and models1.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv48922.2021.00010

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Searched CV transformer models outperform SOTA CNN and manually designed self-attention-based models, such as DeiT, regarding accuracy and number of parameters.

    “The searched CV transformer models outperform SOTA CNN and manually designed self-attention-based models, such as DeiT [52], regarding the accuracy and the number of parameters.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [52]

    Section 4.4.2 reports that CaiT achieves higher ImageNet accuracy than DeiT, and the Introduction says its best model matches a leading CNN's accuracy with fewer parameters. But the paper describes CaiT as a design built on ViT, not as a searched model, and says it is on par with—not more accurate than—the leading CNN on ImageNet1k-val. It also notes that CNNs are better in low-parameter regimes. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Methods such as NASformer have better accuracy and fewer parameters than manually designed models such as Swin and DeiT.

    “Methods such as NASformer [212] have better accuracy and fewer parameters than manually designed models such as Swin [7] and DeiT [52].”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [52]

    Section 4.5.1 compares the paper’s CaiT architecture with DeiT, and Section 4.3 describes DeiT as an existing model. The cited paper is about CaiT, not the original DeiT work; it does not establish the sentence’s NASformer-versus-DeiT claim about accuracy and parameter count. No page numbers were available in our copy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • With increasing input scale, the growth rate of the GFLOPs of PVT is greater than ResNet but lower than ViT.

    “With increasing input scale, the growth rate of the GFLOPs of our PVT is greater than ResNet [22], but lower than ViT [13], as shown in Figure 6.”

    From Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021) · cites this paper as [22]

    The Complexity section says regular self-attention has quadratic complexity in the number of patches, supporting the premise that a ViT-like model becomes more computationally expensive as input scale increases. But the paper does not compare ViT’s GFLOPs growth rate with PVT’s, so it does not establish the cited “lower than ViT” comparison. Page numbers are unavailable in our copy. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Polyak-Ruppert averaging is often used to simulate model ensembling to improve the performance of a network at the end of training.

    “Polyak-Ruppert averaging is often used to simulate model ensembling to improve the performance of a network at the end of the training [38].”

    From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [38]

    The sections on LayerScale and training discuss learnable scaling factors, convergence, and performance, but not Polyak-Ruppert averaging. Searches for that term, model-weight averaging, and ensembling found no passage in our copy supporting the attributed claim. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

  • MobileViT attains six percentage points better accuracy than DeiT with 3.4M fewer parameters.

    “MobileViT [51] is an example of such lightweight ViT, which attains six percentage points better accuracy than the DeiT model [52], with 3.4M fewer parameters.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [52]

    Section 4.4.2 compares CaiT with DeiT, and the table reproduced under “7 Conclusion” reports DeiT-S accuracy and parameter count. The copy does not report a MobileViT comparison or the claimed six-percentage-point accuracy and 3.4-million-parameter differences (searched: MobileViT, DeiT-Tiny, and the stated differences). (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 58

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. 2 · arXiv
  2. Thomas C. Bachlechner, Bodhisattwa Prasad Majumder, H. H. Mao, G. Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. arXiv preprint arXiv:2003.04887, 2020. 2, 3, 4, 10, 22 · arXiv
  3. Irwan Bello. Lambdanetworks: Modeling long-range interactions without atten- tion. In International Conference on Learning Representations, 2021. 21
  4. Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. At- tention augmented convolutional networks. In Conference on Computer Vision and Pattern Recognition, 2019. 21
  5. Maxim Berman, Hervé Jégou, Andrea Vedaldi, Iasonas Kokkinos, and Matthijs Douze. Multigrain: a unified image embedding for classes and instances. arXiv preprint arXiv:1902.05509, 2019. 13 · arXiv
  6. Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aaron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159, 2020. 3, 13, 14 · arXiv
  7. Andrew Brock, Soham De, and Samuel L Smith. Characterizing signal prop- agation to close the performance gap in unnormalized resnets. arXiv preprint arXiv:2101.08692, 2021. 2 · arXiv
  8. A. Brock, Soham De, S. L. Smith, and K. Simonyan. High-performance large-scale image recognition without normalization. arXiv preprint arXiv:2102.06171, 2021. 2, 13, 14, 21 · arXiv
  9. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Ka- plan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 22 · arXiv
  10. Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In Conference on Computer Vision and Pattern Recognition, 2005. 21
  11. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long se- quences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. 2, 4, 21 · arXiv
  12. Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space. arXiv preprint arXiv:1909.13719, 2019. 14 · arXiv
  13. Soham De and Samuel L Smith. Batch normalization biases residual blocks to- wards the identity function in deep networks. arXiv e-prints, pages arXiv-2002, 2020. 2, 3, 4, 22
  14. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. 8
  15. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for im- age recognition at scale. International Conference on Learning Representations, 2021. 2, 3, 8, 13, 14, 15, 21
  16. Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909.11556, 2019. ICLR 2020. · arXiv
  17. Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou, and Armand Joulin. Training with quantization noise for extreme model compression. arXiv preprint arXiv:2004.07320, 2020. 8 · arXiv
  18. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018. I · arXiv
  19. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010. 22
  20. Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non- autoregressive neural machine translation. arXiv preprint arXiv:1711.02281, 2017. 22 · arXiv
  21. Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.00112, 2021. 14 · arXiv
  22. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027, 2016. 2 · arXiv
  23. Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repeti- tion. In Conference on Computer Vision and Pattern Recognition, 2020. 13
  24. Grant Van Horn, Oisin Mac Aodha, Yang Song, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist challenge 2018 dataset. arXiv preprint arXiv:1707.06642, 2018. 15 · arXiv
  25. Grant Van Horn, Oisin Mac Aodha, Yang Song, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist challenge 2019 dataset. arXiv preprint arXiv:1707.06642, 2019. 15 · arXiv
  26. Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. arXiv preprint arXiv:1709.01507, 2017. 21 · arXiv
  27. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, 2016. 8, 22
  28. Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs. Improving trans- former optimization through better initialization. In International Conference on Ma- chine Learning, pages 4475-4483. PMLR, 2020. 2, 4, 22
  29. Shigeki Karita, Nanxin Chen, Tomoki Hayashi, et al. A comparative study on transformer vs rnn in speech applications. arXiv preprint arXiv:1909.06317, 2019. 22 · arXiv
  30. Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. arXiv preprint arXiv:1807.03039, 2018. 4 · arXiv
  31. Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Represen- tation and Recognition (3dRR-13), 2013. 15
  32. Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. Conference on Computer Vision and Pattern Recognition, 2019. 21
  33. I. Loshchilov and F. Hutter. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 2017. 13 · arXiv
  34. Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR, 2018. 21
  35. H. Pham, Qizhe Xie, Zihang Dai, and Quoc V. Le. Meta pseudo labels. arXiv preprint arXiv:2003.10580, 2020. 21 · arXiv
  36. Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. Conference on Computer Vision and Pattern Recognition, 2020. 1, 13, 14
  37. B. Recht, Rebecca Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811, 2019. 3, 13, 14 · arXiv
  38. A. Romero, Nicolas Ballas, S. Kahou, Antoine Chassang, C. Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2015. 22 · arXiv
  39. Noam Shazeer, Zhenzhong Lan, Youlong Cheng, N. Ding, and L. Hou. Talking- heads attention. arXiv preprint arXiv:2003.02436, 2020. 12 · arXiv
  40. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015. 22
  41. A. Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, P. Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. arXiv preprint arXiv:2101.11605, 2021. 21 · arXiv
  42. R. Srivastava, Klaus Greff, and J. Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015. 22 · arXiv
  43. R. Srivastava, Klaus Greff, and J. Schmidhuber. Training very deep networks. In NIPS, 2015. 22
  44. Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-end asr: from supervised to semi-supervised learning with modern archi- tectures. arXiv preprint arXiv:1911.08460, 2019. 22 · arXiv
  45. C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Conference on Computer Vision and Pattern Recognition, 2015. 22
  46. Hugo Touvron, M. Cord, M. Douze, F. Massa, Alexandre Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877, 2020. 3, 7, 8, 12, 13, 14, 15, 16, 17, 18, 21, III · arXiv
  47. Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy: Fixefficientnet. arXiv preprint arXiv:2003.08237, 2020. 14 · arXiv
  48. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. 2, 3, 22 · arXiv
  49. Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Masayoshi Tomizuka, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision. arXiv preprint arXiv:2006.03677, 2020. 21 · arXiv
  50. L. Xiao, Y. Bahri, Jascha Sohl-Dickstein, S. Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10, 000-layer vanilla convolutional neural networks. arXiv preprint arXiv:1806.05393, 2018. 22 · arXiv
  51. Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, A. Yuille, and Quoc V. Le. Adversarial examples improve image recognition. Conference on Computer Vision and Pattern Recognition, 2020. 14
  52. Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggre- gated residual transformations for deep neural networks. Conference on Computer Vision and Pattern Recognition, 2017. 1
  53. Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. Self-training and pre-training are complementary for speech recognition. arXiv preprint arXiv:2010.11430, 2020. 22 · arXiv
  54. L. Yuan, Y. Chen, Tao Wang, Weihao Yu, Yujun Shi, F. Tay, Jiashi Feng, and S. Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986, 2021. 14, 21 · arXiv
  55. Hongyi Zhang, Yann Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv:1901.09321, 2019. 2, 3, 4, 10, 22 · arXiv
  56. Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi-Li Zhang, Haibin Lin, Yu e Sun, Tong He, Jonas Mueller, R. Manmatha, M. Li, and Alex Smola. Resnest: Split-attention networks. arXiv preprint arXiv:2004.08955, 2020. 21 · arXiv
  57. Yu Zhang, James Qin, Daniel S Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V Le, and Yonghui Wu. Pushing the limits of semi-supervised learning for automatic speech recognition. arXiv preprint arXiv:2010.10504, 2020. 22 · arXiv
  58. Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for im- age recognition. In Conference on Computer Vision and Pattern Recognition, 2020. 21

Cited by

Sorted by citations

Authors
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., & Jegou, H.
Affiliations
  • Université Paris Cité (FR)
  • Meta (Israel) (IL)
ORCID
Venue
2021 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2021-10
Pages
32-42
DOI
10.1109/iccv48922.2021.00010
OpenAlex
W3146097248
Cited by
1,259
References
14
Citations by year
2026: 152 · 2025: 310 · 2024: 284 · 2023: 283 · 2022: 168 · 2021: 62
FWCI
65.17
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords
vision transformer, image classification, network structure optimization, ImageNet, match frequency, parameter efficiency, floating-point operations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review