CoChat
Sign Up
Cited by 409Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningVisual Attention and Saliency Detection

DeiT III: Revenge of the ViT

Open access
Touvron, Hugo, Cord, Matthieu, Jégou, Hervé·2022·Lecture Notes in Computer Science
DOI 10.1007/978-3-031-20053-3_30
Open this paper in CoChat — free accountOpen PDF

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/978-3-031-20053-3_30

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Employing superior pre-training techniques on T-Base such as DeiT yields better performance for smaller Vision Transformer models on Cityscapes compared to their larger counterparts.

    “However, employing superior pre-training techniques on 'TBase' such as DeiT Touvron et al. (2022) yields even better performance for these smaller Vision Transformer models on Cityscapes compared to their larger counterparts.”

    From Vision Transformers: From Semantic Segmentation to Dense Prediction (2024) · cites this paper as Touvron et al. (2022)

    Section 4.4.2 (p. 18) reports that the paper's ViT training recipe improves the FLOPs–accuracy trade-off in semantic segmentation, but those experiments use ADE20k, not Cityscapes. Table 11 (p. 19) does not establish that a T-Base model outperforms larger ViTs; its reported ViT-L segmentation scores exceed those of ViT-B. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • DeiT pretrained on ImageNet-1K and T-Base Vision Transformer perform better than ImageNet-1K pretrained T-Large Vision Transformer on Cityscapes but worse on ADE20K.

    “Additionally, employing distillation, Deit Touvron et al. (2022) pretrained on ImageNet-1K and 'T-Base' Vision Transformer perform better than ImageNet-1K pretrained 'T-Large' Vision Transformer on Cityscapes but worse on ADE20K.”

    From Vision Transformers: From Semantic Segmentation to Dense Prediction (2024) · cites this paper as Touvron et al. (2022)

    Table 11 (pp. 18–19) reports that the ImageNet-1k-pretrained ViT-B scores lower than ViT-L on ADE20K, supporting that part of the comparison. The paper does not report a Cityscapes comparison, and §4.3 (p. 14) explicitly says its approach does not use distillation. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • SETR paired with DeiT shows a significant decline in results.

    “Even with DeiT Touvron et al. (2022) to reduce computation costs, the results of SETR paired with DeiT show a significant decline.”

    From Vision Transformers: From Semantic Segmentation to Dense Prediction (2024) · cites this paper as Touvron et al. (2022)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 41 · Cited by 409

Research works printed but not linked: 54

  1. [2][2] Bao, H., Dong, L., Wei, F.: Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 (2021) - · arXiv
  2. [4][4] Chu, P., Bian, X., Liu, S., Ling, H.: Feature space augmentation for long-tailed data. arXiv preprint arXiv:2008.03673 (2020) - · arXiv
  3. [5][5] Cordonnier, J.B., Loukas, A., Jaggi, M.: On the relationship between selfattention and convolutional layers. arXiv preprint arXiv:1911.03584 (2019) - · arXiv
  4. [6][6] Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: RandAugment: Practical automated data augmentation with a reduced search space. arXiv preprint arXiv:1909.13719 (2019) - · arXiv
  5. [7][7] Cubuk, E.D., Zoph, B., Man´ e, D., Vasudevan, V., Le, Q.V.: Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 (2018) - · arXiv
  6. [8][8] d'Ascoli, S., Touvron, H., Leavitt, M.L., Morcos, A.S., Biroli, G., Sagun, L.: Convit: Improving vision transformers with soft convolutional inductive biases. In: ICML (2021) -
  7. [9][9] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A largescale hierarchical image database. In: Conference on Computer Vision and Pattern Recognition. pp. 248-255 (2009) -
  8. [10][10] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: NAACL (2019) -
  9. [11][11] Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., Guo, B.: Cswin transformer: A general vision transformer backbone with cross-shaped windows. arXiv preprint arXiv:2107.00652 (2021) - · arXiv
  10. [12][12] Dong, X., Bao, J., Zhang, T., Chen, D., Zhang, W., Yuan, L., Chen, D., Wen, F., Yu, N.: Peco: Perceptual codebook for bert pre-training of vision transformers. arXiv preprint arXiv:2111.12710 (2021) - · arXiv
  11. [13][13] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021) -
  12. [14][14] El-Nouby, A., Izacard, G., Touvron, H., Laptev, I., Jegou, H., Grave, E.: Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112.10740 (2021) - · arXiv
  13. [15][15] El-Nouby, A., Neverova, N., Laptev, I., J´ egou, H.: Training vision transformers for image retrieval. arXiv preprint arXiv:2102.05644 (2021) - · arXiv
  14. [16][16] El-Nouby, A., Touvron, H., Caron, M., Bojanowski, P., Douze, M., Joulin, A., Laptev, I., Neverova, N., Synnaeve, G., Verbeek, J., et al.: Xcit: Crosscovariance image transformers. arXiv preprint arXiv:2106.09681 (2021) - · arXiv
  15. [18][18] Graham, B., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., J´ egou, H., Douze, M.: Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint arXiv:2104.01136 (2021) - · arXiv
  16. [21][21] Heo, B., Yun, S., Han, D., Chun, S., Choe, J., Oh, S.J.: Rethinking spatial dimensions of vision transformers. arXiv preprint arXiv:2103.16302 (2021) - · arXiv
  17. [22][22] Horn, G.V., Mac Aodha, O., Song, Y., Shepard, A., Adam, H., Perona, P., Belongie, S.J.: The iNaturalist species classification and detection dataset. arXiv preprint arXiv:1707.06642 (2017) - · arXiv
  18. [23][23] Horn, G.V., Mac Aodha, O., Song, Y., Shepard, A., Adam, H., Perona, P., Belongie, S.J.: The inaturalist challenge 2018 dataset. arXiv preprint arXiv:1707.06642 (2018) - · arXiv
  19. [24][24] Huang, G., Sun, Y., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: European Conference on Computer Vision (2016) -
  20. [25][25] Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., Houlsby, N.: Big transfer (bit): General visual representation learning. arXiv preprint arXiv:1912.11370 6 , 3 (2019) - · arXiv
  21. [26][26] Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for finegrained categorization. In: IEEE Workshop on 3D Representation and Recognition (2013) -
  22. [27][27] Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: NeurIPS (2012) -
  23. [29][29] LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86 (11), 2278-2324 (1998) -
  24. [30][30] LingChen, T.C., Khonsari, A., Lashkari, A., Nazari, M.R., Sambee, J.S., Nascimento, M.A.: Uniformaugment: A search-free probabilistic data augmentation approach. arXiv preprint arXiv:2003.14348 (2020) - · arXiv
  25. [32][32] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. arXiv preprint arXiv:2201.03545 (2022) - · arXiv
  26. [33][33] Loshchilov, I., Hutter, F.: Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101 (2017) - · arXiv
  27. [34][34] M¨ uller, S., Hutter, F.: Trivialaugment: Tuning-free yet state-of-the-art data augmentation. arXiv preprint arXiv:2103.10158 (2021) - · arXiv
  28. [35][35] Neimark, D., Bar, O., Zohar, M., Asselmann, D.: Video transformer network. arXiv preprint arXiv:2102.00719 (2021) - · arXiv
  29. [38][38] Radosavovic, I., Kosaraju, R.P., Girshick, R.B., He, K., Doll´ ar, P.: Designing network design spaces. Conference on Computer Vision and Pattern Recognition (2020) -
  30. [39][39] Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do ImageNet classifiers generalize to ImageNet? In: International Conference on Machine Learning (2019) -
  31. [40][40] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115 (3), 211-252 (2015) -
  32. [41][41] Simonyan, K., Zisserman, A.: Very deep convolutional networks for largescale image recognition. In: International Conference on Learning Representations (2015) -
  33. [42][42] Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., Beyer, L.: Howtotrain your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270 (2021) - · arXiv
  34. [43][43] Szegedy, C., Wei Liu, Yangqing Jia, Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Conference on Computer Vision and Pattern Recognition (2015) -
  35. [44][44] Tan, M., Le, Q.V.: EfficientNet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946 (2019) - · arXiv
  36. [45][45] Tan, M., Le, Q.V.: Efficientnetv2: Smaller models and faster training. In: International Conference on Machine Learning (2021) -
  37. [46][46] Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Keysers, D., Uszkoreit, J., Lucic, M., Dosovitskiy, A.: MLP-Mixer: An all-MLP architecture for vision. arXiv preprint arXiv:2105.01601 (2021) - · arXiv
  38. [47][47] Touvron, H., Bojanowski, P., Caron, M., Cord, M., El-Nouby, A., Grave, E., Joulin, A., Synnaeve, G., Verbeek, J., J´ egou, H.: ResMLP: feedforward networks for image classification with data-efficient training. arXiv preprint arXiv:2105.03404 (2021) - · arXiv
  39. [48][48] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., J´ egou, H.: Training data-efficient image transformers & distillation through attention. International Conference on Machine Learning (2021) -
  40. [49][49] Touvron, H., Cord, M., El-Nouby, A., Bojanowski, P., Joulin, A., Synnaeve, G., J´ egou, H.: Augmenting convolutional networks with attention-based aggregation. arXiv preprint arXiv:2112.13692 (2021) - · arXiv
  41. [50][50] Touvron, H., Cord, M., El-Nouby, A., Verbeek, J., J'egou, H.: Three things everyone should know about vision transformers. arXiv preprint arXiv:2203.09795 (2022) - · arXiv
  42. [51][51] Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., J´ egou, H.: Going deeper with image transformers. International Conference on Computer Vision (2021) -
  43. [52][52] Touvron, H., Sablayrolles, A., Douze, M., Cord, M., J´ egou, H.: Grafit: Learning fine-grained image representations with coarse labels. International Conference on Computer Vision (2021) -
  44. [54][54] Touvron, H., Vedaldi, A., Douze, M., J´ egou, H.: Fixing the train-test resolution discrepancy: Fixefficientnet. arXiv preprint arXiv:2003.08237 (2020) - · arXiv
  45. [55][55] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017) -
  46. [57][57] Wightman, R., Touvron, H., J´ egou, H.: Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476 (2021) - · arXiv
  47. [58][58] Wu, H., Xiao, B., Codella, N.C.F., Liu, M., Dai, X., Yuan, L., Zhang, L.: Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.15808 (2021) - · arXiv
  48. [60][60] Xiao, T., Singh, M., Mintun, E., Darrell, T., Doll´ ar, P., Girshick, R.: Early convolutions help transformers see better. arXiv preprint arXiv:2106.14881 (2021) - · arXiv
  49. [61][61] You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., Hsieh, C.J.: Large batch optimization for deep learning: Training BERT in 76 minutes. In: International Conference on Learning Representations (2020) -
  50. [62][62] Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Tay, F., Feng, J., Yan, S.: Tokensto-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986 (2021) - · arXiv
  51. [63][63] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: CutMix: Regularization strategy to train strong classifiers with localizable features. arXiv preprint arXiv:1905.04899 (2019) - · arXiv
  52. [64][64] Zhang, H., Ciss´ e, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017) - · arXiv
  53. [65][65] Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. Conference on Computer Vision and Pattern Recognition (2017) -
  54. [66][66] Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A.L., Kong, T.: ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 (2021) · arXiv

Authors
Touvron, H., Cord, M., & Jégou, H.
Affiliations
Sorbonne Université (FR)
Corresponding author
Touvron, Hugo
ORCID
Venue
Lecture Notes in Computer Science
Publisher
Springer Nature Switzerland
Published
2022
Volume
13684
Pages
516-533
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
DOI
10.1007/978-3-031-20053-3_30
OpenAlex
W4224058739
Cited by
409
References
41
Citations by year
2026: 89 · 2025: 152 · 2024: 106 · 2023: 61 · 2012: 1
FWCI
28.92
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords
vision transformer, supervised exercise, data augmentation, self-supervised pre-training, BEiT, ImageNet-1K, transfer learning, semantic segmentation, image classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review