Vision Transformers Are Robust Learners
Open accessAbstract
Transformers, composed of multiple self-attention layers, hold strong promises toward a generic learning primitive applicable to different data modalities, including the recent breakthroughs in computer vision achieving state-of-the-art (SOTA) standard accuracy. What remains largely unexplored is their robustness evaluation and attribution. In this work, we study the robustness of the Vision Transformer (ViT) (Dosovitskiy et al. 2021) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We use six different diverse ImageNet datasets concerning robust classification to conduct a comprehensive performance comparison of ViT(Dosovitskiy et al. 2021) models and SOTA convolutional neural networks (CNNs), Big-Transfer (Kolesnikov et al. 2020). Through a series of six systematically designed experiments, we then present analyses that provide both quantitative andqualitative indications to explain why ViTs are indeed more robust learners. For example, with fewer parameters and similar dataset and pre-training combinations, ViT gives a top-1accuracy of 28.10% on ImageNet-A which is 4.3x higher than a comparable variant of BiT. Our analyses on image masking, Fourier spectrum sensitivity, and spread on discrete cosine energy spectrum reveal intriguing properties of ViT attributing to improved robustness. Code for reproducing our experiments is available at https://git.io/J3VO0.
AI snapshot
Written from the paper’s full text.
TL;DR
This paper benchmarks Vision Transformers (ViTs) against Big Transfer (BiT) CNNs and ResNet50V2 across six ImageNet robustness datasets, finding ViTs generally more robust. It then uses six experiments to attribute that robustness to attention, large-scale pre-training, and properties such as lower Fourier sensitivity and smoother loss landscapes.
Key findings
- ViTs outperformed comparable BiT models on most robustness benchmarks. On ImageNet-A, ViT reached 28.10% top-1 accuracy, reported as 4.3× higher than a comparable BiT variant. On ImageNet-C, ViT L-16 had mCE 45.45 versus BiT m-r101x3 at 76.7 and DeepAugment+AugMix at 58.27, though Noisy Student Training was lower at 28.3.
- ViT achieved comparable ImageNet-1k accuracy with fewer parameters: ViT B-16 used 86.86M parameters and reached 83.97% top-1, while BiT m-r101x3 used 387.93M parameters and reached 84.7%.
- Under random Cutout masking, ViT L-16 beat BiT m-r101x3 at every tested level: 83 vs. 79 at 0%, 82.3 vs. 76 at 5%, 81.4 vs. 75 at 10%, 77.9 vs. 72.4 at 20%, and 60.4 vs. 52 at 50%.
- Analyses showed ViT had lower Fourier error percentiles (at P=95, 15.1 vs. BiT’s 24.9 and ResNet-50’s 63.6), wider spread of adversarial perturbations in the DCT energy spectrum, and a smoother PGD loss landscape. Larger pre-training helped ViT B/16: ImageNet-21k versus ImageNet-1k gave ImageNet-A 21.75 vs. 8.63, ImageNet-R 41.82 vs. 28.21, and ImageNet-O AUPR 54.61 vs. 26.25.
Methods
- The study compared publicly available ViT, BiT, and ResNet50V2 models on six ImageNet-derived robustness datasets: ImageNet-C, ImageNet-P, ImageNet-R, ImageNet-A, ImageNet-O, and ImageNet-9. Most models were pre-trained on ImageNet-21k and fine-tuned on ImageNet-1k.
- Six experiments examined attention-fused architectures, pre-training data regime, random Cutout masking on 1,000 ImageNet-1k validation images at 5/10/20/50%, Fourier basis perturbations on 1,000 images, DeepFool adversarial perturbations on 1,000 images with DCT energy-spectrum analysis, and PGD loss landscapes on 100 correctly classified images with an ℓ∞ budget of 0.002.
- The paper combines quantitative benchmark comparisons with qualitative visualizations and attribution analyses.
Limitations
- The paper states that it does not investigate how regularization relates to robustness, that computational constraints led to smaller BiT and ViT variants in some experiments, and that ViT was worse than ResNet-50 on detecting vulnerable foregrounds (20.02 vs. 22.3 challenge accuracy) and not consistently better for contrast corruption.
- Evident from the text: the comparison is limited to selected public ViT, BiT, and ResNet models and six ImageNet-derived benchmarks; some baseline numbers are taken from prior reported results; and the robustness evaluation focuses on corruptions, perturbations, distribution shifts, and natural adversarial examples rather than a full adversarial defense evaluation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
2 of 6 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured
- Interpretationcitation
Using a larger dataset during pre-training may be acting as a regularizer for ViT and BiT's low sensitivity.
“Using a larger dataset during pre-training may be acting as a regularizer here (Kolesnikov et al. 2020).”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Kolesnikov et al. 2020 source
Section 2.2 (p. 3) says MixUp was not useful during BiT pre-training, likely because data were abundant. That is consistent with the tentative idea that more pre-training data can reduce the need for regularization, but the paper does not say that dataset size acts as a regularizer or explains low sensitivity in ViT and BiT. (LLM, conf=0.78)
Experimental — accuracy not yet measured
- Premisecitation
ViT needs to be pre-trained on a relatively larger dataset with strong regularization to achieve reasonable downstream performance.
“As noted in (Dosovitskiy et al. 2021;Steiner et al. 2021), because of the lesser number of inductive biases, ViT needs to be pre-trained on a relatively larger dataset (such as ImageNet-21k (Deng et al. 2009)) with strong regularization for achieving reasonable downstream performance.”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Dosovitskiy et al. 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Steiner et al. 2021 source
Section 4.3 (p. 6) finds that pre-training on the larger ImageNet-21k dataset produces better transfer results than ImageNet-1k. But the claim overstates the need for both a larger dataset and strong regularization: Section 4.1 (p. 5) reports that augmentation and regularization can partly substitute for more data, while Section 4.4 (p. 7) says regularization almost always hurts when using ImageNet-21k. (LLM, conf=0.88)
Experimental — accuracy not yet measured
- Findingown data
Different variants of ViT achieve comparable ImageNet-1k top-1 accuracy to BiT but with fewer parameters.
“It is clear that different variants of ViT are able to achieve comparable performance to BiT but with lesser parameters.”
Quote in the paperNot a substantial claim - Findingown data
ViT's treatment to domain adaptation on ImageNet-R is better than that of BiT.
“Figure 3 shows that ViT's treatment to domain adaptation is better than that of BiT.”
Quote in the paperSubstantial claim - Findingown data
ViT consistently beats BiT when square portions of input images have been randomly masked out.
“Table 7 reports that ViT is able to consistently beat BiT when square portions of the input images have been randomly masked out.”
Quote in the paperSubstantial claim
- Findingown data
The baseline ResNet50V2 loses its consistency in the high-frequency regions.
“In Figure 6, it is noticed that both ViT and BiT stay robust (have low sensitivity) to most of the regions present inside the perturbed images while the baseline ResNet50V2 loses its consistency in the highfrequency regions.”
Quote in the paperSubstantial claim - Findingown data
ViT and BiT stay robust (have low sensitivity) to most of the regions present inside the perturbed images.
“In Figure 6, it is noticed that both ViT and BiT stay robust (have low sensitivity) to most of the regions present inside the perturbed images while the baseline ResNet50V2 loses its consistency in the highfrequency regions.”
Quote in the paperSubstantial claim - Findingown data
A ViT B/16 model pre-trained on ImageNet-1k performs worse on robustness benchmarks than one pre-trained on ImageNet-21k and then fine-tuned on ImageNet-1k.
“We notice that the model pre-trained on ImageNet-1k performs worse than the one pre-trained on ImageNet-21k and then fine-tuned on ImageNet-1k.”
Quote in the paperNot a substantial claim - Findingown data
ViT models achieve significantly better performance than BiT models on six diverse ImageNet robustness benchmark datasets.
“We use 6 diverse ImageNet datasets concerning different types of robustness evaluation and conclude that ViTs achieve significantly better performance than BiTs.”
- Also stated in: § Dataset Purpose · boosted
“Notably, in these datasets ViT exhibits significantly better robustness than BiT of comparable parameter counts.”
- Also stated in: § 4.2 Role of Pre-training · boosted
“Throughout Section 3 and the rest of Section 4, we show that ViT models significantly outperform similar BiT models across six robustness benchmark datasets that we use in this work.”
Quote in the paperSubstantial claim - Also stated in: § Dataset Purpose · boosted
- Findingreasoning
ViT outperforms the combination of DeepAugment and AugMix on ImageNet-C corruptions.
“It is indeed interesting to notice that ViT is able to outperform the combination of DeepAugment and AugMix which are specifically designed to provide robustness against the corruptions found in ImageNet-C.”
Quote in the paperSubstantial claim - Findingreasoning
ViT's performance is worse than ResNet-50 on detecting if the foreground of an image is vulnerable.
“Even though we notice ViT's better performance than BiT but it is surprising to see ViT's performance being worse than ResNet-50.”
Quote in the paperNot a substantial claim - Findingown data
ViT and BiT have lower error-rate percentiles than ResNet-50 across all five percentiles reported in Table 10.
“For a more robust model, we should expect to see lower numbers across all the five different percentiles reported in Table 10 and we confirm that is indeed the case.”
Quote in the paperSubstantial claim - Findingown data
ViT significantly outperforms BiT in detecting if the foreground of an image is vulnerable.
“It appears that for this task also, ViT significantly outperforms BiT.”
Quote in the paperSubstantial claim - Findingown data
Adding some form of attention to ImageNet-1k models usually improves robustness compared to a vanilla ResNet-50.
“The results suggest that adding some form of attention is usually a good design choice especially when robustness aspects are concerned as there is almost always a consistent improvement in performance compared to that of a vanilla ResNet-50.”
Quote in the paperSubstantial claim - Findingown data
The classification loss (cross entropy) of ViT increases at a much slower rate than that of BiT as one varies the attack steps.
“Figure 8 shows that the classification loss (cross entropy) of ViT increases at a much slower rate than that of BiT as one varies the attack steps, validating our hypothesis of smoother loss landscape to input perturbations.”
Quote in the paperSubstantial claim - Findingown data
ViT significantly outperforms its CNN counterpart (BiT) and the baseline ResNet50V2 model on 6 different ImageNet datasets.
“This work provides a comprehensive robustness performance assessment of ViTs using 6 different ImageNet datasets and concludes that ViT significantly outperforms its CNN counterpart (BiT) and the baseline ResNet50V2 model.”
Quote in the paperSubstantial claim - Premisecitation
CNNs primarily exploit discriminative features from the low-frequency regions of the input data.
“Furthermore, CNNs primarily exploit discriminative features from the low-frequency regions of the input data.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ortiz-Jimenez et al. 2020 source
- Interpretationown data
The lower error-rate percentiles may help explain the better behavior of BiT and ViT in the Fourier experiment.
“This may also help explain the better behavior of BiT and ViT in this experiment.”
Quote in the paperNot a substantial claim - Interpretationown data
Attention is the strong force behind ViT's significant gain in robustness to image masking.
“This may suggest that attention indeed is the strong force behind this significant gain.”
Quote in the paperSubstantial claim - Premisenone given
ViT and BiT share similar pre-training strategies, such as using larger datasets and longer pre-training schedules.
“Since ViT and BiT share similar pre-training strategies (such as using larger datasets like ImageNet-21k (Deng et al. 2009) and JFT-300 (Sun et al. 2017), longer pre-training schedules, and so on) they are excellent candidates for our comparison purposes.”
Quote in the paperSubstantial claim - Interpretationnone given
ViT has the ability to better exploit information that is only available in a global context.
“This is primarily because ViT has the ability to better exploit information that is only available in a global context.”
Quote in the paperNot a substantial claim - Interpretationown data
ViT has a smoother loss landscape with respect to input perturbations than BiT.
“Figure 8 shows that the classification loss (cross entropy) of ViT increases at a much slower rate than that of BiT as one varies the attack steps, validating our hypothesis of smoother loss landscape to input perturbations.”
Quote in the paperSubstantial claim
- May combine two claims. 4
- The quote only partly states this claim. 3
- Classified as probably not a substantial claim. 2
May combine two claims.
Findingown dataViT performs consistently better than BiT across different corruptions on ImageNet-C, except for contrast.
“Overall, we notice that ViT performs consistently better across different corruptions except for contrast.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataViT's robustness to common perturbations on ImageNet-P is significantly better than BiT as well as AugMix.
“We find ViT's robustness to common perturbations is significantly better than BiT as well as AugMix.”
Quote in the paperSubstantial claimThe quote only partly states this claim.
Findingown dataViT demonstrates superior performance in anomaly detection on ImageNet-O than BiT.
“ViT demonstrates superior performance in anomaly detection than BiT.”
Quote in the paperSubstantial claimMay combine two claims. The quote only partly states this claim.
Findingown dataViT L-16 achieves a top-1 accuracy of 28.10% on ImageNet-A, which is 4.3 times higher than BiT-m r101x3.
“with fewer parameters and similar dataset and pre-training combinations, ViT gives a top-1 accuracy of 28.10% on ImageNet-A which is 4.3x higher than a comparable variant of BiT.”
- Also stated in: § Dataset Purpose · plain
“the top-1 accuracy of ViT L-16 is 4.3x higher than BiT-m r101x3.”
Quote in the paperSubstantial claim- Also stated in: § Dataset Purpose · plain
May combine two claims.
FindingreasoningTo attack ViT, almost the entire frequency spectrum needs to be affected, while it is less so for BiT and ResNet-50.
“It suggests that to attack ViT, (almost) the entire frequency spectrum needs to be affected, while it is less so for BiT and ResNet-50.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationSmall adversarial perturbations can change the decision boundary of neural networks, especially CNNs.
“In (Ortiz-Jimenez et al. 2020), it is shown that small adversarial perturbations can change the decision boundary of neural networks (especially CNNs) and that adversarial training (Madry et al. 2018) exploits this sensitivity to induce robustness.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ortiz-Jimenez et al. 2020 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationAdversarial training exploits the sensitivity of neural networks to adversarial perturbations to induce robustness.
“In (Ortiz-Jimenez et al. 2020), it is shown that small adversarial perturbations can change the decision boundary of neural networks (especially CNNs) and that adversarial training (Madry et al. 2018) exploits this sensitivity to induce robustness.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Madry et al. 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
The quote only partly states this claim.
Interpretationown dataAttention is one of the attributes for the superior robustness performance ViTs show on three robustness benchmark datasets.
“We confirm this in Table 8 (last row).”
Quote in the paperSubstantial claim- Rests on an assumption: The attention mechanism helps a model learn better contextual dependencies.
The paper relies on this without support.
The attention mechanism helps a model learn better contextual dependencies.
“Since the attention mechanism helps a model learn better contextual dependencies we hypothesize that this is one of the attributes for the superior performance ViTs show on three robustness benchmark datasets.”
The paper relies on this without support.
ViT can utilize global information.
“Randomness is desirable here because ViT can utilize global information.”
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 3× · see citing works
“[10] studied improved robustness of ViTs over ResNets [26] against adversarial and natural adversarial examples as well as common corruptions.”
BackgroundMethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2× · see citing works
“The final score is normalized by the mCE of AlexNet [39].”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1× · see citing works
“We use Adam [59] as optimizer here to implement PGD attack (see Section 4.6 for implementation details) with a learning rate of 1e-3.”
MethodInfluentialJakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2× · see citing works
“Since ViT and BiT share similar pre-training strategies (such as using larger datasets like ImageNet-21k [20] and JFT-300 [35], longer pre-training schedules, and so on) they are excellent candidates for our comparison purposes.”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 1× · see citing works
“Inspired by BERT [29], a class token is prepended to the image patches and it flows through the entirety of ViT.”
MethodImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 2× · see citing works
“In this work, our baseline is a ResNet50V2 model [30] pre-trained on the ImageNet-1k dataset [31] except for a few results where we consider ResNet-50 [26] 2 .”
MethodSwin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · arXiv · Cited by 35,608 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 1× · see citing works
“2021) and Swin Transformers (Liu et al. 2021) are also two recent models that make use of Transformers for image recognition problems.”
Background(2014) · Cited by 33,419 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 1× · see citing works
“At its core, BiT networks are scaled-up versions of ResNets with Group Normalization [32] and Weight Standardization [33] layers added in place of Batch Normalization [34].”
MethodGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · cited 6× · see citing works
“Although Transformers have been used in prior works [5, 6] it was only until 2020, the performance of Transformers were on par with the SOTA CNNs on standard image recognition tasks [7, 8, 1].”
BackgroundMethodInfluentialEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · cited 2× · see citing works
“Although Transformers have been used in prior works [5, 6] it was only until 2020, the performance of Transformers were on par with the SOTA CNNs on standard image recognition tasks [7, 8, 1].”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2× · see citing works
“Central to ViT’s model design is self-attention [4].”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1× · see citing works
“DeIT [8] introduces a novel distillation strategy [21] to learn a student Transformers-based network from a well-performing teacher network based on RegNets [22].”
MethodPhoto-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696 · see citing works
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741 · cited 1× · see citing works
“In this work, our baseline is a ResNet50V2 model [30] pre-trained on the ImageNet-1k dataset [31] except for a few results where we consider ResNet-50 [26] 2 .”
MethodShi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.207 · Cited by 7,570 · see citing works
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244 · cited 1× · see citing works
“For the Transformer-based backbone, it follows a deep-narrow network topology inspired by [24].”
BackgroundTowards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602 · cited 2× · see citing works
“In [55], it is shown that small adversarial perturbations can change the decision boundary of neural networks (especially CNNs) and that adversarial training [56] exploits this sensitivity to induce robustness.”
BackgroundMethodEmerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1× · see citing works
“DINO [48] shows when ViT is trained with self-supervised objective on a larger data corpus, its self-attention span is made even more pronounced than their self-supervised counterparts.”
BackgroundDeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · cited 1× · see citing works
“Following [55], we conduct the following experiment on 1000 randomly sampled images from the ImageNet-1k validation set with ResNet-50, BiT-m r50x3 , and ViT B-16 8 : • Generate small adversarial perturbations ( δ ) with DeepFool [51] with a step size of 50 9 .”
MethodESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., & Loy, C. C. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11021-5_5 · Cited by 5,085 · cited 1× · see citing works
“To the best of our knowledge, [16] first explored the use of Transformers [3] for the task of image super-resolution [17–19] which essentially belongs to the category of image generation.”
BackgroundOn Assessing ML Model Robustness: A Methodological Framework (Academic Track)
Awadid, A., & Robert, B. (2025) · OASIcs : OpenAccess Series in Informatics · doi:10.4230/oasics.saia.2024.1 · Cited by 4,723 · see citing works
Wu, Y., & He, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_1 · Cited by 3,601 · see citing works
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · cited 1× · see citing works
“The FFN is comprised of two linear layers with a GELU non-linearity [28] in between them.”
BackgroundLi, X., Wang, W., Hu, X., & Yang, J. J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00060 · Cited by 3,416 · see citing works
Randaugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00359 · Cited by 3,223 · see citing works
Improved regularization of convolutional neural networks with cutout
DeVries, T., & Taylor, G. W. (2017) · arXiv · doi:10.48550/arxiv.1708.04552 · Cited by 2,776 · cited 1× · see citing works
“• Apply Cutout [49] at four different levels: {5,10,20,50}% and calculate the mean top-1 accuracy scores for each of the levels with BiT ( m-r101x3 ) and ViT ( L-16) 7 .”
BackgroundRevisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484 · cited 1× · see citing works
“Since ViT and BiT share similar pre-training strategies (such as using larger datasets like ImageNet-21k [20] and JFT-300 [35], longer pre-training schedules, and so on) they are excellent candidates for our comparison purposes.”
BackgroundTokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415 · cited 2× · see citing works
“T2T-ViT [23] proposes a novel tokenization method enabling the network to have more access to local structures of the images.”
MethodTokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E., Feng, J., & Yan, S. (2021) · arXiv · doi:10.48550/arxiv.2101.11986 · Cited by 2,415 · see citing works
Self-Training With Noisy Student Improves ImageNet Classification
Xie, Q., Luong, M.-T., Hovy, E., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01070 · Cited by 2,393 · cited 3× · see citing works
“It should also be noted that Noisy Student Training [38] incorporates various factors during training such as an iterative training procedure, strong data augmentation transformations from RandAugment for noise injection, test-time augmentation, and so on.”
BackgroundMethodDeep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104 · cited 1× · see citing works
“We include two additional models/methods in Table 3 because of the following: (a) Noisy Student Training [38] uses external data and training choices (such as using RandAugment [40], Stochastic Depth [41], etc.”
MethodDesigning Network Design Spaces
Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., & Dollar, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01044 · Cited by 2,084 · see citing works
Wide Residual NetworksPreprint
Zagoruyko, S., & Komodakis, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07146 · Cited by 1,850 · cited 1× · see citing works
“For the Transformer-based backbone, it follows a deep-narrow network topology inspired by [24].”
BackgroundEfficientNetV2: Smaller Models and Faster TrainingPreprint
Tan, M., & Le, Q. V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.00298 · Cited by 1,473 · cited 1× · see citing works
“These models include EfficientNetV2 (Tan and Le 2021) with Global Context (GC) blocks (Cao et al.”
MethodBenchmarking Neural Network Robustness to Common Corruptions and Perturbations
Hendrycks, D., & Dietterich, T. (2019) · arXiv · doi:10.48550/arxiv.1903.12261 · Cited by 1,335 · cited 9× · see citing works
“In [13], mean corruption error (mCE) is used to quantify the robustness factors of a model on the ImageNet-C dataset.”
BackgroundMethodInfluentialThe Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00823 · Cited by 1,250 · cited 3× · see citing works
“The ImageNet-R dataset [14] contains images labelled with ImageNet labels by collecting renditions of ImageNet classes.”
BackgroundMethodHendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01501 · Cited by 1,090 · cited 10× · see citing works
“Attention has been shown to be an important element for vision networks to achieve better empirical robustness [9].”
BackgroundMethodInfluentialParmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., Ku, A., & Tran, D. (2018) · arXiv · doi:10.48550/arxiv.1802.05751 · Cited by 1,085 · see citing works
Big Transfer (BiT): General Visual Representation Learning
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58558-7_29 · Cited by 1,039 · cited 5× · see citing works
“As pointed out in [43, 2, 44], a longer pre-training schedule on a larger pre-training dataset is transferring effectively to downstream tasks including few-shot learning.”
BackgroundMethodInfluential(2020) · International Conference on Machine Learning · Cited by 1,012 · see citing works
Quantifying Attention Flow in Transformers
Abnar, S., & Zuidema, W. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.385 · Cited by 995 · see citing works
Do Vision Transformers See Like Convolutional Neural Networks?
Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., & Dosovitskiy, A. (2021) · arXiv · doi:10.48550/arxiv.2108.08810 · Cited by 663 · see citing works
Gather-Excite: Exploiting Feature Context in Convolutional Neural NetworksPreprint
Hu, J., Shen, L., Albanie, S., Sun, G., & Vedaldi, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12348 · Cited by 574 · cited 1× · see citing works
“2020), several ResNet variants with Gather-Excite (GE) blocks (Hu et al. 2018) and Selective Kernels (SK) (Li et al.”
BackgroundSharpness-Aware Minimization for Efficiently Improving Generalization
Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. (2020) · arXiv · doi:10.48550/arxiv.2010.01412 · Cited by 521 · cited 1× · see citing works
“(Foret et al. 2021) and stronger regularization techniques to compensate the absence of favored inductive priors (Chen, Hsieh, and Gong 2021; Steiner et al.”
Background(2021) · International Conference on Learning Representations · Cited by 449 · see citing works
Understanding Robustness of Transformers for Image Classification
Bhojanapalli, S., Chakrabarti, A., Glasner, D., Li, D., Unterthiner, T., & Veit, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01007 · Cited by 440 · cited 2× · see citing works
“Various recent works have opened up the investigation on evaluating the robustness of ViTs (Bhojanapalli et al. 2021; Shao et al. 2021; Mahmood, Mahmood, and Van Dijk 2021) but with a relatively limited scope.”
BackgroundUnlabeled Data Improves Adversarial RobustnessPreprint
Carmon, Y., Raghunathan, A., Schmidt, L., Liang, P., & Duchi, J. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.13736 · Cited by 440 · cited 1× · see citing works
“This indicates that the use of Transformers can be orthogonal to the known techniques to improve the robustness of vision models [57, 58, 38].”
MethodA Fourier Perspective on Model Robustness in Computer VisionPreprint
Yin, D., Lopes, R. G., Shlens, J., Cubuk, E. D., & Gilmer, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.08988 · Cited by 337 · cited 2× · see citing works
“…about vision models is that they can easily pick up the spurious correlations present inside input data that may be imperceptible and unintuitive to humans [52–54 To measure how ViT holds up with this end of the bargain, we conduct a Fourier analysis [50] of ViT, BiT, and our baseline ResNet-50.”
BackgroundMethodNightmare at test time: robust learning by feature deletion
Globerson, A., & Roweis, S. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143889 · Cited by 309 · see citing works
Revisiting Unreasonable Effectiveness of Data in Deep Learning EraPreprint
Sun, C., Shrivastava, A., Singh, S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02968 · Cited by 300 · cited 1× · see citing works
“Since ViT and BiT share similar pre-training strategies (such as using larger datasets like ImageNet-21k [20] and JFT-300 [35], longer pre-training schedules, and so on) they are excellent candidates for our comparison purposes.”
BackgroundHow to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., & Beyer, L. (2021) · arXiv · doi:10.48550/arxiv.2106.10270 · Cited by 292 · cited 5× · see citing works
“2021) and stronger regularization techniques to compensate the absence of favored inductive priors (Chen, Hsieh, and Gong 2021; Steiner et al. 2021).”
BackgroundInfluentialSelf-training with Noisy Student improves ImageNet classificationPreprint
Xie, Q., Luong, M.-T., Hovy, E. H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04252 · Cited by 237 · cited 3× · see citing works
“It should also be noted that Noisy Student Training [38] incorporates various factors during training such as an iterative training procedure, strong data augmentation transformations from RandAugment for noise injection, test-time augmentation, and so on.”
BackgroundMethodOn the Robustness of Vision Transformers to Adversarial Examples
Mahmood, K., Mahmood, R., & van Dijk, M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00774 · Cited by 217 · cited 2× · see citing works
“[12] studied adversarial robustness of ViTs through various white-box, black-box and transfer attacks and found that model ensembling can achieve unprecedented robustness without sacrificing clean accuracy against a black-box adversary.”
BackgroundMeasuring the tendency of CNNs to Learn Surface Statistical RegularitiesPreprint
Jo, J., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.11561 · Cited by 201 · see citing works
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182 · see citing works
Cao, Y., Xu, J., Lin, S. C.-F., Wei, F., & Hu, H. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3047209 · Cited by 162 · cited 1× · see citing works
“These models include EfficientNetV2 (Tan and Le 2021) with Global Context (GC) blocks (Cao et al. 2020), several ResNet variants with Gather-Excite (GE) blocks (Hu et al.”
MethodNoise or Signal: The Role of Image Backgrounds in Object Recognition
Xiao, K., Engstrom, L., Ilyas, A., & Madry, A. (2020) · arXiv · doi:10.48550/arxiv.2006.09994 · Cited by 161 · cited 5× · see citing works
“Through a set of carefully designed experiments, we first verify the enhanced robustness of ViTs to common robustness benchmark datasets [13, 14, 9, 15].”
BackgroundMethodInfluentialAll Tokens Matter: Token Labeling for Training Better Vision Transformers
Jiang, Z., Hou, Q., Yuan, L., Zhou, D., Shi, Y., Jin, X., Wang, A., & Feng, J. (2021) · arXiv · doi:10.48550/arxiv.2104.10858 · Cited by 158 · see citing works
When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
Chen, X., Hsieh, C.-J., & Gong, B. (2021) · arXiv · doi:10.48550/arxiv.2106.01548 · Cited by 128 · see citing works
Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
Qiao, S., Wang, H., Liu, C., Shen, W., & Yuille, A. (2019) · arXiv · doi:10.48550/arxiv.1903.10520 · Cited by 126 · see citing works
Are Convolutional Neural Networks or Transformers more like human vision?
Tuli, S., Dasgupta, I., Grant, E., & Griffiths, T. L. (2021) · arXiv · doi:10.48550/arxiv.2105.07197 · Cited by 119 · see citing works
Instance adaptive adversarial training: Improved accuracy tradeoffs in neural netsPreprint
Balaji, Y., Goldstein, T., & Hoffman, J. (2019) · arXiv.org · doi:10.48550/arxiv.1910.08051 · Cited by 116 · cited 1× · see citing works
“This indicates that the use of Transformers can be orthogonal to the known techniques to improve the robustness of vision models [57, 58, 38].”
MethodSelfie: Self-supervised Pretraining for Image EmbeddingPreprint
Trinh, T. H., Luong, M.-T., & Le, Q. V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02940 · Cited by 92 · cited 1× · see citing works
“Although Transformers have been used in prior works [5, 6] it was only until 2020, the performance of Transformers were on par with the SOTA CNNs on standard image recognition tasks [7, 8, 1].”
MethodOn the Adversarial Robustness of Vision Transformers
Shao, R., Shi, Z., Yi, J., Chen, P.-Y., & Hsieh, C.-J. (2021) · arXiv · doi:10.48550/arxiv.2103.15670 · Cited by 68 · see citing works
Self-citationOn the Adversarial Robustness of Visual TransformersPreprint
Shao, R., Shi, Z., Yi, J., Chen, P., & Hsieh, C. (2021) · arXiv (Cornell University) · Cited by 50 · see citing works
Self-citationAttention in natural scenes: contrast affects rapid visual processing and fixations alike
Hart, B. M. ’., Schmidt, H. C. E. F., Klein-Harmeyer, I., & Wolfgang Einhäuser, A. (2013) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2013.0067 · Cited by 38 · cited 1× · see citing works
“Based on our findings, contrast can be an effective but unexplored approach to studying ViT’s robustness, similar to the study of human’s vision performance (Hart et al. 2013; Tuli et al. 2021).”
BackgroundToken Labeling: Training a 85.4% Top-1 Accuracy Vision Transformer with 56M Parameters on ImageNet
Jiang, Z., Hou, Q., Yuan, L., Zhou, D., Jin, X., Wang, A., & Feng, J. (2021) · Cited by 32 · see citing works
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution GeneralizationPreprint
Hendrycks, D., Basart, S., Mu, N., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16241 · Cited by 19 · see citing works
Hold me tight! Influence of discriminative features on deep network boundaries
Ortiz-Jimenez, G., Modas, A., Moosavi-Dezfooli, S.-M., & Frossard, P. (2020) · arXiv · doi:10.48550/arxiv.2002.06349 · Cited by 15 · cited 2× · see citing works
“In (Ortiz-Jimenez et al. 2020), it is shown that small adversarial perturbations can change the decision boundary of neural networks (especially CNNs) and that adversarial training (Madry et al.”
BackgroundMethodUnderstanding Robustness of Transformers for Image ClassificationPreprint
Bhojanapalli, S., Chakrabarti, A., Gläsner, D., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.14586 · Cited by 2 · see citing works
Global Context NetworksPreprint
Cao, Y., Xu, J., Lin, S. C.-F., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.13375 · Cited by 1 · see citing works
On the Robustness of Vision Transformers to Adversarial ExamplesPreprint
Mahmood, K., Mahmood, R., & van Dijk, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.02610 · Cited by 1 · see citing works
Srivastava (2014) · doi:10.5555/2627435.2670313
Flagged first, then by citations
20 of the 99 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- [20]20. Hendrycks*, D.; Mu*, N.; Cubuk, E. D.; Zoph, B.; Gilmer, J.; and Lakshminarayanan, B. 2020. AugMix: A Simple Method to Improve Robustness and Uncertainty under Data Shift. In International Conference on Learning Representations.
- [31]31. Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. Im- ageNet Classification with Deep Convolutional Neural Net- works. In Advances in Neural Information Processing Sys- tems, volume 25.
- [49]49. PMLR.
Cited by
Sorted by citations
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs
Ding, X., Zhang, X., Han, J., & Ding, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01166 · Cited by 1,511 · see citing works
“They provided empirical results to demonstrate that, MHSA is more flexible [50], capable (less inductive bias) [20], more robust to distortions [66, 98], or able to model long-range dependencies [69, 90].”
BackgroundTransformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,412 · see citing works
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
“Although that recent attempts [99], [182], [289], [290] study and evaluate how the Transformer components/sub-layers con-”
BackgroundResultHoyer, L., Dai, D., & Gool, L. V. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00969 · Cited by 663 · see citing works
“Based on recent findings [3, 51, 53] and an architecture comparison for UDA, which we will present in Sec.”
BackgroundArtificial intelligence in histopathology: enhancing cancer research and clinical oncology
Shmatko, A., Ghaffari Laleh, N., Gerstung, M., & Kather, J. N. (2022) · Nature Cancer · doi:10.1038/s43018-022-00436-4 · Cited by 446 · see citing works
“Shortly after their emergence, trans-formers outperformed CNNs in many computer vision tasks and proved more robust against adversarial attacks 120,121 .”
BackgroundStyTr 2 : Image Style Transfer with Transformers
Deng, Y., Tang, F., Dong, W., Ma, C., Pan, X., Wang, L., & Xu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01104 · Cited by 442 · see citing works
“Second, the transformer architecture models relationships in input shapes [41], and different layers extract similar structural information [46] (see Fig.”
BackgroundIntriguing Properties of Vision Transformers
Naseer, M., Ranasinghe, K., Khan, S., Hayat, M., Khan, F. S., & Yang, M.-H. (2021) · arXiv · doi:10.48550/arxiv.2105.10497 · Cited by 431 · see citing works
“A concurrent work from Paul and Chen [18] also develops similar insights on robustness of ViTs but with a somewhat different set of experiments.”
BackgroundHow Do Vision Transformers Work?
Park, N., & Kim, S. (2022) · arXiv · doi:10.48550/arxiv.2202.06709 · Cited by 356 · see citing works
“2 ViTs are robust against data corruptions, image occlusions (Naseer et al., 2021), and adversarial attacks (Shao et al., 2021; Bhojanapalli et al., 2021; Paul & Chen, 2022; Mao et al., 2021).”
BackgroundTowards Robust Vision Transformer
Mao, X., Qi, G., Chen, Y., Li, X., Duan, R., Ye, S., He, Y., & Xue, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01173 · Cited by 224 · see citing works
“Follow up works [2, 29] extend the robustness study on ViTs to much common image corruption and distribution shift, and indicate ViTs are more robust learners.”
BackgroundRethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
Qu, L., Zhou, Y., Liang, P. P., Xia, Y., Wang, F., Adeli, E., Fei-Fei, L., & Rubin, D. L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00982 · Cited by 220 · see citing works
“Among several particularly relevant findings are that ViTs are highly robust to severe occlusions, perturbations, domain shifts [3, 48], as well as synthetic and natural adversarial examples [44,51].”
BackgroundSelva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapés, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3243465 · Cited by 177 · see citing works
“better able to form abstract semantic representations [197], probably due to their ability to leverage non-local contexts [198].”
BackgroundDeep learning for detecting macroplastic litter in water bodies: A review
Jia, T., Kapelan, Z. S., de Vries, R., Vriend, P., Peereboom, E. C., Okkerman, I., & Taormina, R. (2023) · Water Research · doi:10.1016/j.watres.2023.119632 · Cited by 111 · see citing works
Towards Transferable Adversarial Attacks on Vision Transformers
Wei, Z., Chen, J., Goldblum, M., Wu, Z., Goldstein, T., & Jiang, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20169 · Cited by 124 · see citing works
“Apart from the above works, there are also some other papers (Tang et al. 2021; Paul and Chen 2021; Shi and Han 2021) that discuss the robustness of ViTs.”
Do Vision Transformers See Like Convolutional Neural Networks?
Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., & Dosovitskiy, A. (2021) · arXiv · doi:10.48550/arxiv.2108.08810 · Cited by 663 · see citing works
“Since its development, there has also been very recent work analyzing aspects of ViT, particularly robustness [3, 31, 28] and effects of self-supervision [5, 7].”
BackgroundDimitrovski, I., Kitanovski, I., Kocev, D., & Simidjievski, N. (2023) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2023.01.014 · Cited by 102 · see citing works
Delving Deep into the Generalization of Vision Transformers under Distribution Shifts
Zhang, C., Zhang, M., Zhang, S., Jin, D., Zhou, Q., Cai, Z., Zhao, H., Liu, X., & Liu, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00713 · Cited by 107 · see citing works
“Despite the encouraging performance achieved on standard benchmarks and several properties revealed in recent works [1, 5, 23, 24], the generalization ability of Vision Transformers (ViTs) is still less understood.”
BackgroundExploring the Limits of Out-of-Distribution Detection
Fort, S., Ren, J., & Lakshminarayanan, B. (2021) · arXiv · doi:10.48550/arxiv.2106.03004 · Cited by 142 · see citing works
“[2021] show that ViT is more robust to distribution shift and natural adversarial examples [Paul and Chen, 2021], and Mao et al.”
BackgroundA Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking
Liu, C., Dong, Y., Xiang, W., Yang, X. H., Su, H., Zhu, J., Chen, Y., He, Y., Xue, H., & Zheng, S. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02196-3 · Cited by 117 · see citing works
“This is due to the fact that ViT has a smoother loss landscape under input perturbations as discussed in Paul et al. (2022).”
Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification
Xiao, J., Bai, Y., Yuille, A., & Zhou, Z. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00358 · Cited by 104 · see citing works
“In computer vision, there is a heated debate between the adoption of ViTs and CNNs, in terms of performance [63, 110, 6, 97, 91, 25], robustness [4, 69, 105, 109], data requirement [26, 83, 89], computational efficiency [74].”
BackgroundCui, L., Jing, X., Wang, Y., Huan, Y., Xu, Y., & Zhang, Q. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3225150 · Cited by 76 · see citing works
“…can compensate for the abovementioned shortcomings of conventionalCNNsthatfocusonlyonlocalreceptivefieldswith-outconsideringglobalfeatures[18],[19],[20],[21],allowing eachpixeltocontainglobalcorrelationsandthusimproving generalizationabilityandinterferencerobustness[22],[23], [24],[25],[26].”
BackgroundKolides, A., Nawaz, A., Rathor, A., Beeman, D., Hashmi, M., Fatima, S., Berdik, D., Al‐Ayyoub, M., & Jararweh, Y. (2023) · Simulation Modelling Practice and Theory · doi:10.1016/j.simpat.2023.102754 · Cited by 73 · see citing works
A Simple Fix to Mahalanobis Distance for Improving Near-OOD DetectionPreprint
Ren, J., Fort, S., Liu, J. Z., Roy, A. G., Padhy, S., & Lakshminarayanan, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09022 · Cited by 115 · see citing works
“It has been shown that the high-quality features learnt during this pre-training stage can be very useful in boosting the performance of the downstream task [7; 18; 5].”
BackgroundA transformer-based framework for automatic COVID19 diagnosis in chest CTs
Zhang, L., & Wen, Y. (2021) · IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw54120.2021.00063 · Cited by 66 · see citing works
“However, few attempts have been made to adopt the Vison transformer architectures [6, 18], though the latest studies [3, 21] have shown its robustness and superior performance in image classifi-cation compared to the CNN based architectures.”
BackgroundCrimeNet: Neural Structured Learning using Vision Transformer for violence detection
Rendón-Segador, F. J., Álvarez-García, J. A., González, J. L. S., et al. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.01.048 · Cited by 63 · see citing works
PVT-SAR: An Arbitrarily Oriented SAR Ship Detector With Pyramid Vision Transformer
Zhou, Y., Jiang, X., Xu, G., Yang, X., Liu, X., & Li, Z. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3221784 · Cited by 66 · see citing works
“Transformers have dynamic modeling and powerful representation capabilities, which also show robustness to occlusion and noise [37].”
BackgroundTransformers in Self-Supervised Monocular Depth Estimation with Unknown Camera Intrinsics
Varma, A., Chawla, H., Zonooz, B., & Arani, E. (2022) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0010884000003124 · Cited by 63 · see citing works
“Therefore, MT-SfMLearner, where both depth and ego-motion and learned with transformers, provides the highest robustness and generalizability, in line with the studies on image classification (Paul and Chen, 2021; Bhojanapalli et al., 2021).”
MethodFrequency Representation Integration for Camouflaged Object Detection
Xie, C., Xia, C., Yu, T., et al. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611773 · Cited by 58 · see citing works
Leveraging Local and Global Cues for Visual Tracking via Parallel Interaction Network
Zheng, Y., Zhong, B., Liang, Q., Tang, Z., Ji, R., & Li, X. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3212987 · Cited by 59 · see citing works
“For example, compared with convolutions, prior works believe the multi-head attention mechanism is more robust to distractors [11], and can take advantage of long range dependencies (with larger receptive fields) [12] and so on.”
ResultAre Vision Transformers Robust to Patch Perturbations?
Gu, J., Tresp, V., & Qin, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19775-8_24 · Cited by 69 · see citing works
“On the other hand, [5, 31, 35] demonstrate that ViT achieves higher adversarial robustness than CNNs under adversarial attacks.”
BackgroundInfluentialHGFormer: Hierarchical Grouping Transformer for Domain Generalized Semantic Segmentation
Ding, J. J., Xue, N., Xia, G.-S., Schiele, B., & Dai, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01479 · Cited by 61 · see citing works
“Recently, Vision Transformers have been shown to be significantly more robust than traditional CNNs in the outof-distribution generalization [21, 25, 42, 58, 60, 70].”
BackgroundGarrucho, L., Kushibar, K., Jouide, S., Díaz, O., Igual, L., & Lekadir, K. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102386 · Cited by 56 · see citing works
“The OOD robustness of Transformer architectures has been analysed in recent publications since Transformers became more popular in Computer Vision tasks (Fort et al., 2021; Paul and Chen, 2021; Bai et al., 2021), mainly since Visual Transformers (ViT) were introduced (Dosovitskiy et al.”
Wu, Y., Li, S., Zhang, J., Li, Y., Li, Y., & Zhang, Y. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.105166 · Cited by 51 · see citing works
CLAD: A realistic Continual Learning benchmark for Autonomous Driving
Verwimp, E., Yang, K., Parisot, S., Hong, L., McDonagh, S., Pérez-Pellitero, E., Lange, M. D., & Tuytelaars, T. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.02.001 · Cited by 53 · see citing works
RobustART: Benchmarking Robustness on Architecture Design and Training TechniquesPreprint
Tang, S., Gong, R., Wang, Y., Liu, A., Wang, J., Chen, X., Yu, F., Liu, X., Song, D., Yuille, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05211 · Cited by 71 · see citing works
“Some of the existing studies [52], [53], [54], [84] have discussed the robustness of some architectures (e.”
BackgroundLü, M., Christensen, C. N., Weber, J. M., Konno, T., Läubli, N. F., Scherer, K. M., Avezov, E., Lió, P., Lapkin, A. A., Schierle, G. S. K., et al. (2023) · Nature Methods · doi:10.1038/s41592-023-01815-0 · Cited by 44 · see citing works
“In contrast to state-of-the-art CNN models commonly used for image segmentation, ERnet is constructed in a Vision Transformer architecture that outperforms CNNs in terms of image-classification accuracy and requires much smaller computational resources 19,32 .”
MethodMoayeri, M., Pope, P., Balaji, Y., & Feizi, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01850 · Cited by 49 · see citing works
“While there are mixed findings on adversarial robustness [4, 46], there is agreement that ViTs have stronger out-of-distribution generalization, likely due to self attention [5, 39].”
BackgroundKyathanahally, S. P., Hardeman, T., Reyes, M., Merz, E., Bulas, T., Brun, P., Pomati, F., & Baity‐Jesi, M. (2022) · Scientific Reports · doi:10.1038/s41598-022-21910-0 · Cited by 46 · see citing works
An Impartial Take to the CNN vs Transformer Robustness Contest
Pinto, F., Torr, P. H. S., & Dokania, P. K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19778-9_27 · Cited by 42 · see citing works
Self-attention in vision transformers performs perceptual grouping, not attention
Mehrani, P., & Tsotsos, J. K. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1178450 · Cited by 43 · see citing works
Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs
Benz, P., Ham, S., Zhang, C., Karjauv, A., & Kweon, I. S. (2021) · British Machine Vision Conference (BMVC) · doi:10.5244/c.35.68 · Cited by 59 · see citing works
“Specifically, [2, 4, 44, 49, 54, 61] concurrently compare the robustness of transformers to CNNs and independently derive conclusions resembling each other.”
BackgroundGuan, R., Yao, S., Zhu, X., Man, K. L., Lim, E. G., Smith, J. S., Yue, Y., & Yue, Y. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10422325 · Cited by 35 · see citing works
“ViTs are more robust than CNNs on adversarial attacks [18][19][20], vision object occlusions [21] and data corrup-tions [22].”
BackgroundTransformers in Industry 4.0, Industry 5.0, and Society 5.0: Roles and ChallengesPreprint
Rane, N. L. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4609915 · Cited by 33 · see citing works
Hossain, M. Z., Zaman, F. U., & Islam, M. R. (2023) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit60459.2023.10440990 · Cited by 36 · see citing works
“Pre-trained vision transformer ViT-S16 [17] was also used for identifying AI-generated images.”
MethodAre transformer-based models more robust than CNN-based models?
Liu, Z., Qian, S., Xia, C., & Wang, C. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.12.045 · Cited by 32 · see citing works
Robustifying Token Attention for Vision Transformers
Guo, Y. X., Stutz, D., & Schiele, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01610 · Cited by 35 · see citing works
“In this paper, we examine a key component of ViTs, i.e., the self-attention mechanism, to understand these performance drops.”
BackgroundInfluentialMa, W., Li, Y., Jia, X., & Xu, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00427 · Cited by 40 · see citing works
“There exist some studies demonstrating the superior adversarial robustness of ViTs over CNNs [3, 29, 30].”
BackgroundEfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones
Wang, Y., Yang, Y., Lü, R., Liu, T., Zhong, Z., Song, S., & Huang, G. M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00538 · Cited by 33 · see citing works
“Towards this direction, several existing works also explore decomposing the inputs of models in the frequency domain [52, 53, 54] in order to understand or improve the robustness of the networks.”
BackgroundAI Maintenance: A Robustness Perspective
Chen, P.-Y., & Das, P. (2023) · Computer · doi:10.1109/mc.2022.3218005 · Cited by 29 · see citing works
Self-citationFight Detection from Still Images in the Wild
Aktı, Ş., Ofli, F., Imran, M., et al. (2022) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw54805.2022.00061 · Cited by 29 · see citing works
“In addition, it is shown in [25] that ViT is robust to blurry images which is a common case for the fight images due to motion.”
Li, P., Wang, Y., Si, T., Ullah, K., Han, W., & Wang, L. (2023) · International Journal of Digital Earth · doi:10.1080/17538947.2023.2229794 · Cited by 30 · see citing works
A Light Recipe to Train Robust Vision Transformers
Debenedetti, E., Sehwag, V., & Mittal, P. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00024 · Cited by 36 · see citing works
“On the one hand, many recent works [2, 19, 20] agree on the fact that ViTs are more robust than CNNs when it comes to out-of-distribution samples.”
BackgroundCausal Transportability for Visual Recognition
Mao, C., Xia, K., Wang, J., Wang, H., Yang, J., Bareinboim, E., & Vondrick, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00737 · Cited by 28 · see citing works
“Recently, the attention operation is also shown to be effective for improving robustness [21,39,41].”
BackgroundYang, Z., Zhang, Y., Xu, K., Sun, J., Wu, Y., & Zhou, M. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107148 · Cited by 21 · see citing works
Liu, H., Yao, M., Xiao, X., Zheng, B., & Cui, H. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3343109 · Cited by 32 · see citing works
“The emerging transformers have shown superiority over previous CNNs in UDA tasks [68], [69].”
BackgroundSaednia, K., Tran, W. T., & Naini, A. S. (2023) · Medical Physics · doi:10.1002/mp.16574 · Cited by 25 · see citing works
Assaying Out-Of-Distribution Generalization in Transfer LearningPreprint
Wenzel, F., Dittadi, A., Gehler, P., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.09239 · Cited by 24 · see citing works
“(3) Exhaustive comparisons attempted to disentangle intrinsic architectural robustness from specific training schedules [9, 16, 17, 67, 72, 78], addressing underspecification [24] with inductive biases.”
BackgroundA Volumetric Transformer for Accurate 3D Tumor SegmentationPreprint
Peiris, H., Hayat, M., Chen, Z., et al. (2021) · arXiv (Cornell University) · Cited by 24 · see citing works
A Study on the Generality of Neural Network Structures for Monocular Depth Estimation
Bae, J., Hwang, K., & Im, S. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3332407 · Cited by 26 · see citing works
“Moreover, many works [27], [36], [37], [38] reveal that Transformers have a strong shape bias notwithstanding the lack of a spatial locality, compared to CNNs. Due to its strong shape bias, Transformer is considered more robust than a CNN and more similar to human cognitive processes [39].”
BackgroundReliability in Semantic Segmentation: Are we on the Right Track?
de Jorge, P., Volpi, R., Torr, P. H. S., & Rogez, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00693 · Cited by 27 · see citing works
“Following the rise of transformer architectures in computer vision [4,15,29,50], several studies have compared recent self-attention and CNN-based classification models in terms of robustness [2, 3, 30, 32, 36, 43] and predictive uncertainty [33, 44].”
BackgroundMethodEspejo-García, B., Panoutsopoulos, H., Anastasiou, E., Rodríguez-Rigueiro, F. J., & Fountas, S. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.108055 · Cited by 23 · see citing works
Teaching Matters: Investigating the Role of Supervision in Vision Transformers
Walmer, M., Suri, S., Gupta, K., & Shrivastava, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00723 · Cited by 26 · see citing works
“A few works [45, 55, 75] have analyzed the robustness of ViT features against corruptions, perturbations, distribution shifts, and adversarial examples while also analyzing the role of selfattention for robustness.”
BackgroundVision transformers: The next frontier for deep learning-based ophthalmic image analysis
Wu, J., Köseoğlu, N. D., Jones, C. K., et al. (2023) · Saudi Journal of Ophthalmology · doi:10.4103/sjopt.sjopt_91_23 · Cited by 22 · see citing works
Trade-off between Robustness and Accuracy of Vision Transformers
Li, Y., & Xu, C. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00730 · Cited by 23 · see citing works
“Although empirical analyses have demonstrated that ViTs demonstrates robustness against various kinds of perturbations [1, 33, 37], there are only a limited number of works [9,19,34,60] focus on improving the robustness.”
BackgroundPelosin, F., Jha, S., Andrea Torsello, A., Raducanu, B., & van de Weijer, J. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00427 · Cited by 24 · see citing works
“The conclusion remains in line with previous works [35].”
ResultAutomated Scoring of Figural Tests of Creativity with Computer Vision
Acar, S., Organisciak, P., & Dumas, D. (2024) · The Journal of Creative Behavior · doi:10.1002/jocb.677 · Cited by 26 · see citing works
Improving Robustness of Vision Transformers by Reducing Sensitivity to Patch Corruptions
Guo, Y. X., Stutz, D., & Schiele, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00400 · Cited by 22 · see citing works
“Besides improving clean accuracy, many works seek to study and improve the robustness of ViTs [2, 4, 5, 18, 20, 37, 43].”
BackgroundJahan, M., Hasan, M. Z., Samia, I. J., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3435572 · Cited by 19 · see citing works
“Vision Transformer (ViT), which applies a pure self-attention-based model to sequences of image patches and achieves competitive performance when compared to CNNs, is the most noteworthy development in this field [14].”
BackgroundContext-Aware Robust Fine-Tuning
Mao, X., Chen, Y., Jia, X., Zhang, R., Xue, H., & Zhao, L. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01951-2 · Cited by 23 · see citing works
“…et al., 2019, 2021a), adversarial training mechanism (Xie et al., 2020; Mao et al., 2022a; Herrmann et al., 2022), advanced network design (Paul and Chen, 2022; Mao et al., 2022b; Bai et al., 2021; Wang et al., 2022) or regularization for a flatter minimum (Cha et al., 2021; Foret et al.,…”
Are Vision Transformers Robust to Spurious Correlations?
Ghosal, S. S., & Li, Y. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01916-5 · Cited by 20 · see citing works
“Specifically, BiT-M-R50x3 only improves the worst-group accuracy by 1.5% with ImageNet-21k.”
BackgroundInfluentialCOCO-O: A Benchmark for Object Detectors under Natural Distribution Shifts
Mao, X., Chen, Y., Zhu, Y. P., Chen, D., Su, H., Zhang, R., & Xue, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00583 · Cited by 23 · see citing works
“An interesting phenomenon is that detection transformers have the worst OOD generalization ability, contrary to the general conclusion in the field of image classification that transformers can enhance OOD robustness [61, 2, 59].”
BackgroundKojima, T., Matsuo, Y., & Iwasawa, Y. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/141 · Cited by 23 · see citing works
PatchCleanser: Certifiably Robust Defense against Adversarial Patches for Any Image ClassifierPreprint
Xiang, C., Mahloujifar, S., & Mittal, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.09135 · Cited by 34 · see citing works
“We believe that this is because ViT has been found more robust to pixel masking [28].”
BackgroundDecoding the User’s Movements Preparation From EEG Signals Using Vision Transformer Architecture
Al-Quraishi, M. S., Elamvazuthi, I., Tang, T. B., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3213996 · Cited by 18 · see citing works
“the field of computer vision, mainly for tasks like image classification, image creation and enhancement, object detection, scene segmentation, video processing and 3D processing [29], [30].”
Rethinking the Design Principles of Robust Vision TransformerPreprint
Mao, X., Qi, G., Chen, Y., Li, X., Ye, S., He, Y., & Xue, H. (2021) · arXiv (Cornell University) · Cited by 17 · see citing works
Kumar, D., William, P., Deshmukh, S. R., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3646075 · Cited by 17 · see citing works
“This has created growing enthusi-asm in real-time, real-time automated garbage detection and segregation systems able to provide data to optimize waste treatment plans [8], [9], [10].”
Holistic Adversarial Robustness of Deep Learning Models
Chen, P., & Liu, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.26797 · Cited by 18 · see citing works
“(Paul and Chen 2022) extends the analysis to a variety of robustness aspects beyond adversarial robustness.”
BackgroundSelf-citationStatic hand gesture recognition method based on the Vision Transformer
Zhang, Y., Wang, J., Wang, X., Jing, H., Sun, Z., & Cai, Y. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14732-3 · Cited by 22 · see citing works
Pang, J., Jiang, C., Chen, Y., et al. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3191974 · Cited by 17 · see citing works
“The more vision transformer components are added, the more stable the model will be, which is also consistent with empirical knowledge [56] in general vision tasks.”
BackgroundSelf-supervised approach for diabetic retinopathy severity detection using vision transformer
Ohri, K., Kumar, M., & Sukheja, D. (2024) · Progress in Artificial Intelligence · doi:10.1007/s13748-024-00325-0 · Cited by 14 · see citing works
“Vision transformers (ViT) are scalable and can process large amounts of data by operating on image patches [48, 49].”
Leveraging the Power of Data Augmentation for Transformer-based Tracking
Zhao, J. W., Edstedt, J., Felsberg, M., Wang, D., & Lu, H. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00634 · Cited by 17 · see citing works
“Trans-formers have been found to exhibit different properties [33] and be more robust to perturbations [4,34] than CNNs.”
BackgroundOptimization-Free Test-Time Adaptation for Cross-Person Activity Recognition
Wang, S., Wang, J., Xi, H., Zhang, B., Zhang, L., & Wei, H. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3631450 · Cited by 18 · see citing works
Khan, I. U., Azam, S., Montaha, S., Mahmud, A. A., Rafid, A. K. M. R. H., Hasan, M. Z., & Jonkman, M. (2022) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2022.200147 · Cited by 16 · see citing works
Anjali, T., Abhishek, S., & Remya, S. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2024.12.080 · Cited by 20 · see citing works
A survey on cell nuclei instance segmentation and classification: Leveraging context and attention
Nunes, J. D., Montezuma, D., Oliveira, D., Pereira, T., & Cardoso, J. S. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103360 · Cited by 18 · see citing works
An Image-Text Sentiment Analysis Method Using Multi-Channel Multi-Modal Joint Learning
Gong, L., He, X., & Jianzhong, Y. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2371712 · Cited by 17 · see citing works
“ViT (Paul and Chen 2022; Xie and Liao 2023) is a vision model based on the Transformer architecture that achieves impressive performance in computer vision tasks with promising applications.”
Towards Transferable Adversarial Attacks on Image and Video Transformers
Wei, Z.-P., Chen, J.-J., Goldblum, M., Wu, Z.-X., Goldstein, T., Jiang, Y.-G., & Davis, L. S. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3331582 · Cited by 18 · see citing works
“Apart from the above works, there are also some other papers [38], [39], [40] that discuss the robustness of ViTs.”
ResultKim, J. H., & Kim, H. R. (2024) · Research Square · doi:10.21203/rs.3.rs-4328966/v1 · Cited by 13 · see citing works
Exploring Adversarial Robustness of Vision Transformers in the Spectral Perspective
Kim, G., Kim, J., & Lee, J.-S. (2022) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00393 · Cited by 16 · see citing works
“(2) Some previous works explain different levels of vulnerability of CNNs and Transformers in terms of their reliance on different frequency components [4], [5], [6], [28].”
BackgroundInfluentialQin, Y., Zhang, C., Chen, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07858 · Cited by 13 · see citing works
“It is hypothesized in recent studies (Naseer et al., 2021; Paul & Chen, 2021; Bhojanapalli et al., 2021) that the non-local attention based interactions contribute to better robustness of ViTs than ConvNets.”
BackgroundMethodResultInfluentialPure Vision Transformer (CT-ViT) with Noise2Neighbors Interpolation for Low-Dose CT Image Denoising
Marcos, L., Babyn, P., & Alirezaie, J. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01108-8 · Cited by 13 · see citing works
“Mathematically, (1) where the parameters contained in B represent the relative position bias [37], and d K refers to the dimensionality of the key vectors.”
Rethinking Architecture Design for Tackling Data Heterogeneity in Federated LearningPreprint
Qu, L., Zhou, Y., Liang, P. P., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06047 · Cited by 12 · see citing works
“Among several particularly relevant findings are that ViTs are highly robust to severe occlusions, perturbations, domain shifts [3, 48], as well as synthetic and natural adversarial examples [44,51].”
BackgroundAI Applications in Education: Enhancing Human Creativity Through Collaborative Design
Agboola, O. P., & Yassin, Y. N. H. M. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-95901-1_4 · Cited by 12 · see citing works
On Improving Adversarial Transferability of Vision TransformersPreprint
Naseer, M., Ranasinghe, K., Khan, S. H., Khan, F. S., & Porikli, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04169 · Cited by 55 · see citing works
Food Recognition with Visual Transformers
Bianco, S., Buzzelli, M., Chiriaco, G., Napoletano, P., & Piccoli, F. (2023) · IEEE International Conference on Consumer Electronics - Berlin (ICCE-Berlin) · doi:10.1109/icce-berlin58801.2023.10375660 · Cited by 11 · see citing works
“Their demonstrated robustness to perturbations [7] and their ability to capture long-range dependencies can be particularly helpful in extracting robust representations of food images.”
BackgroundMR Image Harmonization with Transformer
Han, D., Yu, R. H., Li, S., et al. (2023) · IEEE International Conference on Mechatronics and Automation (ICMA) · doi:10.1109/icma57826.2023.10215948 · Cited by 10 · see citing works
“This contributes to a holistic understanding of global information [7], providing advantages in capturing the complex structures of visual data.”
BackgroundSwin-PAFF: A SAR Ship Detection Network with Contextual Cross-Information Fusion
Zhang, Y., Han, D., & Chen, P. (2023) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.042311 · Cited by 12 · see citing works
Self-citationDiscrete Representations Strengthen Vision Transformer RobustnessPreprint
Mao, C., Jiang, L., Dehghani, M., Vondrick, C., Sukthankar, R., & Essa, I. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10493 · Cited by 16 · see citing works
“Existing works on robustness are targeted at closing the gap in a subset of these ImageNet robustness benchmarks and were extensively verified with the CNNs.”
BackgroundInfluentialProducing plankton classifiers that are robust to dataset shift
Chen, C., Kyathanahally, S. P., Reyes, M., Merkli, S., Merz, E., Francazi, E., Hoege, M., Pomati, F., & Baity‐Jesi, M. (2024) · Limnology and Oceanography Methods · doi:10.1002/lom3.10659 · Cited by 9 · see citing works
“This has the advantage of not requiring retraining models.”
ViT-PGC: vision transformer for pedestrian gender classification on small-size dataset
Abbas, F., Yasmin, M., Fayyaz, M., & Asim, U. (2023) · Pattern Analysis and Applications · doi:10.1007/s10044-023-01196-2 · Cited by 9 · see citing works
Yuan, J., Liu, T., Xia, H., & Zou, X. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3336546 · Cited by 9 · see citing works
“Some researchers believe that this is due to the use of multi-head self-attention (MHSA) in VIT [17].”
Background
- Authors
- Paul, S., & Chen, P.-Y.
- Affiliations
- IBM (United States) (US)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2022-06-28
- Volume
- 36
- Issue
- 2
- Pages
- 2071-2081
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W3161120562
- Cited by
- 213
- References
- 99
- Citations by year
- 2026: 16 · 2025: 25 · 2024: 36 · 2023: 81 · 2022: 34 · 2021: 21
- FWCI
- 28.37
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- vision transformer, robustness, self-attention, adversarial examples, image corruptions, distribution shift, image masking, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
