The DOI matches its Crossref record; 43 of the 46 references it cites were found in a registry or an index.
Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
Open accessAbstract
Pretrained models from self-supervision are prevalently used in fine-tuning downstream tasks faster or for better accuracy. However, gaining robustness from pretraining is left unexplored. We introduce adversarial training into self-supervision, to provide general-purpose robust pretrained models for the first time. We find these robust pretrained models can benefit the subsequent fine-tuning in two ways: i) boosting final model robustness; ii) saving the computation cost, if proceeding towards adversarial fine-tuning. We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins (eg, 3.83% on robust accuracy and 1.3% on standard accuracy, on the CIFAR-10 dataset), compared with the conventional end-to-end adversarial training baseline. Moreover, we find that different self-supervised pretrained models have diverse adversarial vulnerability. It inspires us to ensemble several pretraining tasks, which boosts robustness more. Our ensemble strategy contributes to a further improvement of 3.59% on robust accuracy, while maintaining a slightly higher standard accuracy on CIFAR-10. Our codes are available at https://github.com/TAMU-VITA/Adv-SS-Pretraining.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Chen et al. [35, 36] used a hybrid SSL approach to regularize the training of a text classification task.”
From Self-supervised learning: A succinct review (2023) · cites this paper as [35]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602
DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
On Assessing ML Model Robustness: A Methodological Framework (Academic Track)
Awadid, A., & Robert, B. (2025) · OASIcs : OpenAccess Series in Informatics · doi:10.4230/oasics.saia.2024.1 · Cited by 4,723
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051
Zhang, R., Isola, P., & Efros, A. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,417
Region Filling and Object Removal by Exemplar-Based Image Inpainting
Criminisi, A., Perez, P., & Toyama, K. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2004.833105 · Cited by 3,258 · cited 2×
“Examples of tasks proven to attain high accuracy through self-supervised pretraining include position predicting tasks (Selfie [35], Jigsaw [25, 3]), rotation predicting tasks (Rotation [9]), and a variety of other perception tasks [6, 41, 8].”
BackgroundUnsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775
Unsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477
Adversarial machine learning at scale
Kurakin, A., Goodfellow, I., & Bengio, S. (2016) · arXiv · doi:10.48550/arxiv.1611.01236 · Cited by 2,325
Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
Xu, W., Evans, D., & Qi, Y. (2018) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23198 · Cited by 2,204
Ensemble Adversarial Training: Attacks and Defenses
Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D., & McDaniel, P. (2017) · arXiv · doi:10.48550/arxiv.1705.07204 · Cited by 2,176
Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples
Athalye, A., Carlini, N., & Wagner, D. (2018) · arXiv · doi:10.48550/arxiv.1802.00420 · Cited by 2,172
80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
Torralba, A., Fergus, R., & Freeman, W. T. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.128 · Cited by 1,921
Theoretically principled trade-off between robustness and accuracy
Zhang, H., Yu, Y., Jiao, J., Xing, E. P., Ghaoui, L. E., & Jordan, M. I. (2019) · arXiv · doi:10.48550/arxiv.1901.08573 · Cited by 1,560
Raina, R., Battle, A. J., Lee, H., Packer, B., & Ng, A. Y. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273592 · Cited by 1,534
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
Hendrycks, D., & Dietterich, T. (2019) · arXiv · doi:10.48550/arxiv.1903.12261 · Cited by 1,335
Certified Adversarial Robustness via Randomized SmoothingPreprint
Cohen, J. M., Rosenfeld, E., & Kolter, J. Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02918 · Cited by 1,232
Meng, D., & Chen, H. (2017) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3133956.3134057 · Cited by 1,064
Defense Against Adversarial Attacks Using High-Level Representation Guided Denoiser
Liao, F., Liang, M., Dong, Y., Pang, T., Hu, X., & Zhu, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00191 · Cited by 987
Adversarially Robust Generalization Requires More DataPreprint
Schmidt, L., Santurkar, S., Tsipras, D., Talwar, K., & Ma̧dry, A. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1804.11285 · Cited by 891
Domain Generalization by Solving Jigsaw Puzzles
Carlucci, F. M., D’Innocente, A., Bucci, S., Barbara Caputo, B., & Tommasi, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00233 · Cited by 858
Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
Dosovitskiy, A., Fischer, P., Springenberg, J. T., Riedmiller, M., & Brox, T. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2496141 · Cited by 762
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
Hendrycks, D., Mazeika, M., Kadavath, S., & Song, D. (2019) · arXiv · doi:10.48550/arxiv.1906.12340 · Cited by 634
Stochastic Activation Pruning for Robust Adversarial DefensePreprint
Dhillon, G. S., Azizzadenesheli, K., Lipton, Z. C., Bernstein, J., Kossaifi, J., Khanna, A., & Anandkumar, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.01442 · Cited by 447
Unlabeled Data Improves Adversarial RobustnessPreprint
Carmon, Y., Raghunathan, A., Schmidt, L., Liang, P., & Duchi, J. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.13736 · Cited by 440
FreeLB: Enhanced Adversarial Training for Natural Language UnderstandingPreprint
Zhu, C., Cheng, Y., Gan, Z., Sun, S., Goldstein, T., & Liu, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11764 · Cited by 402
Self-citationImproving Adversarial Robustness via Promoting Ensemble DiversityPreprint
Pang, T., Xu, K., Du, C., Chen, N., & Zhu, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08846 · Cited by 300
Are Labels Required for Improving Adversarial Robustness?Preprint
Uesato, J., Alayrac, J.-B., Huang, P.-S., Stanforth, R., Fawzi, A., & Kohli, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.13725 · Cited by 207
Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural NetworksPreprint
Strauss, T., Hanselmann, M., Junginger, A., & Ulmer, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.03423 · Cited by 168
Self-Supervised Learning for Generalizable Out-of-Distribution Detection
Mohseni, S., Pitale, M., Yadawa, J., & Wang, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5966 · Cited by 159
Self-citationAdversarially Robust Generalization Just Requires More Unlabeled DataPreprint
Zhai, R., Cai, T., He, D., Chen, D., He, K., Hopcroft, J. E., & Wang, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.00555 · Cited by 144
Selfie: Self-supervised Pretraining for Image EmbeddingPreprint
Trinh, T. H., Luong, M.-T., & Le, Q. V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02940 · Cited by 122
Testing Robustness Against Unforeseen AdversariesPreprint
Kaufmann, M., Kang, D. D., Sun, Y. J., Basart, S., Yin, X., Mazeika, M., Arora, A., Dziedzic, A., Boenisch, F., Brown, T., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.08016 · Cited by 121
Model Compression with Adversarial Robustness: A Unified Optimization FrameworkPreprint
Gui, S., Wang, H., Yu, C., Yang, H., Wang, Z., & Liu, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.03538 · Cited by 96
Self-citationExtending Defensive DistillationPreprint
Papernot, N., & McDaniel, P. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.05264 · Cited by 78
Hu, T.-K., Chen, T., Wang, H., & Wang, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10025 · Cited by 61
Self-citationTowards Understanding the Transferability of Deep RepresentationsPreprint
Liu, H., Long, M., Wang, J., & Jordan, M. I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.12031 · Cited by 24
Computer Vision with a Single (Robust) Classifier.Preprint
Santurkar, S., Tsipras, D., Tran, B., Ilyas, A., Engstrom, L., & Mądry, A. (2019) · arXiv (Cornell University) · Cited by 19
Towards A Unified Min-Max Framework for Adversarial Exploration and RobustnessPreprint
Wang, J., Zhang, T., Liu, S., Chen, P., Xu, J., Fardad, M., & Li, B. (2019) · arXiv (Cornell University) · Cited by 19
Self-citation
Cited by
Sorted by citations
Graph Contrastive Learning with AugmentationsPreprint
You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., & Shen, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13902 · Cited by 1,400
Self-citationRecent advances and clinical applications of deep learning in medical image analysis
Chen, X., Wang, X., Zhang, K., Fung, K.-M. A., Thai, T., Moore, K. N., Mannel, R. S., Liu, H., Zheng, B., & Qiu, Y. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102444 · Cited by 1,180
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends
Gui, J., Chen, T., Zhang, J., Cao, Q., Sun, Z., Luo, H., & Tao, D. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3415112 · Cited by 730
Self-citationRGB-D salient object detection: A survey
Zhou, T., Fan, D.-P., Cheng, M.-M., Shen, J., & Shao, L. (2021) · Computational Visual Media · doi:10.1007/s41095-020-0199-z · Cited by 331
Advances in Adversarial Attacks and Defenses in Computer Vision: A Survey
Akhtar, N., Mian, A. S., Kardan, N., & Shah, M. (2021) · IEEE Access · doi:10.1109/access.2021.3127960 · Cited by 355
Metadata
- Authors
- Chen, T., Liu, S., Chang, S., Cheng, Y., Amini, L., & Wang, Z.
- Affiliations
- Texas A&M University (US)
- IBM (United States) (US)
- Microsoft (United States) (US)
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 696-705
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3035106315
- Cited by
- 205
- References
- 46
- Citations by year
- 2026: 13 · 2025: 32 · 2024: 35 · 2023: 34 · 2022: 31 · 2021: 46 · 2020: 14
- FWCI
- 11.94
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- adversarial robustness, self-supervised pre-training, adversarial fine-tuning, adversarial vulnerability, robust accuracy, CIFAR-10, computational complexity reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
