CoChat
Sign Up
PreprintCited by 98Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsAdversarial Robustness in Machine Learning

The DOI matches its DataCite record.

On Robustness and Transferability of Convolutional Neural Networks

Open access
Djolonga, Josip, Yung, Jessica, Tschannen, Michael, Romijnders, Rob, Beyer, Lucas, Kolesnikov, Alexander, Puigcerver, Joan, Minderer, Matthias, D'Amour, Alexander, Moldovan, Dan, Gelly, Sylvain, Houlsby, Neil, Zhai, Xiaohua, Lucic, Mario·2020·arXiv
DOI 10.48550/arxiv.2007.08558arXiv 2007.08558
Open this paper in CoChat — free accountOpen PDF

Abstract

Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts. However, several recent breakthroughs in transfer learning suggest that these networks can cope with severe distribution shifts and successfully adapt to new tasks from a few training examples. In this work we study the interplay between out-of-distribution and transfer performance of modern image classification CNNs for the first time and investigate the impact of the pre-training data size, the model scale, and the data preprocessing pipeline. We find that increasing both the training set and model sizes significantly improve the distributional shift robustness. Furthermore, we show that, perhaps surprisingly, simple changes in the preprocessing such as modifying the image resolution can significantly mitigate robustness issues in some cases. Finally, we outline the shortcomings of existing robustness evaluation datasets and introduce a synthetic dataset SI-Score we use for a systematic analysis across factors of variation common in visual data such as object size and position.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2007.08558

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Robustness to synthetic shifts need not transfer to the kinds of shifts that arise in real-world deployments.

    “robustness to these synthetic shifts need not transfer to the kinds of shifts that arise in real-world deployments (Taori et al., 2020; Djolonga et al., 2020; Xie et al., 2020)”

    From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Djolonga et al., 2020

    Section 3 compares synthetic ImageNet-C corruptions with naturally collected robustness datasets and finds that metrics based on different datasets or video are more informative than ImageNet-derived metrics. This supports caution about treating synthetic-shift robustness as a complete measure of natural-shift robustness. The paper also says the tested metrics are highly rank-correlated, however, and does not show a specific synthetic robustness gain failing to transfer to a real-world deployment. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Model robustness need not transfer across shifts: models can be robust to image corruptions but not to shifts across datasets.

    “model robustness need not transfer across shifts: e.g., models can be robust to image corruptions but not to shifts across datasets (Taori et al., 2020; Djolonga et al., 2020)”

    From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Djolonga et al., 2020

    The Section 3 summary (page unavailable) says that transfer highlights failures somewhat independent of robustness, supporting the general point that robustness does not guarantee transfer. Section 2 distinguishes corrupted ImageNet images from separately collected test datasets, but our copy does not show a model that is robust to corruptions yet fails on a shift across datasets. The specific example is therefore not established by this source. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 71

  1. Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Nat- sev, George Toderici, Balakrishnan Varadarajan, and Sud- heendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv:1609.08675, 2016. 3 · arXiv
  2. Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transforma- tions? Journal of Machine Learning Research, 20, 2019.
  3. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Ad- vances in Neural Information Processing Systems, 2019. 2, 7, 13
  4. Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Si- mon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016. 13 · arXiv
  5. Ali Borji. Objectnet dataset: Reanalysis and correction. In arXiv 2004.02042, 2020. 2 · arXiv
  6. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geof- frey Hinton. A simple framework for contrastive learning of visual representations. arXiv:2002.05709, 2020. 3, 22 · arXiv
  7. Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised GANs via auxiliary rotation loss. In Conference on Computer Vision and Pattern Recogni- tion, 2019. 7
  8. Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 2017. 13
  9. M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In IEEE Conference on Computer Vision and Pattern Recognition, 2014. 13
  10. Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasude- van, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Conference on Computer Vision and Pattern Recognition, 2019. 3
  11. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, 2009. 2, 5
  12. Samuel Dodge and Lina Karam. Understanding how im- age quality affects deep neural networks. In International Conference on Quality of Multimedia Experience, 2016. 7
  13. Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsuper- vised visual representation learning by context prediction. In International Conference on Computer Vision, 2015. 8
  14. Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems, 2019. 3, 8, 22
  15. Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springen- berg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE transactions on pattern analysis and machine intelligence, 38(9), 2015. 2
  16. Debidatta Dwibedi, Ishan Misra, and Martial Hebert. Cut, paste and learn: Surprisingly easy synthesis for instance de- tection. In International Conference on Computer Vision, 2017. 8
  17. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Jacob Steinhardt, and Aleksander Madry. Identifying statistical bias in dataset replication. arXiv: 2005.09619, 2020. 2, 7 · arXiv
  18. Logan Engstrom, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. A rotation and a translation suf- fice: Fooling CNNs with simple transformations. CoRR, abs/1712.02779, 2017. 8 · arXiv
  19. Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. Interna- tional Journal of Robotics Research, 2013. 13
  20. Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Bren- del. Imagenet-trained CNNs are biased towards texture; in- creasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. 7
  21. Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsu- pervised representation learning by predicting image rotations. arXiv:1803.07728, 2018. 2 · arXiv
  22. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv:1412.6572, 2014. 1, 7 · arXiv
  23. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, 2016. 3, 5
  24. Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019. 13
  25. Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robust- ness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006.16241, 2020. 8 · arXiv
  26. Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. arXiv: 1807.01697, 2018. 1, 2, 3, 4, 13 · arXiv
  27. Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, 2019. 7
  28. Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv:1912.02781, 2019. 7 · arXiv
  29. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. arXiv: 1907.07174, 2019. 2, 13 · arXiv
  30. Hossein Hosseini, Baicen Xiao, and Radha Poovendran. Google's cloud vision api is not robust to noise. In Inter- national Conference on Machine Learning and Applications, 2017. 7
  31. Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning? arXiv:1608.08614, 2016. 8 · arXiv
  32. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, 2017. 13
  33. Samil Karahan, Merve Kilinc Yildirim, Kadir Kirtaç, Fer- hat Sükrü Rende, Gultekin Butun, and Hazim Kemal Ekenel. How image degradations affect deep CNN-based face recog- nition? In International Conference of the Biometrics Special Interest Group, 2016. 7
  34. Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (BiT): General visual representation learning. European Conference on Computer Vision, 2020. 1, 3, 4, 5, 7, 8, 13, 14, 22
  35. Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do better imagenet models transfer better? In Conference on Computer Vision and Pattern Recognition, 2019. 1, 3
  36. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Im- agenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012. 3, 13
  37. Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified im- age classification, object detection, and visual relationship detection at scale. arXiv: 1811.00982, 2020. 6 · arXiv
  38. Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In IEEE Conference on Computer Vision and Pattern Recognition, 2004. 13
  39. Fei-Fei Li, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006. 12
  40. Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D Cubuk. Improving robustness with- out sacrificing accuracy with patch gaussian augmentation. arXiv:1906.02611, 2019. 7 · arXiv
  41. Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In European Conference on Computer Vision, 2018. 8
  42. Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. 13
  43. M-E. Nilsback and A. Zisserman. Automated flower classifi- cation over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008. 13
  44. Sinno Jialin Pan and Qiang Yang. A survey on transfer learn- ing. IEEE Transactions on knowledge and data engineering, 22(10), 2009. 2
  45. O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012. 13
  46. Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. The MIT Press, 2009. 1, 2
  47. Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. Youtube-boundingboxes: A large high-precision human-annotated data set for object detection in video. In Conference on Computer Vision and Pattern Recognition, 2017. 2
  48. Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to ima- genet? arXiv: 1902.10811, 2019. 2, 4, 8, 13 · arXiv
  49. Prasun Roy, Subhankar Ghosh, Saumik Bhattacharya, and Umapada Pal. Effects of degradations on deep neural network architectures. arXiv:1807.10108, 2018. 7 · arXiv
  50. Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ra- manan, Benjamin Recht, and Ludwig Schmidt. A sys- tematic framework for natural perturbations from videos. arXiv:1906.02168, 2019. 1, 2, 13 · arXiv
  51. Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshmi- narayanan, Sebastian Nowozin, D. Sculley, Joshua V. Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncer- tainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, 2019. 8
  52. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In International Conference on Computer Vision, 2017. 5, 14
  53. Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A Efros, and Moritz Hardt. Test-time training for out-of- distribution generalization. arXiv:1909.13231, 2019. 7 · arXiv
  54. C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Conference on Computer Vision and Pattern Recognition, 2015. 14
  55. Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv:1905.11946, 2019. 1, 3, 4, 22 · arXiv
  56. Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33, 2020. 3, 8
  57. Dogancan Temel, Jinsol Lee, and Ghassan AlRegib. Cure-or: Challenging unreal and real environments for object recogni- tion. In International Conference on Machine Learning and Applications, 2018. 7
  58. Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. In Ad- vances in Neural Information Processing Systems, 2019. 5, 6
  59. Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy: Fixeffi- cientnet. arXiv:2003.08237, 2020. 4, 5, 6 · arXiv
  60. Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv:1903.03096, 2019. 8 · arXiv
  61. Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby, Sylvain Gelly, and Mario Lucic. Self-supervised learning of video-induced visual invariances. In Conference on Computer Vision and Pattern Recognition, 2020. 3, 8, 22
  62. Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Co- hen, and Max Welling. Rotation equivariant cnns for digital pathology. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2018. 13
  63. Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recog- nition from abbey to zoo. In IEEE Conference on Computer Vision and Pattern Recognition, 2010. 13
  64. Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. Adversarial examples improve image recognition. arXiv:1911.09665, 2019. 3, 7, 22 · arXiv
  65. Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Self-training with noisy student improves imagenet clas- sification. arXiv:1911.04252, 2019. 1, 3, 4, 7, 8, 22 · arXiv
  66. Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lu- cas Beyer. S4l: Self-supervised semi-supervised learning. In International Conference on Computer Vision, 2019. 3, 7, 22
  67. Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv: 1910.04867, 2019. 1, 2, 3, 8, 12 · arXiv
  68. Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. In International Conference on Learning Representa- tions, 2018. 14
  69. Richard Zhang. Making convolutional networks shift- invariant again. In International Conference on Machine Learning, 2019. 7
  70. Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. Distilling localization for self-supervised representation learning. arXiv: 2004.06638, 2020. 8 · arXiv
  71. Yiren Zhou, Sibo Song, and Ngai-Man Cheung. On classi- fication of distorted images with deep convolutional neural networks. In International Conference on Acoustics, Speech and Signal Processing, 2017. 7

Cited by

Sorted by citations

Metadata

Authors
Djolonga, J., Yung, J., Tschannen, M., Romijnders, R., Beyer, L., Kolesnikov, A., Puigcerver, J., Minderer, M., D'Amour, A., Moldovan, D., Gelly, S., Houlsby, N., Zhai, X., & Lucic, M.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
16453-16463
ISSN
1063-6919 (ISSN-L)
DOI
10.48550/arxiv.2007.08558
arXiv
2007.08558
OpenAlex
W3042608254
Cited by
98
References
0
Citations by year
2026: 5 · 2025: 17 · 2024: 7 · 2023: 16 · 2022: 20 · 2021: 26 · 2020: 7
FWCI
7.97
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
distribution shift, out-of-distribution robustness, transfer learning, convolutional neural networks, image classification, model scale, data preprocessing, image resolution

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review