CoChat
Sign Up
PreprintCited by 2

The DOI matches its DataCite record; 7 of the 7 references it cites were found in a registry or an index.

Improving robustness against common corruptions by covariate shift adaptation

Schneider, Steffen, Rusak, Evgenia, Eck, Luisa, Bringmann, Oliver, Brendel, Wieland, Bethge, Matthias·2020·arXiv
DOI 10.48550/arxiv.2006.16971arXiv 2006.16971
Open this paper in CoChat — free account

Abstract

Today's state-of-the-art machine vision models are vulnerable to image corruptions like blurring or compression artefacts, limiting their performance in many real-world applications. We here argue that popular benchmarks to measure model robustness against common corruptions (like ImageNet-C) underestimate model robustness in many (but not all) application scenarios. The key insight is that in many scenarios, multiple unlabeled examples of the corruptions are available and can be used for unsupervised online adaptation. Replacing the activation statistics estimated by batch normalization on the training set with the statistics of the corrupted images consistently improves the robustness across 25 different popular computer vision models. Using the corrected statistics, ResNet-50 reaches 62.2% mCE on ImageNet-C compared to 76.7% without adaptation. With the more robust DeepAugment+AugMix model, we improve the state of the art achieved by a ResNet50 model up to date from 53.6% mCE to 45.4% mCE. Even adapting to a single sample improves robustness for the ResNet-50 and AugMix models, and 32 samples are sufficient to improve the current state of the art for a ResNet-50 architecture. We argue that results with adapted statistics should be included whenever reporting scores in corruption benchmarks and other out-of-distribution generalization settings.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.16971

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Accessing multiple test points can be powerful for benchmarks such as ImageNet-C and ImageNet-R, in which inferred statistics from the test input distribution may aid in prediction.

    “As Table 2 and prior work both show [40, 46], accessing multiple test points can be powerful for benchmarks such as ImageNet-C and ImageNet-R, in which inferred statistics from the test input distribution may aid in prediction.”

    From MEMO: Test Time Robustness Via Adaptation and Augmentation (2022) · cites this paper as [40]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 57

  1. Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann. Generalisation in humans and deep neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 7538-7550.
  2. Curran Associates, Inc., 2018. URL http://papers.nips.cc/paper/ 7982-generalisation-in-humans-and-deep-neural-networks.pdf.
  3. Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
  4. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICLR), 2015.
  5. Steffen Schneider, Alexander S Ecker, Jakob H Macke, and Matthias Bethge. Multi-task generalization and adaptation between noisy digit datasets: An empirical study. In Neural Information Processing Systems (NeurIPS), Workshop on Continual Learning, 2018.
  6. Fabio Maria Cariucci, Lorenzo Porzi, Barbara Caputo, Elisa Ricci, and Samuel Rota Bulo. Autodial: Automatic domain alignment layers. In 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
  7. Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. In International Conference on Machine Learning (ICLR), 2017.
  8. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision (IJCV), 2015.
  9. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
  10. Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? In Advances in Neural Information Processing Systems (NIPS), 2018.
  11. Masashi Sugiyama and Motoaki Kawanabe. Machine learning in non-stationary environments: Introduc- tion to covariate shift adaptation. MIT press, 2012.
  12. Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML'12, page 459-466, Madison, WI, USA, 2012. Omnipress. ISBN 9781450312851.
  13. Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg, 2006. ISBN 0387310738.
  14. Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  15. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  16. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Conference on computer vision and pattern recognition (CVPR), 2016.
  17. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Conference on computer vision and pattern recognition (CVPR), 2018.
  18. Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transfor- mations for deep neural networks. In Conference on computer vision and pattern recognition (CVPR), 2017.
  19. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations (ICLR), 2015.
  20. Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. CoRR, abs/1605.07146, 2016. · arXiv
  21. Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In ACM International Conference on Multimedia, 2010.
  22. Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  23. Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. CoRR, abs/2006.10029, 2020. · arXiv
  24. Evgenia Rusak, Lukas Schott, Roland Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. Increasing the robustness of dnns against image corruptions by playing the game of noise. CoRR, abs/2001.06057, 2020. · arXiv
  25. Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representations (ICLR), 2020.
  26. Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  27. Jungkyu Lee, Taeryun Won, and Kiho Hong. Compounding the performance improvements of assembled techniques in a convolutional neural network. CoRR, abs/2001.06268, 2020. · arXiv
  28. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CoRR, abs/1907.07174, 2019. · arXiv
  29. Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  30. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems 32, 2019.
  31. Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. CoRR, abs/2006.16241, 2020. · arXiv
  32. A Emin Orhan. Robustness properties of facebook's resnext wsl models. CoRR, abs/1907.07640, 2019. · arXiv
  33. Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W Taylor. Batch normalization is a cause of adversarial vulnerability. CoRR, abs/1905.02161, 2019. · arXiv
  34. Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. CoRR, abs/1901.09321, 2019. · arXiv
  35. Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  36. Norman Mu and Justin Gilmer. MNIST-C: A robustness benchmark for computer vision. CoRR, abs/1906.02337, 2019. · arXiv
  37. Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. CoRR, abs/1907.07484, 2019. · arXiv
  38. Christoph Kamann and Carsten Rother. Benchmarking the robustness of semantic segmentation models. CoRR, abs/1908.05005, 2019. · arXiv
  39. Nic Ford, Justin Gilmer, Nicolas Carlini, and Dogus Cubuk. Adversarial examples are a natural consequence of test error in noise. In International Conference on Machine Learning (ICML), 2019.
  40. Agnieszka Mikołajczyk and Michał Grochowski. Data augmentation for improving deep learning in image classification problem. In International Interdisciplinary PhD Workshop (IIPhDW), 2018.
  41. Richard Zhang. Making convolutional networks shift-invariant again. International Conference on Machine Learning (ICML), 2019.
  42. Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adaptation. In Domain Adaptation in Computer Vision Applications, pages 153-171. Springer, 2017.
  43. Daniel Bug, Steffen Schneider, Anne Grote, Eva Oswald, Friedrich Feuerhake, Julia Schüler, and Dorit Merhof. Context-based normalization of histological stains using deep convolutional features. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer, 2017.
  44. Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A Efros, and Moritz Hardt. Test-time training for out-of-distribution generalization. CoRR, abs/1909.13231, 2019. · arXiv
  45. Geoffrey French, Michal Mackiewicz, and Mark H. Fisher. Self-ensembling for domain adaptation. CoRR, abs/1706.05208, 2017. · arXiv
  46. Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10687-10698, 2020.
  47. Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Fully test-time adaptation by entropy minimization. CoRR, abs/2006.10726, 2020. · arXiv
  48. Cihang Xie and Alan L. Yuille. Intriguing properties of adversarial training. In International Conference on Learning Representations (ICLR), 2020.
  49. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems (NIPS), 2017.
  50. Jonathan Frankle, David J Schwab, and Ari S Morcos. Training batchnorm and only batchnorm: On the expressive power of random features in cnns. CoRR, abs/2003.00152, 2020. · arXiv
  51. Zachary Nado, Shreyas Padhy, D Sculley, Alexander D'Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. CoRR, abs/2006.10963, 2020. · arXiv
  52. Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
  53. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Jacob Steinhardt, and Aleksander Madry. Identifying statistical bias in dataset replication. CoRR, abs/2005.09619, 2020. · arXiv
  54. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake Vand erPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1. 0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: https://doi.org/10.1038/s41592-019-0686-2. · DOI · Paper page
  55. O. Tange. Gnu parallel -the command-line power tool. ;login: The USENIX Magazine, 36(1):42-47, Feb 2011. URL http://www.gnu.org/s/parallel.
  56. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16), pages 265-283, 2016.
  57. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS). 2012. (0.00, 0.03) (0.00, 0.05) (0.00, 0.08) (0.00, 0.10) (0.00, 0.13) (0.00, 0.15) (0.00, 0.18) (0.00, 0.20) (0.00, 0.23) (0.00, 0.25) (0.01, 0.03) (0.01, 0.05) (0.01, 0.08) (0.01, 0.10) (0.01, 0.13) (0.01, 0.15) (0.01, 0.18) (0.01, 0.20) (0.01, 0.23) (0.01, 0.25) (0.01, 0.03) (0.01, 0.05) (0.01, 0.08) (0.01, 0.10) (0.01, 0.13) (0.01, 0.15) (0.01, 0.18) (0.01, 0.20) (0.01, 0.23) (0.01, 0.25) (0.02, 0.03) (0.02, 0.05) (0.02, 0.08) (0.02, 0.10) (0.02, 0.13) (0.02, 0.15) (0.02, 0.18) (0.02, 0.20) (0.02, 0.23) (0.02, 0.25) (0.02, 0.03) (0.02, 0.05) (0.02, 0.08) (0.02, 0.10) (0.02, 0.13) (0.02, 0.15) (0.02, 0.18) (0.02, 0.20) (0.02, 0.23) (0.02, 0.25) (0.03, 0.03) (0.03, 0.05) (0.03, 0.08) (0.03, 0.10) (0.03, 0.13) (0.03, 0.15) (0.03, 0.18) (0.03, 0.20) (0.03, 0.23) (0.03, 0.25) (0.03, 0.03) (0.03, 0.05) (0.03, 0.08) (0.03, 0.10) (0.03, 0.13) (0.03, 0.15) (0.03, 0.18) (0.03, 0.20) (0.03, 0.23) (0.03, 0.25) (0.04, 0.03) (0.04, 0.05) (0.04, 0.08) (0.04, 0.10) (0.04, 0.13) (0.04, 0.15) (0.04, 0.18) (0.04, 0.20) (0.04, 0.23) (0.04, 0.25) (0.04, 0.03) (0.04, 0.05) (0.04, 0.08) (0.04, 0.10) (0.04, 0.13) (0.04, 0.15) (0.04, 0.18) (0.04, 0.20) (0.04, 0.23) (0.04, 0.25) (0.05, 0.03) (0.05, 0.05) (0.05, 0.08) (0.05, 0.10) (0.05, 0.13) (0.05, 0.15) (0.05, 0.18) (0.05, 0.20) (0.05, 0.23) (0.05, 0.25)

Cited by

Sorted by citations

Metadata

Authors
Schneider, S., Rusak, E., Eck, L., Bringmann, O., Brendel, W., & Bethge, M.
Venue
arXiv
Publisher
arXiv
Published
2020
DOI
10.48550/arxiv.2006.16971
arXiv
2006.16971
References
7
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review