CoChat
Sign Up
PreprintCited by 206Adversarial Robustness in Machine LearningDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

Measuring Robustness to Natural Distribution Shifts in Image Classification

Open access
Taori, Rohan, Dave, Achal, Shankar, Vaishaal, Carlini, Nicholas, Recht, Benjamin, Schmidt, Ludwig·2020·arXiv
DOI 10.48550/arxiv.2007.00644arXiv 2007.00644
Open this paper in CoChat — free accountOpen PDF

Abstract

We study how robust current ImageNet models are to distribution shifts arising from natural variations in datasets. Most research on robustness focuses on synthetic image perturbations (noise, simulated weather artifacts, adversarial examples, etc.), which leaves open how robustness on synthetic distribution shift relates to distribution shift arising in real data. Informed by an evaluation of 204 ImageNet models in 213 different test conditions, we find that there is often little to no transfer of robustness from current synthetic to natural distribution shift. Moreover, most current techniques provide no robustness to the natural distribution shifts in our testbed. The main exception is training on larger and more diverse datasets, which in multiple cases increases robustness, but is still far from closing the performance gaps. Our results indicate that distribution shifts arising in real data are currently an open research problem. We provide our testbed and data as a resource for future work at https://modestyachts.github.io/imagenet-testbed/ .

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2007.00644

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Taori et al. use a log-linear fit on ImageNet models to measure the expected accuracy on noisy test data given a new model's performance on clean test data.

    “Taori et al., use a log-linear fit on the set of (accuracy on noisy test data, accuracy on clean test data) points across a range of models trained on ImageNet to measure the expected accuracy on noisy test data given a new model's performance on clean test data.”

    From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [149]

    Appendix I.1 (p. 59) says the authors fit a predictor of shifted-test accuracy from standard-test accuracy using ImageNet-trained standard models. Section 3.1.2 (p. 5) includes image noise among the shifts studied. The method is not a log-linear fit as stated: it is a linear fit after transforming the accuracies with logit scaling. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Prior work has shown that for many tasks, ID and OOD performance can be highly correlated across different model architectures and hyperparameters.

    “Prior work has shown that for many tasks, ID and OOD performance can be highly correlated across different model architectures and hyperparameters (Taori et al., 2020; Liu et al., 2021b; Miller et al., 2021).”

    From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Taori et al., 2020

    Section 4.1 reports that standard test accuracy predicts accuracy under four natural distribution shifts, and Figure 1 describes an almost perfect relationship across 78 ImageNet models. This supports the ID–OOD correlation, but the paper’s evidence concerns ImageNet image classification, not many distinct tasks; the strongest relationship is also qualified as applying especially to models without robustness interventions. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Taori et al. propose to measure whether a model can offer higher accuracy on the noisy test set beyond what is expected from having higher accuracy on the original test set.

    “Taori et al., [149] therefore propose to measure whether a model can offer higher accuracy on the noisy test set beyond what is expected from having higher accuracy on the original test set .”

    From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [149]

  • MULTIBENCH extends the quantitative robustness measures proposed in Taori et al. to deal with multimodal imperfections across a range of imperfection levels.

    “To design such a metric, we extend the quantitative robustness measures proposed in Taori et al., [149] to deal with multimodal imperfections across a range of imperfection levels σ ∈ [ 0 . 0 , 1 . 0 ] .”

    From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [149]

  • Taori et al. introduce two notions of robustness: relative and effective robustness.

    “Taori et al., [149] introduce two notions of robustness: relative and effective robustness.”

    From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [149]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

Research works printed but not linked: 96

  1. [2][2] Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., and Katz, B. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurIPS), 2019. http://papers.nips.cc/paper/9142-objectnet-a-large-scale-bias-controlled-data set-for-pushing-the-limits-of-object-recognition-models.
  2. [4][4] Belinkov, Y. and Bisk, Y. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations (ICLR), 2018. https://arxiv.org/ abs/1711.02173. · arXiv
  3. [5][5] Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G., and Roli, F. Evasion attacks against machine learning at test time. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD), 2013. https://arxiv.org/abs/1708.06131. · arXiv
  4. [6][6] Blanchard, G., Lee, G., and Scott, C. Generalizing from several related classification tasks to a new unlabeled sample. In Advances in Neural Information Processing Systems (NIPS), 2011. https://papers.nips.cc/paper/4312-generalizing-from-several-related-clas sification-tasks-to-a-new-unlabeled-sample.
  5. [7][7] Borji, A. Objectnet dataset: Reanalysis and correction, 2020. https://arxiv.org/abs/2004 .02042.
  6. [8][8] Bras, R. L., Swayamdipta, S., Bhagavatula, C., Zellers, R., Peters, M. E., Sabharwal, A., and Choi, Y. Adversarial filters of dataset biases. In International Conference on Machine Learning (ICML), 2020. https://arxiv.org/abs/2002.04108. · arXiv
  7. [9][9] Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., Goodfellow, I., Madry, A., and Kurakin, A. On evaluating adversarial robustness, 2019. https://arxiv.org/ abs/1902.06705. · arXiv
  8. [10][10] Carmon, Y., Raghunathan, A., Schmidt, L., Liang, P., and Duchi, J. C. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv.org/abs/1905.13736. · arXiv
  9. [11][11] Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S., and Feng, J. Dual path networks. In Advances in Neural Information Processing Systems (NeurIPS), 2017. https://arxiv.org/abs/1707.016
  10. [29]29. 18 = http://papers.nips.cc/paper/9142-objectnet-a-large-scale-bias-controlled-dataset-for-pushing-the-limits-of-object-recognition-models http://papers.nips.cc/paper/9142-objectnet-a-large-scale-bias-controlled-dataset-for-pushing-the-limits-of-object-recognition-models https://dl.acm.org/doi/abs/10.1145/3313831.3376718 https://arxiv.org/abs/1711.02173 https://arxiv.org/abs/1711.02173 https://arxiv.org/abs/1708.06131 https://papers.nips.cc/paper/4312-generalizing-from-several-related-classification-tasks-to-a-new-unlabeled-sample https://papers.nips.cc/paper/4312-generalizing-from-several-related-classification-tasks-to-a-new-unlabeled-sample https://arxiv.org/abs/2004.02042 https://arxiv.org/abs/2004.02042 https://arxiv.org/abs/2002.04108 https://arxiv.org/abs/1902.06705 https://arxiv.org/abs/1902.06705 https://arxiv.org/abs/1905.13736 https://arxiv.org/abs/1707.01629 https://arxiv.org/abs/1707.01629 · DOI · Paper page
  11. [13][13] Cohen, J. M., Rosenfeld, E., and Kolter, J. Z. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning (ICML), 2019. https://arxiv. org/abs/1902.02918.
  12. [15][15] Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation policies from data. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. https://arxiv.org/abs/1805.09501. · arXiv
  13. [16][16] Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmentation with a reduced search space, 2019. https://arxiv.org/abs/1909.13719. · arXiv
  14. [18][18] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009. http://www.image-net.org/papers/imagenet_cvpr09.pdf.
  15. [20][20] DeVries, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout, 2017. https://arxiv.org/abs/1708.04552. · arXiv
  16. [21][21] Djolonga, J., Yung, J., Tschannen, M., Romijnders, R., Beyer, L., Kolesnikov, A., Puigcerver, J., Minderer, M., D'Amour, A., Moldovan, D., Gelly, S., Houlsby, N., Zhai, X., and Lucic, M. On robustness and transferability of convolutional neural networks, 2020. https://arxiv.or g/abs/2007.08558.
  17. [22][22] Duchi, J. and Namkoong, H. Learning models with uniform performance via distributionally robust optimization. To appear in the Annals of Statistics, 2018. https://arxiv.org/abs/18 10.08750.
  18. [23][23] Duchi, J. and Namkoong, H. Variance-based regularization with convex objectives. Journal of Machine Learning Research (JMLR), 2019. https://arxiv.org/abs/1610.02581. · arXiv
  19. [24][24] Duchi, J., Hashimoto, T., and Namkoong, H. Distributionally robust losses for latent covariate mixtures, 2019. https://arxiv.org/abs/2007.13982. · arXiv
  20. [25][25] Dulhanty, C. and Wong, A. Auditing ImageNet: Towards a model-driven framework for annotating demographic attributes of large-scale image datasets, 2019. https://arxiv.org/ abs/1905.01347. 19 https://arxiv.org/abs/1610.02357 https://arxiv.org/abs/1610.02357 https://arxiv.org/abs/1902.02918 https://arxiv.org/abs/1902.02918 https://www.excavating.ai/ https://arxiv.org/abs/1805.09501 https://arxiv.org/abs/1909.13719 https://ieeexplore.ieee.org/document/1467360 https://ieeexplore.ieee.org/document/1467360 http://www.image-net.org/papers/imagenet_cvpr09.pdf https://www.aclweb.org/anthology/N19-1423/ https://www.aclweb.org/anthology/N19-1423/ https://arxiv.org/abs/1708.04552 https://arxiv.org/abs/2007.08558 https://arxiv.org/abs/2007.08558 https://arxiv.org/abs/1810.08750 https://arxiv.org/abs/1810.08750 https://arxiv.org/abs/1610.02581 https://arxiv.org/abs/2007.13982 https://arxiv.org/abs/1905.01347 https://arxiv.org/abs/1905.01347 · arXiv
  21. [26][26] Dunn, M., Sagun, L., Higgins, M., Guney, V. U., Cirik, V., and Cho, K. Searchqa: A new q&a dataset augmented with context from a search engine, 2017. https://arxiv.org/abs/1704 .05179.
  22. [28][28] Engstrom, L., Tran, B., Tsipras, D., Schmidt, L., and Madry, A. Exploring the landscape of spatial robustness. In International Conference on Machine Learning (ICML), 2019. https: //arxiv.org/abs/1712.02779.
  23. [29][29] Fawzi, A. and Frossard, P. Manitest: Are classifiers really invariant? In BMVC, 2015. https://arxiv.org/abs/1507.06535. · arXiv
  24. [31][31] Ford, N., Gilmer, J., Carlini, N., and Cubuk, E. D. Adversarial examples are a natural consequence of test error in noise. In International Conference on Machine Learning (ICML), 2019. http://arxiv.org/abs/1901.10513. · arXiv
  25. [32][32] Galloway, A., Tanay, T., and Taylor, G. W. Adversarial training versus weight decay, 2018. https://arxiv.org/abs/1804.03308. · arXiv
  26. [33][33] Geirhos, R., Temme, C. R. M., Rauber, J., Schütt, H. H., Bethge, M., and Wichmann, F. A. Generalisation in humans and deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2018. https://papers.nips.cc/paper/7982-generalisati on-in-humans-and-deep-neural-networks.
  27. [35][35] Gu, K., Yang, B., Ngiam, J., Le, Q., and Shlens, J. Using videos to evaluate image model robustness. In SafeML workshop International Conference on Learning Representations (ICLR), 2019. https://arxiv.org/abs/1904.10076. · arXiv
  28. [36][36] Gulrajani, I. and Lopez-Paz, D. In search of lost domain generalization, 2020. https: //arxiv.org/abs/2007.01434.
  29. [38][38] Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019. https://arxiv.org/abs/1903.12261. · arXiv
  30. [39][39] Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples, 2019. https://arxiv.org/abs/1907.07174. 20 https://arxiv.org/abs/1704.05179 https://arxiv.org/abs/1704.05179 https://github.com/MadryLab/robustness https://arxiv.org/abs/1712.02779 https://arxiv.org/abs/1712.02779 https://arxiv.org/abs/1507.06535 https://ieeexplore.ieee.org/document/5255236 http://arxiv.org/abs/1901.10513 https://arxiv.org/abs/1804.03308 https://papers.nips.cc/paper/7982-generalisation-in-humans-and-deep-neural-networks https://papers.nips.cc/paper/7982-generalisation-in-humans-and-deep-neural-networks https://arxiv.org/abs/1811.12231 https://arxiv.org/abs/1904.10076 https://arxiv.org/abs/2007.01434 https://arxiv.org/abs/2007.01434 https://arxiv.org/abs/1512.03385 https://arxiv.org/abs/1512.03385 https://arxiv.org/abs/1903.12261 https://arxiv.org/abs/1907.07174 · arXiv
  31. [40][40] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of-distribution generalization, 2020. https://arxiv.org/abs/2006 .16241.
  32. [41][41] Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. AugMix: A simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representations (ICLR), 2020. https://arxiv.org/abs/1912.02781. · arXiv
  33. [42][42] Hu, J., Shen, L., Albanie, S., Sun, G., and Wu, E. Squeeze-and-excitation networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018. https://arxiv.or g/abs/1709.01507.
  34. [43][43] Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017. https: //arxiv.org/abs/1608.06993.
  35. [44][44] Huang, X. and Belongie, S. Arbitrary style transfer in real-time with adaptive instance normalization. International Conference on Computer Vision (ICCV), 2017. https://arxiv. org/abs/1703.06868.
  36. [45][45] Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., and Keutzer, K. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size, 2016. https://arxiv.org/abs/1602.07360. · arXiv
  37. [46][46] Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015. https://arxiv.org/abs/1502.03167. · arXiv
  38. [47][47] Kang, D., Sun, Y., Brown, T., Hendrycks, D., and Steinhardt, J. Transfer of adversarial robustness between perturbation types, 2019. https://arxiv.org/abs/1905.01034. · arXiv
  39. [48][48] Kang, D., Sun, Y., Hendrycks, D., Brown, T., and Steinhardt, J. Testing robustness against unforeseen adversaries, 2019. https://arxiv.org/abs/1908.08016. · arXiv
  40. [49][49] Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning, 2019. https://arxiv.org/abs/1912.1 1370.
  41. [50][50] Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS), 2012. https: //papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional- neural-networks.
  42. [51][51] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., and et al. The open images dataset v4. International Journal of Computer Vision (IJCV), 2020. https://arxiv.org/abs/1811.00982. · arXiv
  43. [53][53] Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., and Murphy, K. Progressive neural architecture search. In European Conference on Computer Vision (ECCV), 2018. https://arxiv.org/abs/1712.00559. · arXiv
  44. [55][55] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018. https://arxiv.org/abs/1706.06083. · arXiv
  45. [56][56] Mahajan, D. K., Girshick, R. B., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and van der Maaten, L. Exploring the limits of weakly supervised pretraining. In European Conference on Computer Vision (ECCV), 2018. https://arxiv.org/abs/1805.00932. · arXiv
  46. [57][57] Maini, P., Wong, E., and Kolter, J. Z. Adversarial robustness against the union of multiple perturbation models, 2019. https://arxiv.org/abs/1909.04068. · arXiv
  47. [59][59] Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T. Model cards for model reporting. In Conference on Fairness, Accountability, and Transparency (FAT), 2019. https://arxiv.org/abs/1810.03993. · arXiv
  48. [60][60] Muandet, K., Balduzzi, D., and Schölkopf, B. Domain generalization via invariant feature representation. In International Conference on Machine Learning (ICML), 2013. https: //arxiv.org/abs/1301.2115.
  49. [61][61] Najafi, A., Maeda, S.-i., Koyama, M., and Miyato, T. Robustness to adversarial perturbations in learning from incomplete data. In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv.org/abs/1905.13021. · arXiv
  50. [62][62] Oren, Y., Sagawa, S., Hashimoto, T. B., and Liang, P. Distributionally robust language modeling. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. https://arxiv.org/abs/1909.02060. · arXiv
  51. [63][63] Orhan, A. E. Robustness properties of facebook's resnext wsl models, 2019. https://arxiv. org/abs/1907.07640.
  52. [66][66] Raghunathan, A., Steinhardt, J., and Liang, P. Certified defenses against adversarial examples. In International Conference on Learning Representations (ICLR), 2018. https://arxiv.org/ abs/1801.09344. · arXiv
  53. [67][67] Real, E., Shlens, J., Mazzocchi, S., Pan, X., and Vanhoucke, V. Youtube-boundingboxes: A large high-precision human-annotated data set for object detection in video. In CVPR, 2017.
  54. [68][68] Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML), 2019. https: //arxiv.org/abs/1902.10811.
  55. [69][69] Rowley, H. A., Baluja, S., and Kanade, T. Human face detection in visual scenes. In Advances in Neural Information Processing Systems (NIPS). 1996. https://papers.nips.cc/paper/1 168-human-face-detection-in-visual-scenes.
  56. [70][70] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 2015. https://arxiv.or g/abs/1409.0575.
  57. [71][71] Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations (ICLR), 2020. https://arxiv.org/ abs/1911.08731. · arXiv
  58. [72][72] Salman, H., Yang, G., Li, J., Zhang, P., Zhang, H., Razenshteyn, I., and Bubeck, S. Provably robust deep learning via adversarially trained smoothed classifiers. In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv.org/abs/1906.04584. · arXiv
  59. [73][73] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018. https://arxiv.org/abs/1801.04381. · arXiv
  60. [74][74] Shafahi, A., Najibi, M., Ghiasi, A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G., and Goldstein, T. Adversarial training for free! In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv.org/abs/1904.12843. · arXiv
  61. [75][75] Shankar, S., Halpern, Y., Breck, E., Atwood, J., Wilson, J., and Sculley, D. No classification without representation: Assessing geodiversity issues in open data sets for the developing world, 2017. https://arxiv.org/abs/1711.08536. · arXiv
  62. [76][76] Shankar, V., Dave, A., Roelofs, R., Ramanan, D., Recht, B., and Schmidt, L. Do image classifiers generalize across time?, 2019. https://arxiv.org/abs/1906.02168. · arXiv
  63. [77][77] Shankar, V., Roelofs, R., Mania, H., Fang, A., Recht, B., and Schmidt, L. Evaluating machine accuracy on imagenet. In International Conference on Machine Learning (ICML), 2020.
  64. [78][78] Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015. https: //arxiv.org/abs/1409.1556. 23 https://arxiv.org/abs/1801.09344 https://arxiv.org/abs/1801.09344 https://arxiv.org/abs/1902.10811 https://arxiv.org/abs/1902.10811 https://papers.nips.cc/paper/1168-human-face-detection-in-visual-scenes https://papers.nips.cc/paper/1168-human-face-detection-in-visual-scenes https://arxiv.org/abs/1409.0575 https://arxiv.org/abs/1409.0575 https://arxiv.org/abs/1911.08731 https://arxiv.org/abs/1911.08731 https://arxiv.org/abs/1906.04584 https://arxiv.org/abs/1801.04381 https://arxiv.org/abs/1904.12843 https://arxiv.org/abs/1711.08536 https://arxiv.org/abs/1906.02168 https://arxiv.org/abs/1409.1556 https://arxiv.org/abs/1409.1556 · arXiv
  65. [79][79] Sinha, A., Namkoong, H., Volpi, R., and Duchi, J. Certifying some distributional robustness with principled adversarial training. In International Conference on Learning Representations (ICLR), 2018. https://arxiv.org/abs/1710.10571. · arXiv
  66. [80][80] Sperber, M., Niehues, J., and Waibel, A. Toward robust neural machine translation for noisy input sequences. In International Workshop on Spoken Language Translation (IWSLT), 2017. http://workshop2017.iwslt.org/downloads/P04-Paper.pdf.
  67. [81][81] Srivastava, M., Hashimoto, T., and Liang, P. Robustness to spurious correlations via human annotations. In International Conference on Machine Learning (ICML), 2020. https://arxi v.org/abs/2007.06661.
  68. [82][82] Sun, C., Shrivastava, A., Singh, S., and Gupta, A. Revisiting unreasonable effectiveness of data in deep learning era. In International Conference on Computer Vision (ICCV), 2017. https://arxiv.org/abs/1707.02968. · arXiv
  69. [83][83] Sung, K. . and Poggio, T. Example-based learning for view-based human face detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998. https://ieeexplore.ieee. org/document/655648.
  70. [84][84] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fer- gus, R. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014. https://arxiv.org/abs/1312.6199. · arXiv
  71. [85][85] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015. https://arxiv.org/abs/1409.4842v1. · arXiv
  72. [86][86] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. https://arxiv.org/abs/1512.00567. · arXiv
  73. [87][87] Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI Conference on Artificial Intelligence (AAAI), 2017. https://arxiv.org/abs/1602.07261. · arXiv
  74. [88][88] Tan, M. and Le, Q. V. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (ICML), 2019. https://arxiv.org/abs/ 1905.11946.
  75. [90][90] Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. When robustness doesn't promote robustness: Synthetic vs. natural distribution shifts on imagenet, 2019. https://openreview.net/pdf?id=HyxPIyrFvH.
  76. [92][92] Touvron, H., Vedaldi, A., Douze, M., and Jégou, H. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv.or g/abs/1906.06423.
  77. [93][93] Tramer, F. and Boneh, D. Adversarial training and robustness for multiple perturbations. In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv.org/ abs/1904.13000. · arXiv
  78. [94][94] Uesato, J., Alayrac, J.-B., Huang, P.-S., Stanforth, R., Fawzi, A., and Kohli, P. Are labels required for improving adversarial robustness? In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv.org/abs/1905.13725. · arXiv
  79. [95][95] Wang, H., Ge, S., Xing, E. P., and Lipton, Z. C. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv.org/abs/1905.13549. · arXiv
  80. [96][96] Wong, E., Schmidt, F. R., and Kolter, J. Z. Wasserstein adversarial examples via projected sinkhorn iterations. In International Conference on Machine Learning (ICML), 2019. https: //arxiv.org/abs/1902.07906.
  81. [97][97] Wu, B., Chen, W., Fan, Y., Zhang, Y., Hou, J., Liu, J., and Zhang, T. Tencent ml- images: A large-scale multi-label image database for visual representation learning. IEEE Access, 7:172683–172693, 2019. ISSN 2169-3536. doi: 10.1109/access.2019.2956775. URL http://dx.doi.org/10.1109/ACCESS.2019.2956775. · DOI
  82. [98][98] Wu, B., Chen, W., Fan, Y., Zhang, Y., Hou, J., Liu, J., and Zhang, T. Tencent ml-images: A large-scale multi-label image database for visual representation learning. IEEE Access, 7, 2019.
  83. [100][100] Xie, C., Tan, M., Gong, B., Wang, J., Yuille, A., and Le, Q. V. Adversarial examples improve image recognition, 2019. https://arxiv.org/abs/1911.09665. · arXiv
  84. [101][101] Xie, C., Wu, Y., van der Maaten, L., Yuille, A., and He, K. Feature denoising for improving adversarial robustness. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. https://arxiv.org/abs/1812.03411. · arXiv
  85. [102][102] Xie, Q., Luong, M.-T., Hovy, E., and Le, Q. V. Self-training with noisy student improves imagenet classification. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. https://arxiv.org/abs/1911.04252. · arXiv
  86. [103][103] Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. Aggregated residual transformations for deep neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. https://arxiv.org/abs/1611.05431. · arXiv
  87. [104][104] Yalniz, I. Z., Jégou, H., Chen, K., Paluri, M., and Mahajan, D. Billion-scale semi-supervised learning for image classification, 2019. https://arxiv.org/abs/1905.00546. · arXiv
  88. [106][106] Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. https://arxiv.org/ab s/1809.09600.
  89. [107][107] Yin, D., Lopes, R. G., Shlens, J., Cubuk, E. D., and Gilmer, J. A fourier perspective on model robustness in computer vision. In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv.org/abs/1906.08988. · arXiv
  90. [108][108] Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In International Conference on Computer Vision (ICCV), 2019. https://arxiv.org/abs/1905.04899. · arXiv
  91. [111][111] Zellers, R., Bisk, Y., Schwartz, R., and Choi, Y. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Empirical Methods in Natural Language Processing (EMNLP), 2018. https://www.aclweb.org/anthology/D18-1009/.
  92. [112][112] Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., Beyer, L., Bachem, O., Tschannen, M., Michalski, M., Bousquet, O., Gelly, S., and Houlsby, N. The visual task adaptation benchmark, 2019. https://arxiv.org/abs/1910.04867. · arXiv
  93. [113][113] Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), 2017. https: //arxiv.org/abs/1710.09412.
  94. [114][114] Zhang, H., Yu, Y., Jiao, J., Xing, E., Ghaoui, L. E., and Jordan, M. I. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning (ICML), 2019. http://proceedings.mlr.press/v97/zhang19p.html.
  95. [115][115] Zhang, R. Making convolutional networks shift-invariant again. In International Conference on Machine Learning (ICML), 2019. https://arxiv.org/abs/1904.11486. · arXiv
  96. [116][116] Zhang, X., Li, Z., Loy, C. C., and Lin, D. Polynet: A pursuit of structural diversity in very deep networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. https://arxiv.org/abs/1611.05725. · arXiv

Cited by

Sorted by citations

Authors
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., & Schmidt, L.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2007.00644
arXiv
2007.00644
OpenAlex
W3040002795
Cited by
206
References
18
Citations by year
2025: 34 · 2024: 46 · 2023: 66 · 2022: 32 · 2021: 24 · 2020: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
distribution shift, robustness, image classification, natural variation, adversarial examples, data diversity, robustness analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review