CoChat
Sign Up
PreprintCited by 1Adversarial Robustness in Machine LearningAdvanced Neural Network ApplicationsGaussian Processes and Bayesian Inference

The DOI matches its DataCite record; 4 of the 4 references it cites were found in a registry or an index.

Uncertainty Quantification in Deep Residual Neural Networks

Open access
Wandzik, Lukasz, Garcia, Raul Vicente, Krüger, Jörg·2020·arXiv
DOI 10.48550/arxiv.2007.04905arXiv 2007.04905
Open this paper in CoChat — free accountOpen PDF

Abstract

Uncertainty quantification is an important and challenging problem in deep learning. Previous methods rely on dropout layers which are not present in modern deep architectures or batch normalization which is sensitive to batch sizes. In this work, we address the problem of uncertainty quantification in deep residual networks by using a regularization technique called stochastic depth. We show that training residual networks using stochastic depth can be interpreted as a variational approximation to the intractable posterior over the weights in Bayesian neural networks. We demonstrate that by sampling from a distribution of residual networks with varying depth and shared weights, meaningful uncertainty estimates can be obtained. Moreover, compared to the original formulation of residual networks, our method produces well-calibrated softmax probabilities with only minor changes to the network's structure. We evaluate our approach on popular computer vision datasets and measure the quality of uncertainty estimates. We also test the robustness to domain shift and show that our method is able to express higher predictive uncertainty on out-of-distribution samples. Finally, we demonstrate how the proposed approach could be used to obtain uncertainty estimates in facial verification applications.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2007.04905

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 31

  1. [1][1] A. Atanov, A. Ashukha, D. Molchanov, K. Neklyudov, and D. Vetrov. Uncertainty estimation via stochastic batch normalization. In International Conference on Learning Representations , 2018. 2, 4, 5, 6 -
  2. [2][2] C. M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics) . Springer-Verlag, Berlin, Heidelberg, 2006. 1 -
  3. [3][3] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37 , ICML'15, pages 16131622. JMLR.org, 2015. 2 -
  4. [4][4] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman. Vggface2: A dataset for recognising faces across pose and age. In International Conference on Automatic Face and Gesture Recognition , 2018. 5 -
  5. [5][5] Y. Chen, J. Li, H. Xiao, X. Jin, S. Yan, and J. Feng. Dual path networks. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA , pages 4470-4478, 2017. 2 -
  6. [6][6] M. Ferrara, A. Franco, and D. Maltoni. The magic passport. In IEEE International Joint Conference on Biometrics , pages 1-7, Sep. 2014. 6 -
  7. [7][7] Y. Gal. Uncertainty in Deep Learning . PhD thesis, University of Cambridge, 2016. 1, 2, 3, 7 -
  8. [9][9] K. Greff, R. K. Srivastava, and J. Schmidhuber. Highway and residual networks learn unrolled iterative estimation. CoRR , abs/1612.07771, 2016. 5 - · arXiv
  9. [10][10] R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker. Multi-pie. Image Vision Comput. , 28(5):807-813, May 2010. 5 -
  10. [12][12] D. Han, J. Kim, and J. Kim. Deep pyramidal residual networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages 6307-6315, 2017. 2 -
  11. [13][13] T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning . Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001. 4 -
  12. [15][15] J. M. Hern´ andez-Lobato and R. P. Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37 , ICML'15, pages 1861-1869. JMLR.org, 2015. 2 -
  13. [16][16] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages 2261-2269, 2017. 2 -
  14. [17][17] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In B. Leibe, J. Matas, N. Sebe, and M. Welling, editors, ECCV (4) , volume 9908 of Lecture Notes in Computer Science , pages 646-661. Springer, 2016. 1, 2, 4, 5 -
  15. [18][18] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37 , ICML'15, pages 448-456. JMLR.org, 2015. 1 -
  16. [19][19] M. E. Khan, D. Nielsen, V. Tangkaratt, W. Lin, Y. Gal, and A. Srivastava. Fast and scalable bayesian deep learning by weight-perturbation in adam. In ICML , volume 80 of JMLR Workshop and Conference Proceedings , pages 2616-2625. JMLR.org, 2018. 2 -
  17. [21][21] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30 , pages 64026413. Curran Associates, Inc., 2017. 2 -
  18. [22][22] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. P. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. Photo-realistic single image super-resolution using a generative adversarial network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages 105-114, 2017. 1 -
  19. [23][23] X. Li, S. Chen, X. Hu, and J. Yang. Understanding the disharmony between dropout and batch normalization by variance shift. CoRR , abs/1801.05134, 2018. 5 - · arXiv
  20. [24][24] C. Louizos and M. Welling. Multiplicative normalizing flows for variational bayesian neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , pages 2218-2227, 2017. 2, 5 -
  21. [25][25] M. P. Naeini, G. F. Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI , pages 2901-2907. AAAI Press, 2015. 5 -
  22. [26][26] R. M. Neal. Bayesian Learning for Neural Networks . Springer-Verlag, Berlin, Heidelberg, 1996. 2 -
  23. [27][27] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 , 2011. 5 -
  24. [28][28] P. O. Pinheiro, T.-Y . Lin, R. Collobert, and P. Doll ˜ A¡r. Learning to refine object segments. In ECCV , 2016. 1 -
  25. [29][29] J. C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In ADVANCES IN LARGE MARGIN CLASSIFIERS , pages 61-74. MIT Press, 1999. -
  26. [31][31] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research , 15(1):1929-1958, Jan. 2014. 1 -
  27. [32][32] M. Teye, H. Azizpour, and K. Smith. Bayesian uncertainty estimation for batch normalized deep networks. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 4907-4916, Stockholmsm¨ assan, Stockholm Sweden, 10-15 Jul 2018. PMLR. 2 -
  28. [33][33] J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler. Efficient object localization using convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015 , pages 648-656, 2015. 4, 5 -
  29. [34][34] L. Wandzik, R. V. Garcia, G. Kaeding, and X. Chen. Cnns under attack: On the vulnerability of deep neural networks based face recognition to image morphing. In C. Kraetzer, Y.Q. Shi, J. Dittmann, and H. J. Kim, editors, Digital Forensics and Watermarking , pages 121-135, Cham, 2017. Springer International Publishing. 5 -
  30. [35][35] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In L. Getoor and T. Scheffer, editors, ICML , pages 681-688. Omnipress, 2011. 2 -
  31. [36][36] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired imageto-image translation using cycle-consistent adversarial networkss. In Computer Vision (ICCV), 2017 IEEE International Conference on , 2017. 1

Cited by

Sorted by citations

See all 1

Metadata

Authors
Wandzik, L., Garcia, R. V., & Krüger, J.
Affiliations
Fraunhofer-Gesellschaft (DE)
ORCID
Krüger, Jörg
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2007.04905
arXiv
2007.04905
OpenAlex
W3042075083
Cited by
1
References
4
Citations by year
2021: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
Keywords
uncertainty quantification, ResNet, stochastic depth, Bayesian neural networks, variational approximation, out-of-distribution robustness, domain shift, face verification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review