CoChat
Sign Up
PreprintCited by 12Adversarial Robustness in Machine LearningGaussian Processes and Bayesian InferenceNeural Networks and Applications

The DOI matches its DataCite record; 5 of the 5 references it cites were found in a registry or an index.

Variance Networks: When Expectation Does Not Meet Your Expectations

Open access
Neklyudov, Kirill, Molchanov, Dmitry, Ashukha, Arsenii, Vetrov, Dmitry·2018·arXiv
DOI 10.48550/arxiv.1803.03764arXiv 1803.03764
Open this paper in CoChat — free accountOpen PDF

Abstract

Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging. In this paper, we introduce variance layers, a different kind of stochastic layers. Each weight of a variance layer follows a zero-mean distribution and is only parameterized by its variance. We show that such layers can learn surprisingly well, can serve as an efficient exploration tool in reinforcement learning tasks and provide a decent defense against adversarial attacks. We also show that a number of conventional Bayesian neural networks naturally converge to such zero-mean posteriors. We observe that in these cases such zero-mean parameterization leads to a much better training objective than conventional parameterizations where the mean is being learned.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1803.03764

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In Bragman et al.'s technique, parameter uncertainty was estimated through approximate Bayesian inference.

    “Parameter uncertainty was estimated through a approximate Bayesian inference whilst intrinsic uncertainty was modelled using a heteroscedastic noise technique.”

    From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [263]

    Section 4.1 (p. 5) describes approximate Bayesian inference over model weights, supporting the parameter-uncertainty part in general. Our copy does not identify this as Bragman et al.’s technique or describe its use of heteroscedastic noise to model intrinsic uncertainty. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Multitask learning demonstrated efficacy for MR-only radiotherapy planning by jointly automating contouring of organs-at-risk and simulating synthetic CT from MRI scans.

    “For example, MTL demonstrated its efficacy for MR-only radiotherapy planning as it can jointly automate contour of organs-at-risk - a segmentation task - and simulate a synthetic CT (synCT) scan - a regression task from MRI scans.”

    From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [263]

    The Experiments section (p. 7) describes classification and reinforcement-learning evaluations, not MR-only radiotherapy planning. Searches for organs-at-risk segmentation and MRI-to-synthetic-CT generation found no passage addressing the claimed multitask application. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Bragman et al. suggested a probabilistic deep-learning technique to estimate parameter and intrinsic uncertainty.

    “Bragman et al. [263] suggested utilizing a probabilistic deep-learning technique to estimate the parameter and intrinsic uncertainty.”

    From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [263]

    The Introduction (p. 1) says stochastic methods can estimate uncertainty, and §4.1 (p. 5) discusses approximating a posterior over model weights. Searches for Bragman and intrinsic uncertainty found no passage attributing to Bragman a technique for estimating both parameter and intrinsic uncertainty. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • In Bragman et al.'s technique, intrinsic uncertainty was modelled using a heteroscedastic noise technique.

    “Parameter uncertainty was estimated through a approximate Bayesian inference whilst intrinsic uncertainty was modelled using a heteroscedastic noise technique.”

    From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [263]

    The searched passages do not mention Bragman or describe modelling intrinsic uncertainty with heteroscedastic noise. Section 4.1 (p. 5) discusses approximate inference over network weights, but that does not establish the claim about Bragman's technique. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Bragman et al. demonstrated competitive performance in segmentation and regression of prostate cancer scans.

    “They demonstrated competitive performance in the segmentation and regression of prostate cancer scans.”

    From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [263]

    Section 6 (p. 7) describes experiments on classification and reinforcement learning, and Section 6.1 (p. 8) reports results for MNIST and CIFAR datasets. Searches for Bragman, prostate cancer scans, segmentation, and regression found no passage addressing the attributed result. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics , 1983.
  2. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016. · arXiv
  3. Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295 , 2017. · arXiv
  4. Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. arXiv preprint arXiv:1802.10026 , 2018. · arXiv
  5. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, 2016. http: //www.deeplearningbook.org .
  6. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014. · arXiv
  7. Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Sch¨ olkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research , 13(Mar):723-773, 2012.
  8. Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems , pp. 1135-1143, 2015.
  9. Jiri Hron, Alexander G de G Matthews, and Zoubin Ghahramani. Variational gaussian dropout is not bayesian. arXiv preprint arXiv:1711.02989 , 2017. · arXiv
  10. Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013. · arXiv
  11. Diederik P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems , pp. 2575-2583, 2015.
  12. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems , pp. 6405-6416, 2017.
  13. David JC MacKay et al. Bayesian nonlinear modeling for the prediction competition. ASHRAE transactions , 100(2):1053-1062, 1994.
  14. Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. arXiv preprint arXiv:1802.10501 , 2018. · arXiv
  15. Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. arXiv preprint arXiv:1701.05369 , 2017. · arXiv
  16. Radford M Neal. Bayesian learning for neural networks , volume 118. Springer Science & Business Media, 1996.
  17. Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov. Structured bayesian pruning via log-normal multiplicative noise. In Advances in Neural Information Processing Systems , pp. 6778-6787, 2017.
  18. Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv:1706.01905 , 2017. · arXiv
  19. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082 , 2014. · arXiv
  20. Samuel L. Smith and Quoc V. Le. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations , 2018. URL https:// openreview.net/forum?id=BJij4yg0Z .
  21. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research , 15(1):1929-1958, 2014.
  22. Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances in neural information processing systems , pp. 1038-1044, 1996.
  23. Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems , pp. 1057-1063, 2000.
  24. Michalis Titsias and Miguel L´ azaro-Gredilla. Doubly stochastic variational bayes for non-conjugate inference. Proceedings of The 31st International Conference on Machine Learning , 32:19711979, 2014.
  25. Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International Conference on Machine Learning , pp. 1058-1066, 2013.
  26. Sida Wang and Christopher Manning. Fast dropout training. In international conference on machine learning , pp. 118-126, 2013.
  27. Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11) , pp. 681-688, 2011.
  28. Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems , pp. 2074-2082, 2016.
  29. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Reinforcement Learning , pp. 5-32. Springer, 1992.

Cited by

Sorted by citations

Metadata

Authors
Neklyudov, K., Molchanov, D., Ashukha, A., & Vetrov, D.
Affiliations
National Research University Higher School of Economics (RU)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
Pages
1-16
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1803.03764
arXiv
1803.03764
OpenAlex
W2793058334
Cited by
12
References
5
Citations by year
2024: 1 · 2023: 1 · 2022: 4 · 2021: 2 · 2019: 3 · 2018: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-sa/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Learning Representations
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
stochastic neural networks, Bayesian neural networks, adversarial robustness, uncertainty estimation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review