CoChat
Sign Up
PreprintCited by 157Gaussian Processes and Bayesian InferenceAdversarial Robustness in Machine LearningAdvanced Neural Network Applications

The DOI matches its DataCite record.

DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networks

Open access
Mobiny, Aryan, Nguyen, Hien V., Moulik, Supratik, Garg, Naveen, Wu, Carol C.·2019·arXiv
DOI 10.48550/arxiv.1906.04569arXiv 1906.04569PMC PMC7943811
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep neural networks (DNNs) have achieved state-of-the-art performances in many important domains, including medical diagnosis, security, and autonomous driving. In these domains where safety is highly critical, an erroneous decision can result in serious consequences. While a perfect prediction accuracy is not always achievable, recent work on Bayesian deep networks shows that it is possible to know when DNNs are more likely to make mistakes. Knowing what DNNs do not know is desirable to increase the safety of deep learning technology in sensitive applications. Bayesian neural networks attempt to address this challenge. However, traditional approaches are computationally intractable and do not scale well to large, complex neural network architectures. In this paper, we develop a theoretical framework to approximate Bayesian inference for DNNs by imposing a Bernoulli distribution on the model weights. This method, called MC-DropConnect, gives us a tool to represent the model uncertainty with little change in the overall model structure or computational cost. We extensively validate the proposed algorithm on multiple network architectures and datasets for classification and semantic segmentation tasks. We also propose new metrics to quantify the uncertainty estimates. This enables an objective comparison between MC-DropConnect and prior approaches. Our empirical results demonstrate that the proposed framework yields significant improvement in both prediction accuracy and uncertainty estimation quality compared to the state of the art.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1906.04569

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • DNNs have been effectively employed in a wide variety of machine learning tasks and have achieved state-of-the-art performance in domains such as bioinformatics, NLP, speech recognition, and computer vision.

    “Deep neural networks (DNNs) have been effectively employed in a wide variety of machine learning tasks and have achieved state-of-the-art performance in different domains such as bioinformatics, natural language processing (NLP), speech recognition and computer vision [201], [202].”

    From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [202]

    The Abstract (page not provided) says DNNs have revolutionized fields including language processing, computer vision, and biology, and have achieved state-of-the-art performance in several domains. It does not mention speech recognition or bioinformatics specifically, or establish state-of-the-art performance in each of the four domains named in the claim. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 39

  1. Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1097-1105 (2012).
  2. He, K., Gkioxari, G., Dollár, P. & Girshick, R. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision 2961-2969 (2017).
  3. Mnih, V. et al. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013). · arXiv
  4. Mobiny, A., Yuan, P., Cicalese, P. A. & Van Nguyen, H. Decaps: Detail-oriented capsule networks. In International Conference on Medical Image Computing and Computer-Assisted Intervention 148-158 (Springer, 2020).
  5. Mobiny, A. & Van Nguyen, H. Fast capsnet for lung cancer screening. In International Conference on Medical Image Computing and Computer-Assisted Intervention 741-749 (Springer, 2018).
  6. Cicalese, P. A. et al. Kidney level lupus nephritis classification using uncertainty guided Bayesian convolutional neural networks. IEEE J. Biomed. Health Inform. 25, 315-324 (2020).
  7. Mobiny, A. et al. Memory-augmented capsule network for adaptable lung nodule classification. IEEE Trans. Med. Imaging (2021).
  8. Gal, Y. Uncertainty in Deep Learning (University of Cambridge, 2016).
  9. Mobiny, A., Singh, A. & Van Nguyen, H. Risk-aware machine learning classifier for skin lesion diagnosis. J. Clin. Med. 8, 1241 (2019).
  10. Neal, R. M. Bayesian Learning for Neural Networks Vol. 118 (Springer, 2012).
  11. Blei, D. M., Kucukelbir, A. & McAuliffe, J. D. Variational inference: A review for statisticians. J. Am. Stat. Assoc. 112, 859-877 (2017).
  12. Graves, A. Practical variational inference for neural networks. Advances in neural information processing systems 2348-2356, (2011).
  13. Neal, R. M. Bayesian learning via stochastic dynamics. In Advances in Neural Information Processing Systems 475-482 (1993).
  14. Gal, Y. & Ghahramani, Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning 1050-1059 (2016).
  15. Gal, Y. & Ghahramani, Z. Bayesian convolutional neural networks with Bernoulli approximate variational inference. arXiv preprint arXiv:1506.02158 (2015). · arXiv
  16. Kendall, A., Badrinarayanan, V. & Cipolla, R. Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder archi- tectures for scene understanding. arXiv preprint arXiv:1511.02680 (2015). · arXiv
  17. Kingma, D. P., Salimans, T. & Welling, M. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems 2575-2583 (2015).
  18. Gal, Y., Hron, J. & Kendall, A. Concrete dropout. In Advances in Neural Information Processing Systems 3581-3590 (2017).
  19. Boluki, S., Ardywibowo, R., Dadaneh, S. Z., Zhou, M. & Qian, X. Learnable bernoulli dropout for Bayesian deep learning. arXiv preprint arXiv:2002.05155 (2020). · arXiv
  20. Louizos, C. & Welling, M. Multiplicative normalizing flows for variational Bayesian neural networks. In Proceedings of the 34th International Conference on Machine Learning Vol. 70, 2218-2227 (JMLR. org, 2017).
  21. Lakshminarayanan, B., Pritzel, A. & Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 6402-6413 (2017).
  22. DeVries, T. & Taylor, G. W. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint arXiv:1802.04865 (2018). · arXiv
  23. Kendall, A. & Gal, Y. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Infor- mation Processing Systems 5574-5584 (2017).
  24. Kohl, S. et al. A probabilistic u-net for segmentation of ambiguous images. In Advances in Neural Information Processing Systems 6965-6975 (2018).
  25. Guzman-Rivera, A., Batra, D. & Kohli, P. Multiple choice learning: Learning to produce multiple structured outputs. In Advances in Neural Information Processing Systems 1799-1807 (2012).
  26. Lee, S., Purushwalkam, S., Cogswell, M., Crandall, D. & Batra, D. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314 (2015). · arXiv
  27. Rupprecht, C. et al. Learning in an uncertain world: Representing ambiguity through multiple hypotheses. In Proceedings of the IEEE International Conference on Computer Vision 3591-3600 (2017).
  28. MacKay, D. J. Probable networks and plausible predictions'a review of practical Bayesian methods for supervised neural networks. Netw. Comput. Neural Syst. 6, 469-505 (1995).
  29. Blundell, C., Cornebise, J., Kavukcuoglu, K. & Wierstra, D. Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05424 (2015). · arXiv
  30. Gal, Y. & Ghahramani, Z. Dropout as a Bayesian approximation: Insights and applications. In Deep Learning Workshop, ICML Vol. 1, 2 (2015).
  31. Wan, L., Zeiler, M., Zhang, S., Le Cun, Y. & Fergus, R. Regularization of neural networks using dropconnect. In International Conference on Machine Learning 1058-1066 (2013).
  32. Smith, L. & Gal, Y. Understanding measures of uncertainty for adversarial example detection. arXiv preprint arXiv:1803.08533 (2018). · arXiv
  33. Abadi, M. et al. Tensorflow: A system for large-scale machine learning. In 12th{USENIX}Symposium on Operating Systems Design and Implementation({OSDI}16) 265-283 (2016).
  34. LeCun, Y. et al. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278-2324 (1998).
  35. Hendrycks, D. & Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136 (2016). · arXiv
  36. Long, J., Shelhamer, E. & Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 3431-3440 (2015).
  37. Cordts, M. et al. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 3213-3223 (2016).
  38. Paszke, A., Chaurasia, A., Kim, S. & Culurciello, E. Enet: A deep neural network architecture for real-time semantic segmentation. arXiv preprint arXiv:1606.02147 (2016). · arXiv
  39. Milletari, F., Navab, N. & Ahmadi, S.-A. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D Vision (3DV) 565-571 (IEEE, 2016).

Cited by

Sorted by citations

Metadata

Authors
Mobiny, A., Nguyen, H. V., Moulik, S., Garg, N., & Wu, C. C.
Affiliations
  • University of Houston (US)
  • The University of Texas MD Anderson Cancer Center (US)
Corresponding author
Mobiny, Aryan
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
11
Issue
1
Pages
5458-5458
ISSN
2045-2322 (ISSN-L)
DOI
10.48550/arxiv.1906.04569
arXiv
1906.04569
PMC
PMC7943811
OpenAlex
W2953374819
Cited by
157
References
0
Citations by year
2026: 16 · 2025: 39 · 2024: 32 · 2023: 26 · 2022: 16 · 2021: 17 · 2020: 9 · 2019: 2
FWCI
7.97
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
model uncertainty estimation, approximate Bayesian inference, uncertainty measures, classification, semantic segmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review