CoChat
Sign Up
Cited by 247Gaussian Processes and Bayesian InferenceBayesian Modeling and Causal InferenceTopic Modeling

A Survey on Bayesian Deep Learning

Open access
Wang, Hao, Yeung, Dit-Yan·2021·ACM Computing Surveys
DOI 10.1145/3409383
Open this paper in CoChat — free accountOpen PDF

Abstract

A comprehensive artificial intelligence system needs to not only perceive the environment with different “senses” (e.g., seeing and hearing) but also infer the world’s conditional (or even causal) relations and corresponding uncertainty. The past decade has seen major advances in many perception tasks, such as visual object recognition and speech recognition, using deep learning models. For higher-level inference, however, probabilistic graphical models with their Bayesian nature are still more powerful and flexible. In recent years, Bayesian deep learning has emerged as a unified probabilistic framework to tightly integrate deep learning and Bayesian models. 1 In this general framework, the perception of text or images using deep learning can boost the performance of higher-level inference and, in turn, the feedback from the inference process is able to enhance the perception of text or images. This survey provides a comprehensive introduction to Bayesian deep learning and reviews its recent applications on recommender systems, topic models, control, and so on. We also discuss the relationship and differences between Bayesian deep learning and other related topics, such as Bayesian treatment of neural networks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3409383

Cites

52 of 52 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Hybrid algorithms still lack capabilities such as reasoning or modelling the uncertainty between input data and predictions.

    “However, these algorithms still lack capabilities such as reasoning [129] or modelling the uncertainty between the input data and the predictions made [130].”

    From A survey on deep learning-based monocular spacecraft pose estimation: Current state, limitations and prospects (2023) · cites this paper as [130]

    The citation [130] is attached to the uncertainty claim. In the Introduction (page unavailable), the survey says that Bayesian deep learning unifies deep learning and probabilistic graphical models to model uncertainty effectively; it does not say hybrid algorithms generally lack that capability. It identifies a narrower limitation: separately training the two components assumes no uncertainty when they exchange information. (LLM, conf=0.82)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 83

  1. Anoop Korattikara Balan, Vivek Rathod, Kevin P. Murphy, and Max Welling. 2015. Bayesian dark knowledge. In Proceedings of the NIPS. 3420-3428.
  2. Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. 2013. Generalized denoising auto-encoders as genera- tive models. In Proceedings of the NIPS. 899-907.
  3. David Blei and John Lafferty. 2006. Correlated topic models. In Proceedings of the NIPS. 147-154.
  4. David M. Blei and John D. Lafferty. 2006. Dynamic topic models. In Proceedings of the ICML. ACM, 113-120.
  5. Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. Weight uncertainty in neural network. In Proceedings of the ICML. 1613-1622.
  6. Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. 2016. Importance weighted autoencoders. In Proceedings of the ICLR.
  7. Minmin Chen, Kilian Q. Weinberger, Fei Sha, and Yoshua Bengio. 2014. Marginalized denoising auto-encoders for nonlinear representations. In Proceedings of the ICML. 1476-1484.
  8. Minmin Chen, Zhixiang Eddie Xu, Kilian Q. Weinberger, and Fei Sha. 2012. Marginalized denoising autoencoders for domain adaptation. In Proceedings of the ICML.
  9. Tianqi Chen, Emily B. Fox, and Carlos Guestrin. 2014. Stochastic Gradient Hamiltonian Monte Carlo. In Proceedings of the ICML. 1683-1691.
  10. Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C. Courville, and Yoshua Bengio. 2015. A re- current latent variable model for sequential data. In Proceedings of the NIPS. 2980-2988.
  11. Yulai Cong, Bo Chen, Hongwei Liu, and Mingyuan Zhou. 2017. Deep latent dirichlet allocation with topic-layer- adaptive stochastic gradient riemannian MCMC. In Proceedings of the ICML. 864-873.
  12. Andreas Doerr, Christian Daniel, Martin Schiegg, Duy Nguyen-Tuong, Stefan Schaal, Marc Toussaint, and Sebastian Trimpe. 2018. Probabilistic recurrent state-space models. In Proceedings of the ICML. 1279-1288.
  13. S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E. Hinton. 2016. Attend, infer, repeat: Fast scene understanding with generative models. In Proceedings of the NIPS. 3225-3233.
  14. Yarin Gal and Zoubin Ghahramani. 2015. Dropout as a Bayesian approximation: Insights and applications. In Pro- ceedings of the ICML.
  15. Zhe Gan, Changyou Chen, Ricardo Henao, David E. Carlson, and Lawrence Carin. 2015. Scalable deep Poisson factor analysis for topic modeling. In Proceedings of the ICML. 1823-1832.
  16. Zhe Gan, Ricardo Henao, David E. Carlson, and Lawrence Carin. 2015. Learning deep sigmoid belief networks with data augmentation. In Proceedings of the AISTATS.
  17. Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. 2019. Probabilistic forecasting with spline quantile function RNNs. In Proceedings of the AISTATS. 1901-1910.
  18. Kostadin Georgiev and Preslav Nakov. 2013. A non-IID framework for collaborative filtering with restricted Boltz- mann machines. In Proceedings of the ICML. 1148-1156.
  19. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the NIPS. 2672-2680.
  20. Alex Graves. 2011. Practical variational inference for neural networks. In Proceedings of the NIPS. 2348-2356.
  21. Aditya Grover, Aaron Zweig, and Stefano Ermon. 2019. Graphite: Iterative generative modeling of graphs. In Pro- ceedings of the ICML. 2434-2444.
  22. Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. 2019. Learning latent dynamics for planning from pixels. In Proceedings of the ICML. 2555-2565.
  23. Jeff Harrison and Mike West. 1999. Bayesian Forecasting & Dynamic Models. Springer.
  24. Andrew C. Harvey. 1990. Forecasting, Structural Time Series Models and the Kalman Filter. Cambridge University Press.
  25. Hao He, Hao Wang, Guang-He Lee, and Yonglong Tian. 2019. ProbGAN: Towards probabilistic GAN with theoretical guarantees. In Proceedings of the ICLR.
  26. José Miguel Hernández-Lobato and Ryan Adams. 2015. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In Proceedings of the ICML. 1861-1869.
  27. Geoffrey E. Hinton and Drew Van Camp. 1993. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the COLT. 5-13.
  28. Geoffrey E. Hinton and Richard S. Zemel. 1994. Autoencoders, minimum description length, and Helmholtz free energy. In Proceedings of the NIPS. 3-10.
  29. Matthew Hoffman, Francis R. Bach, and David M. Blei. 2010. Online learning for latent Dirichlet allocation. In Proceedings of the NIPS. 856-864.
  30. Wei-Ning Hsu and James R. Glass. 2018. Scalable factorized hierarchical variational autoencoder training. In Pro- ceedings of the INTERSPEECH. 1462-1466.
  31. Wei-Ning Hsu, Yu Zhang, and James R. Glass. 2017. Unsupervised learning of disentangled and interpretable repre- sentations from sequential data. In Proceedings of the NIPS. 1878-1889.
  32. Wei-Ning Hsu, Yu Zhang, Ron J. Weiss, Heiga Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Zhifeng Chen, Jonathan Shen, Patrick Nguyen, and Ruoming Pang. 2019. Hierarchical generative modeling for controllable speech synthesis. In Proceedings of the ICLR.
  33. Hengguan Huang, Fuzhao Xue, Hao Wang, and Ye Wang. 2020. Deep graph random process for relational-thinking- based speech recognition. In Proceedings of the ICML.
  34. Finn V. Jensen et al. 1996. An Introduction to Bayesian Networks, Vol. 210. UCL Press London.
  35. Maximilian Karl, Maximilian Sölch, Justin Bayer, and Patrick van der Smagt. 2017. Deep variational Bayes filters: Unsupervised learning of state space models from raw data. In Proceedings of the ICLR.
  36. Diederik P. Kingma and Max Welling. 2013. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114. · arXiv
  37. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Pro- ceedings of the ICLR.
  38. Adam R. Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. 2018. Sequential attend, infer, repeat: Generative modelling of moving objects. In Proceedings of the NIPS. 8615-8625.
  39. Rahul G. Krishnan, Uri Shalit, and David A. Sontag. 2017. Structured inference networks for nonlinear state space models. In Proceedings of the AAAI. 2101-2109.
  40. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of the NIPS. 1097-1105.
  41. Y. LeCun. 1987. Modeles connexionnistes de l'apprentissage (connectionist learning models). Ph.D. Dissertation. Uni- versité P. et M. Curie (Paris 6).
  42. Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng. 2009. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the ICML. 609-616.
  43. Wu-Jun Li and Dit-Yan Yeung. 2009. Relation regularized matrix factorization. In Proceedings of the IJCAI.
  44. Zhongqi Lu, Zhicheng Dou, Jianxun Lian, Xing Xie, and Qiang Yang. 2015. Content-based collaborative filtering for news topic recommendation. In Proceedings of the AAAI. 217-223.
  45. Wesley J. Maddox, Pavel Izmailov, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. 2019. A simple baseline for Bayesian uncertainty in deep learning. In Proceedings of the NIPS. 13132-13143.
  46. Takamitsu Matsubara, Vicenc Gómez, and Hilbert J. Kappen. 2014. Latent Kullback Leibler control for continuous- state systems using probabilistic graphical models. In Proceedings of the UAI. 583-592.
  47. Nikhil Mehta, Lawrence Carin, and Piyush Rai. 2019. Stochastic blockmodels meet graph neural networks. In Pro- ceedings of the ICML. 4466-4474.
  48. Abdel-rahman Mohamed, Tara N. Sainath, George E. Dahl, Bhuvana Ramabhadran, Geoffrey E. Hinton, and Michael A. Picheny. 2011. Deep belief networks using discriminative features for phone recognition. In Proceedings of the ICASSP. 5060-5063.
  49. Jonas Mueller, David K. Gifford, and Tommi S. Jaakkola. 2017. Sequence to better sequence: Continuous revision of combinatorial structures. In Proceedings of the ICML. 2536-2544.
  50. Radford M. Neal. 1995. Bayesian learning for neural networks. Ph.D. Dissertation. University of Toronto.
  51. Takahiro Omi, Naonori Ueda, and Kazuyuki Aihara. 2019. Fully neural network based model for general temporal point processes. In Proceedings of the NIPS. 2120-2129.
  52. Aäron Van Den Oord, Sander Dieleman, and Benjamin Schrauwen. 2013. Deep content-based music recommenda- tion. In Proceedings of the NIPS. 2643-2651.
  53. Christopher Poultney, Sumit Chopra, Yann L. Cun et al. 2006. Efficient learning of sparse representations with an energy-based model. In Proceedings of the NIPS. 1137-1144.
  54. Sanjay Purushotham, Yan Liu, and C.-C. Jay Kuo. 2012. Collaborative topic regression with social matrix factorization for recommendation systems. In Proceedings of the ICML. 759-766.
  55. Syama Sundar Rangapuram, Matthias W. Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. 2018. Deep state space models for time series forecasting. In Proceedings of the NIPS. 7796-7805.
  56. Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. 2011. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the ICML. 833-840.
  57. Sheldon M. Ross, John J. Kelly, Roger J. Sullivan, William James Perry, Donald Mercer, Ruth M. Davis, Thomas Dell Washburn, Earl V. Sager, Joseph B. Boyce, and Vincent L. Bristow. 1996. Stochastic Processes. Vol. 2. Wiley, New York.
  58. Ruslan Salakhutdinov and Andriy Mnih. 2007. Probabilistic matrix factorization. In Proceedings of the NIPS. 1257- 1264.
  59. Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey E. Hinton. 2007. Restricted Boltzmann machines for collaborative filtering. In Proceedings of the ICML. 791-798.
  60. Oleksandr Shchur, Marin Bilos, and Stephan Günnemann. 2020. Intensity-free learning of temporal point processes. In Proceedings of the ICLR.
  61. Alexander Shekhovtsov and Boris Flach. 2019. Feed-forward propagation in probabilistic neural networks with cat- egorical and max layers. In Proceedings of the ICLR.
  62. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1 (2014), 1929-1958.
  63. Nitish Srivastava and Russ R. Salakhutdinov. 2012. Multimodal learning with deep Boltzmann machines. In Proceed- ings of the NIPS. 2222-2230.
  64. Karl Stelzner, Robert Peharz, and Kristian Kersting. 2019. Faster attend-infer-repeat with tractable probabilistic mod- els. In Proceedings of the ICML. 5966-5975.
  65. Jiahao Su, Milan Cvitkovic, and Furong Huang. 2020. Sampling-free learning of Bayesian quantized neural networks. In Proceedings of the ICLR.
  66. Ilya Sutskever, Oriol Vinyals, and Quoc V. V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the NIPS. 3104-3112.
  67. Wesley Tansey, Yixin Wang, David M. Blei, and Raul Rabadan. 2018. Black box FDR. In Proceedings of the ICML. 4874-4883.
  68. Chong Wang and David M. Blei. 2011. Collaborative topic modeling for recommending scientific articles. In Pro- ceedings of the KDD. 448-456.
  69. Chong Wang, David M. Blei, and David Heckerman. 2008. Continuous time dynamic topic models. In Proceedings of the UAI. 579-586.
  70. Hao Wang. 2017. Bayesian Deep Learning for Integrated Intelligence: Bridging the Gap between Perception and Infer- ence. Ph.D. Dissertation. Hong Kong University of Science and Technology.
  71. Hao Wang, Binyi Chen, and Wu-Jun Li. 2013. Collaborative topic regression with social regularization for tag rec- ommendation. In Proceedings of the IJCAI. 2719-2725.
  72. Hao Wang, Xingjian Shi, and Dit-Yan Yeung. 2015. Relational stacked denoising autoencoder for tag recommenda- tion. In Proceedings of the AAAI. 3052-3058.
  73. Hao Wang, Xingjian Shi, and Dit-Yan Yeung. 2016. Natural-parameter networks: A class of probabilistic neural networks. In Proceedings of the NIPS. 118-126.
  74. Hao Wang, Xingjian Shi, and Dit-Yan Yeung. 2017. Relational deep learning: A deep latent variable model for link prediction. In Proceedings of the AAAI. 2688-2694.
  75. Hao Wang, Shi Xingjian, and Dit-Yan Yeung. 2016. Collaborative recurrent autoencoder: Recommend while learning to fill in the blanks. In Proceedings of the NIPS. 415-423.
  76. Yuyang Wang, Alex Smola, Danielle C. Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski. 2019. Deep factors for forecasting. In Proceedings of the ICML. 6607-6617.
  77. Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. 2015. Embed to control: A locally linear latent dynamics model for control from raw images. In Proceedings of the NIPS. 2728-2736.
  78. Max Welling and Yee Whye Teh. 2011. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the ICML. 681-688.
  79. Andrew Gordon Wilson. 2020. The case for Bayesian deep learning. arXiv preprint arXiv:2001.10995. · arXiv
  80. Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoen- coders for text modeling using dilated convolutions. In Proceedings of the ICML. 3881-3890.
  81. He Zhao, Lan Du, Wray L. Buntine, and Mingyuan Zhou. 2018. Dirichlet belief networks for topic structure learning. In Proceedings of the NIPS. 7966-7977.
  82. Vincent Wenchen Zheng, Bin Cao, Yu Zheng, Xing Xie, and Qiang Yang. 2010. Collaborative filtering meets mobile recommendation: A user-centered approach. In Proceedings of the AAAI.
  83. Mingyuan Zhou, Lauren Hannah, David B. Dunson, and Lawrence Carin. 2012. Beta-negative binomial process and Poisson factor analysis. In Proceedings of the AISTATS. 1462-1471.

Cited by

Sorted by citations

Related

Authors
Wang, H., & Yeung, D.-Y.
Affiliations
  • Massachusetts Institute of Technology (US)
  • Hong Kong University of Science and Technology (HK)
ORCID
Venue
ACM Computing Surveys
Publisher
Association for Computing Machinery (ACM)
Published
2020-09-28
Volume
53
Issue
5
Pages
1-37
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
DOI
10.1145/3409383
OpenAlex
W3089611792
Cited by
247
References
52
Citations by year
2026: 23 · 2025: 54 · 2024: 50 · 2023: 37 · 2022: 43 · 2021: 28 · 2020: 6 · 2019: 4 · 2018: 1 · 2017: 1
FWCI
11.20
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Computing Surveys (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
Bayesian deep learning, uncertainty quantification, probabilistic graphical models, Bayesian neural networks, recommender systems, topic modeling, Bayesian control

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review