Bayesian Recurrent Neural Networks
Open accessAbstract
In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks. Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training, also reducing the amount of parameters by 80\%. Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs. We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics. We show how this technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks. We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them. We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In 2019, Fortunato et al. [124] used Deep learning with BBB (Bayesian RNN) on 1 dataset.
“Fortunato et al. [124] | 2019 | arXiv | 1 | Deep learning | BBB (Bayesian RNN)”
From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [124]
The Discussion (p. 9) confirms that Fortunato and colleagues applied Bayes by Backprop to RNNs. But §6.1 (p. 6) evaluates the Penn Treebank benchmark and §6.2 (p. 9) uses the MSCOCO dataset, so the table’s count of one dataset is incorrect. The passages in our copy do not establish the table’s 2019 date. (LLM, conf=0.88)
Experimental — accuracy not yet measuredBayesian RNNs performed better on an image captioning task and a language modelling benchmark than traditional RNNs.
“They empirically yielded that Bayesian RNNs performed better on an image captioning task and a language modelling benchmark than traditional RNNs.”
From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [124]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405 · see citing works
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
Self-citationPointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,051 · see citing works
Recurrent Neural Network RegularizationPreprint
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv.org · doi:10.48550/arxiv.1409.2329 · Cited by 3,086 · see citing works
Self-citationDeep Speech 2: End-to-End Speech Recognition in English and MandarinPreprint
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,191 · see citing works
Graphical Models, Exponential Families, and Variational Inference
Wainwright, M. J., & Jordan, M. I. (2008) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000001 · Cited by 1,822 · see citing works
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. M. (2015) · arXiv · doi:10.48550/arxiv.1508.06615 · Cited by 1,530 · see citing works
Weight Uncertainty in Neural Networks
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1505.05424 · Cited by 1,342 · see citing works
Self-citationExploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y.-H. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02410 · Cited by 1,202 · see citing works
Self-citationKeeping the neural networks simple by minimizing the description length of the weights
Hinton, G. E., & van Camp, D. (1993) · Proceedings of the sixth annual conference on Computational learning theory - COLT '93 · doi:10.1145/168304.168306 · Cited by 994 · see citing works
Mackay, D. J. C. (1995) · Network Computation in Neural Systems · doi:10.1088/0954-898x/6/3/011 · Cited by 915 · see citing works
Improved Image Captioning via Policy Gradient optimization of SPIDEr
Liu, S., Zhu, Z., Ye, N., Guadarrama, S., & Murphy, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.100 · Cited by 430 · see citing works
Learning Stochastic Recurrent NetworksPreprint
Bayer, J., & Osendorfer, C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.7610 · Cited by 191 · see citing works
Variational Recurrent Auto-EncodersPreprint
Fabius, O., & van Amersfoort, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6581 · Cited by 189 · see citing works
Learning to OptimizePreprint
Li, K., & Malik, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01885 · Cited by 159 · see citing works
Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural NetworksPreprint
Li, C., Chen, C., Carlson, D., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.07666 · Cited by 156 · see citing works
Bayesian Recurrent Neural Network for Language Modeling
Chien, J., & Ku, Y.-C. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2499302 · Cited by 132 · see citing works
Lipton, Z. C., Li, X., Gao, J., Li, L. C., Ahmed, F., & Deng, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.05081 · Cited by 82 · see citing works
Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural NetworksPreprint
Houthooft, R., Chen, X., Duan, Y., Schulman, J. D., de Turck, F., & Abbeel, P. (2016) · arXiv.org · doi:10.48550/arxiv.1605.09674 · Cited by 81 · see citing works
Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image CaptioningPreprint
Lu, J., Xiong, C., Parikh, D., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.01887 · Cited by 52 · see citing works
Stochastic Backpropagation through Mixture Density DistributionsPreprint
Graves, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.05690 · Cited by 33 · see citing works
Hasenclever, L., Webb, S., Lienart, T., Vollmer, S. J., Lakshminarayanan, B., Blundell, C., & Teh, Y. W. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.09327 · Cited by 13 · see citing works
Self-citationScalable Bayesian Learning of Recurrent Neural Networks for Language ModelingPreprint
Gan, Z., Li, C., Chen, C., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08034 · Cited by 6 · see citing works
Modeling D st with Recurrent EM Neural Networks
Mirikitani, D. T., & Ouarbya, L. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-04274-4_100
Two problems with variational expectation maximisation for time series models
Turner, R. E., & Sahani, M. (2011) · Bayesian Time Series Models · doi:10.1017/cbo9780511984679.006
Improved Image Captioning via Policy Gradient optimization of SPIDEr
Liu, S., Zhu, Z., Ye, N., Guadarrama, S., & Murphy, K. (2016) · arXiv · doi:10.48550/arxiv.1612.00370
Zilly, J. G., Srivastava, R. K., Koutník, J., & Schmidhuber, J. (2016) · arXiv · doi:10.48550/arxiv.1607.03474
Flagged first, then by citations
Research works printed but not linked: 19
- Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems , pp. 3981-3989, 2016.
- Wray L Buntine and Andreas S Weigend. Bayesian back-propagation. Complex systems , 5(6): 603-643, 1991.
- Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems , pp. 2980-2988, 2015.
- John S Denker and Yann Lecun. Transforming neural-net output levels to probability distributions. In Advances in neural information processing systems , pp. 853-859, 1991.
- John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12(Jul):2121-2159, 2011.
- Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In Advances in Neural Information Processing Systems , pp. 1019-1027, 2016.
- Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems , pp. 2348-2356, 2011.
- Jos´ e Miguel Hern´ andez-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In ICML , pp. 1861-1869, 2015.
- Sepp Hochreiter, J¨ urgen Schmidhuber, et al. Simplifying neural nets by discovering flat minima. Advances in Neural Information Processing Systems , pp. 529-536, 1995.
- Diederik P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems , pp. 2575-2583, 2015.
- Maren Mahsereci and Philipp Hennig. Probabilistic line searches for stochastic optimization. In Advances in Neural Information Processing Systems , pp. 181-189, 2015.
- Radford M Neal. Bayesian learning for neural networks , volume 118. Springer Science & Business Media, 2012.
- Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pp. 324-333, 2016.
- Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of The 31st International Conference on Machine Learning , pp. 1278-1286, 2014.
- Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights. In Advances in Neural Information Processing Systems , pp. 963-971, 2014.
- Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research , 15(1):1929-1958, 2014.
- Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions on pattern analysis and machine intelligence , 2016.
- Sida I Wang and Christopher D Manning. Fast dropout training. In ICML (2) , pp. 118-126, 2013.
- Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11) , pp. 681-688, 2011.
Cited by
Sorted by citations
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 2,720 · see citing works
“The authors in [119] investigated a variational Bayes scheme for RNN.”
BackgroundAn Introduction to Variational Autoencoders
Kingma, D. P., & Welling, M. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000056 · Cited by 2,609 · see citing works
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning
Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., & Finn, C. (2018) · arXiv · doi:10.48550/arxiv.1803.11347 · Cited by 370 · see citing works
Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness
Chen, J., & Mueller, J. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.283 · Cited by 33 · see citing works
- Authors
- Fortunato, M., Blundell, C., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1704.02798
- OpenAlex
- W2606554264
- Cited by
- 145
- References
- 33
- Citations by year
- 2025: 7 · 2024: 11 · 2023: 10 · 2022: 17 · 2021: 26 · 2020: 33 · 2019: 26 · 2018: 11 · 2017: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- variational inference, uncertainty estimation, regularization, posterior approximation, Bayesian neural networks, language modeling, image captioning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
