The DOI matches its Crossref record; 102 of the 103 references it cites were found in a registry or an index.
Bayesian Methods for Neural Networks and Related Models
Abstract
Models such as feed-forward neural networks and certain other structures investigated in the computer science literature are not amenable to closed-form Bayesian analysis. The paper reviews the various approaches taken to overcome this difficulty, involving the use of Gaussian approximations, Markov chain Monte Carlo simulation routines and a class of non-Gaussian but “deterministic” approximations called variational approximations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Bayesian Neural Networks (BNNs) [68,71,72] are stochastic ANNs trained using Bayesian inference.”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [72]
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Neural networks for pattern recognition
Bishop, C. (1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,537
Statistical Decision Theory and Bayesian Analysis
Berger, J. O. (1985) · Springer Series in Statistics · doi:10.1007/978-1-4757-4286-2 · Cited by 7,722
Pattern Recognition and Neural Networks
Ripley, B. D. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511812651 · Cited by 6,500
Reversible Jump Markov Chain Monte Carlo Computation and Bayesian Model Determination
Green, P. J. (1995) · Biometrika · doi:10.1093/biomet/82.4.711 · Cited by 5,941
Mackay, D. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.415 · Cited by 4,440
Bayesian Learning for Neural Networks
Neal, R. M. (1996) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-0745-0 · Cited by 4,276
Duane, S., Kennedy, A. D., Pendleton, B. J., & Roweth, D. (1987) · Physics Letters B · doi:10.1016/0370-2693(87)91197-x · Cited by 3,920
An Introduction to Variational Methods for Graphical Models
Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999) · Machine Learning · doi:10.1023/a:1007665907178 · Cited by 3,791
A practical Bayesian framework for backpropagation networks
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.448 · Cited by 2,917
Statistical Analysis of Finite Mixture Distributions.
Lindsay, B. G., Titterington, D. M., Smith, A. F. M., & Makov, U. E. (1987) · Journal of the American Statistical Association · doi:10.2307/2289504 · Cited by 2,906
Hierarchical Mixtures of Experts and the EM Algorithm
Jordan, M. I., & Jacobs, R. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.181 · Cited by 2,786
Jordan, M. I. (1998) · Springer eBooks · doi:10.1007/978-94-011-5014-9 · Cited by 2,449
A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
Neal, R. M., & Hinton, G. E. (1998) · Learning in Graphical Models · doi:10.1007/978-94-011-5014-9_12 · Cited by 2,187
(2000) · Applied Physics Letters · doi:10.1162/15324430152748236 · Cited by 1,819
A Tutorial on Learning with Bayesian Networks
Heckerman, D. E. (2008) · Studies in computational intelligence · doi:10.1007/978-3-540-85066-3_3 · Cited by 1,712
Bishop, C. (1996) · Taylor & Francis eBooks · doi:10.1201/9781420050646.ptb6 · Cited by 1,592
A Tutorial on Learning with Bayesian Networks
Heckerman, D. E. (1998) · Springer eBooks · doi:10.1007/978-94-011-5014-9_11 · Cited by 1,242
Factorial Hidden Markov Models
Ghahramani, Z., & Jordan, M. I. (1997) · Machine Learning · doi:10.1023/a:1007425814087 · Cited by 1,157
Neural Networks: A Review from a Statistical Perspective
Cheng, B., & Titterington, D. M. (1994) · Statistical Science · doi:10.1214/ss/1177010638 · Cited by 1,061
Keeping the neural networks simple by minimizing the description length of the weights
Hinton, G. E., & van Camp, D. (1993) · Proceedings of the sixth annual conference on Computational learning theory - COLT '93 · doi:10.1145/168304.168306 · Cited by 994
Mackay, D. J. C. (1995) · Network Computation in Neural Systems · doi:10.1088/0954-898x/6/3/011 · Cited by 915
Tipping, M. E. (1999) · Cited by 869
The EM Algorithm—an Old Folk-song Sung to a Fast New Tune
Meng, X.-L., & Van Dyk, D. (1997) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00082 · Cited by 802
The Evidence Framework Applied to Classification Networks
Mackay, D. (1992) · Neural Computation · doi:10.1162/neco.1992.4.5.720 · Cited by 733
A Variational Baysian Framework for Graphical Models
Attias, H. T. (1999) · neural information processing systems · Cited by 694
Neural Networks and Related Methods for Classification
Ripley, B. D. (1994) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1994.tb01990.x · Cited by 630
Bayesian parameter estimation via variational methods
Jaakkola, T., & Jordan, M. I. (2000) · Statistics and Computing · doi:10.1023/a:1008932416310 · Cited by 614
An Introduction to Variational Methods for Graphical Models
Jordan, M. I., Ghahramani, Z., Jaakkola, T., & Saul, L. K. (1998) · Springer eBooks · doi:10.1007/978-94-011-5014-9_5 · Cited by 604
Information geometry and alternating minimization procedures
Csiszár, I. (1984) · Medical Entomology and Zoology · Cited by 596
Inferring Parameters and Structure of Latent Variable Models by Variational BayesPreprint
Attias, H. T. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.6676 · Cited by 525
Reversible Jump Markov Chain Monte Carlo Computation and Bayesian Model Determination
Green, P. J. (1995) · Biometrika · doi:10.2307/2337340 · Cited by 503
Attias, H. T. (1999) · Neural Computation · doi:10.1162/089976699300016458 · Cited by 498
Moody, J. E. (1991) · Cited by 437
Bayesian approach for neural networks—review and case studies
Lampinen, J., & Vehtari, A. (2001) · Neural Networks · doi:10.1016/s0893-6080(00)00098-8 · Cited by 392
Advanced mean field methods: theory and practice
Opper, M., & Saad, D. (2001) · Cited by 385
Variational Inference for Bayesian Mixtures of Factor Analysers
Ghahramani, Z., & Beal, M. J. (1999) · Cited by 380
Online Model Selection Based on the Variational Bayes
Sato, M.-A. (2001) · Neural Computation · doi:10.1162/089976601750265045 · Cited by 378
Variational Relevance Vector MachinesPreprint
Bishop, C., & Tipping, M. E. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3838 · Cited by 357
Buntine, W., & Weigend, A. S. (1991) · Complex Systems · Cited by 357
The mean field theory in EM procedures for Markov random fields
Zhang, J. (1992) · IEEE Transactions on Signal Processing · doi:10.1109/78.157297 · Cited by 347
Variational Bayesian Model Selection for Mixture Distributions
Corduneanu, A., & Bishop, C. (2001) · Cited by 345
(2001) · The MIT Press eBooks · doi:10.7551/mitpress/1100.001.0001 · Cited by 335
Gestel, T. V., Suykens, J. A. K., Lanckriet, G. R. G., Lambrechts, A., Moor, B. D., & Vandewalle, J. P. L. (2002) · Neural Computation · doi:10.1162/089976602753633411 · Cited by 291
Propagation Algorithms for Variational Bayesian Learning
Ghahramani, Z., & Beal, M. J. (2000) · Cited by 283
Variational principal components
Bishop, C. (1999) · doi:10.1049/cp:19991160 · Cited by 274
Tutorial on Variational Approximation Methods
Jaakkola, T. (2001) · The MIT Press eBooks · doi:10.7551/mitpress/1100.003.0014 · Cited by 270
Comparison of Approximate Methods for Handling Hyperparameters
Mackay, D. (1999) · Neural Computation · doi:10.1162/089976699300016331 · Cited by 247
Variational Gaussian process classifiers
Mackay, D., & Gibbs, M. (2000) · IEEE Transactions on Neural Networks · doi:10.1109/72.883477 · Cited by 225
Sequential Monte Carlo Methods to Train Neural Network Models
Freitas, J., Niranjan, M., Gee, A. H., & Arnaud Doucet, B. (2000) · Neural Computation · doi:10.1162/089976600300015664 · Cited by 213
Developments of the generative topographic mapping
Bishop, C., Svensén, M., & Williams, C. K. I. (1998) · Neurocomputing · doi:10.1016/s0925-2312(98)00043-5 · Cited by 199
A review of Bayesian neural networks with an application to near infrared spectroscopy
Thodberg, H. H. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.478392 · Cited by 184
Bayesian Methods for Mixtures of Experts
Waterhouse, S. R., Mackay, D., & Robinson, A. J. (1995) · Cited by 178
Gençay, R., & Qi, M. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.935086 · Cited by 174
Ensemble Learning for Blind Image Separation and Deconvolution
Miskin, J. W., & Mackay, D. (2000) · Perspectives in neural computing · doi:10.1007/978-1-4471-0443-8_7 · Cited by 164
Bethe free energy, Kikuchi approximations, and belief propagation algorithms
Yedidia, J. S., Freeman, W. T., & Weiss, Y. (2001) · Cited by 162
Moderating the outputs of support vector machine classifiers
Kwok, J. T.-Y. (1999) · IEEE Transactions on Neural Networks · doi:10.1109/72.788642 · Cited by 155
The evidence framework applied to support vector machines
Kwok, J. T.-Y. (2000) · IEEE Transactions on Neural Networks · doi:10.1109/72.870047 · Cited by 152
Bayesian Measures of Model Complexity and Fit
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van Der Linde, A. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00353 · Cited by 151
A Bayesian Analysis of Self-Organizing Maps
Luttrell, S. P. (1994) · Neural Computation · doi:10.1162/neco.1994.6.5.767 · Cited by 149
Computation with Infinite Neural Networks
Williams, C. K. I. (1998) · Neural Computation · doi:10.1162/089976698300017412 · Cited by 149
The mean field theory in EM procedures for blind Markov random field image restoration
Zhang, J. (1993) · IEEE Transactions on Image Processing · doi:10.1109/83.210863 · Cited by 149
Peng, F., Jacobs, R. A., & Tanner, M. A. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476965 · Cited by 149
Multiresolution forecasting for futures trading using wavelet decompositions
Zhang, B., Coggins, R., Jabri, M. A., Dersch, D. R., & Flower, B. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.935090 · Cited by 137
Bayesian information criteria and smoothing parameter selection in radial basis function networks
Konishi, S. (2004) · Biometrika · doi:10.1093/biomet/91.1.27 · Cited by 135
Bayesian model search for mixture models based on optimizing variational bounds
Ueda, N., & Ghahramani, Z. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00040-0 · Cited by 133
[Neural Networks: A Review from Statistical Perspective]: Rejoinder
Cheng, B., & Titterington, D. M. (1994) · Statistical Science · doi:10.1214/ss/1177010646 · Cited by 131
Issues in Bayesian Analysis of Neural Network Models
Müller, P., & Insua, D. R. (1998) · Neural Computation · doi:10.1162/089976698300017737 · Cited by 115
Bayesian neural networks for classification: how useful is the evidence framework?
Penny, W. D., & Roberts, S. (1999) · Neural Networks · doi:10.1016/s0893-6080(99)00040-4 · Cited by 104
Seeger, M. (1999) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 104
Robust Full Bayesian Learning for Radial Basis Networks
Andrieu, C., de Freitas, N., & Doucet, A. (2001) · Neural Computation · doi:10.1162/089976601750541831 · Cited by 94
Bayesian Radial Basis Functions of Variable Dimension
Holmes, C. C., & Mallick, B. K. (1998) · Neural Computation · doi:10.1162/089976698300017421 · Cited by 93
VIBES: A Variational Inference Engine for Bayesian Networks
Bishop, C., Spiegelhalter, D. J., & Winn, J. M. (2002) · neural information processing systems · Cited by 84
Asymptotic Model Selection for Directed Networks with Hidden Variables
Geiger, D., Heckerman, D. E., & Meek, C. (1998) · Springer eBooks · doi:10.1007/978-94-011-5014-9_16 · Cited by 77
Bayesian approach to neural-network modeling with input uncertainty
Wright, W. A. (1999) · IEEE Transactions on Neural Networks · doi:10.1109/72.809073 · Cited by 74
Approximating Posterior Distributions in Belief Networks Using Mixtures
Bishop, C., Lawrence, N. D., Jaakkola, T., & Jordan, M. I. (1997) · Cited by 67
The relevance vector machine technique for channel equalization application
Chen, S., Gunn, S. R., & Harris, C. J. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.963792 · Cited by 62
Hyperparameter Selection for Self-Organizing Maps
Utsugi, A. (1997) · Neural Computation · doi:10.1162/neco.1997.9.3.623 · Cited by 58
Variational Bayesian Learning of ICA with Missing Data
Chan, K., Lee, T.-W., & Sejnowski, T. J. (2003) · Neural Computation · doi:10.1162/08997660360675116 · Cited by 56
de Freitas, N., Højen-Sørensen, P. A. D. F. R., Jordan, M. I., & Russell, S. (2001) · Uncertainty in Artificial Intelligence · Cited by 56
Statistics and neural networks: advances at the interface
Kay, J., & Titterington, D. M. (2000) · Cited by 51
A Bayesian Approach to Model Selection in Hierarchical Mixtures-of-Experts Architectures
Jacobs, R. A., Peng, F., & Tanner, M. A. (1997) · Neural Networks · doi:10.1016/s0893-6080(96)00050-0 · Cited by 50
The application of neural networks to the papermaking industry
Edwards, P. J., Murray, A., Papadopoulos, G. T., Wallace, A. R., Barnard, J., & Smith, G. (1999) · IEEE Transactions on Neural Networks · doi:10.1109/72.809090 · Cited by 47
Modeling exchange rates: smooth transitions, neural networks, and linear models
Medeiros, M. C., Veiga, A., & Pedreira, C. E. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.935089 · Cited by 46
Bayesian nonlinear model selection and neural networks: a conjugate prior approach
Vila, J.-P., Wagner, V., & Neveu, P. (2000) · IEEE Transactions on Neural Networks · doi:10.1109/72.838999 · Cited by 42
Bayesian retrieval in associative memories with storage errors
Sommer, F. T., & Dayan, P. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/72.701183 · Cited by 40
Parameter estimation for hidden Markov chains
Archer, G. E., & Titterington, D. M. (2002) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(02)00318-x · Cited by 39
Husmeier, D., Penny, W. D., & Roberts, S. (1999) · Neural Networks · doi:10.1016/s0893-6080(99)00020-9 · Cited by 35
Model Selection for Neural Network Classification
Lee, H. K. H. (2001) · Journal of Classification · doi:10.1007/s00357-001-0017-y · Cited by 33
Approximate Bayesian inference for simple mixtures
Humphreys, K., & Titterington, D. M. (2000) · COMPSTAT · doi:10.1007/978-3-642-57678-2_42 · Cited by 30
The Bayesian Evidence Scheme for Regularizing Probability-Density Estimating Neural Networks
Husmeier, D. (2000) · Neural Computation · doi:10.1162/089976600300014890 · Cited by 28
Hall, P., Humphreys, K., & Titterington, D. M. (2002) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/1467-9868.00350 · Cited by 25
Comparing Bayesian neural network algorithms for classifying segmented outdoor images
Vivarelli, F., & Williams, C. K. I. (2001) · Neural Networks · doi:10.1016/s0893-6080(01)00024-7 · Cited by 24
A Noninformative Prior for Neural Networks
Lee, H. K. H. (2003) · Machine Learning · doi:10.1023/a:1020258113913 · Cited by 23
Probabilistic image processing by means of the Bethe approximation for theQ-Ising model
Tanaka, K., Inoue, J.-I., & Titterington, D. M. (2003) · Journal of Physics A Mathematical and General · doi:10.1088/0305-4470/36/43/025 · Cited by 23
A Tighter Bound for Graphical Models
Leisink, M. A. R., & Kappen, H. J. (2001) · Neural Computation · doi:10.1162/089976601750399344 · Cited by 22
Density Estimation by Mixture Models with Smoothing Priors
Utsugi, A. (1998) · Neural Computation · doi:10.1162/089976698300016990 · Cited by 20
Robust Full Bayesian Methods for Neural Networks
Andrieu, C., de Freitas, J. F. G., & Doucet, A. (1999) · Bristol Research (University of Bristol) · Cited by 12
Bayesian inference in neural networks
Paige, R. L. (2001) · Biometrika · doi:10.1093/biomet/88.3.623 · Cited by 11
Asymptotic Model Selection for Directed Networks with Hidden VariablesPreprint
Geiger, D., Heckerman, D. E., & Meek, C. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1302.3580 · Cited by 10
Humphreys, K., & Titterington, D. M. (2000) · Neural Processing Letters · doi:10.1023/a:1009617914949 · Cited by 6
Some examples of recursive variational approximations
Humphreys, K., & Titterington, M. (2001) · Cited by 3
Research works printed but not linked: 3
- Choudrey, R. and Roberts, S. J. (2001). Variational Bayesian independent component analysis with flexible sources. Technical Report PARG-01-03, Dept. Engineering Science, Univ. Oxford.
- MacKay, D. J. C. (1997). Ensemble learning for hidden Markov models. Technical report, Cavendish Lab., Univ. Cambridge.
- Ripley, B. D. (1995). Choosing network complexity. In Probabilistic Reasoning and Bayesian Belief Networks (A. Gammerman, ed.) 97--108. Waller, Henley-on-Thames, UK.
Cited by
Sorted by citations
Rue, H., Martino, S., & Chopin, N. (2009) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2008.00700.x · Cited by 5,279
Bioinformatics and Computational Biology Solutions Using R and Bioconductor
Gentleman, R. C., Carey, V. J., Huber, W. K., Irizarry, R. A., & Dudoit, S. (2005) · Statistics in the health sciences · doi:10.1007/0-387-29362-0 · Cited by 3,470
“A review of Bayesian inference methods with neural network models is given in Titterington (2004).”
MethodHands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users
Jospin, L. V., Laga, H., Boussaid, F., Buntine, W., & Bennamoun, M. (2022) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2022.3155327 · Cited by 879
“Bayesian neural networks (BNNs) [8, 9, 10] are stochastic neural networks trained using a Bayesian approach.”
MethodConfronting the Challenge of Modeling Cloud and Precipitation Microphysics
Morrison, H., van Lier‐Walqui, M., Fridlind, A. M., Grabowski, W. W., Harrington, J. Y., Hoose, C., Korolev, A. V., Kumjian, M. R., Milbrandt, J. A., Pawłowska, H., et al. (2020) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2019ms001689 · Cited by 672
Jordan, M. I. (2004) · Statistical Science · doi:10.1214/088342304000000026 · Cited by 440
“…were formulated in terms of the minimization of a Kullback–Leibler (KL) divergence , and the space over which the optimization was performed was a set of " simplified " probability distributions , generally obtained by removing edges from a graphical model (see Titterington, 2004, this volume ).”
Metadata
- Authors
- Titterington, D. M.
- Corresponding author
- Titterington, D. M.
- Venue
- Statistical Science
- Publisher
- Institute of Mathematical Statistics
- Published
- 2004-02-01
- Volume
- 19
- Issue
- 1
- Pages
- 128-139
- ISSN
- 0883-4237 (ISSN-L)
- 2168-8745
- OpenAlex
- W2019449965
- Cited by
- 162
- References
- 103
- Citations by year
- 2026: 2 · 2025: 9 · 2024: 20 · 2023: 16 · 2022: 17 · 2021: 9 · 2020: 9 · 2019: 7 · 2018: 6 · 2017: 7 · 2016: 10 · 2015: 9 · 2014: 8 · 2013: 3 · 2012: 1
- FWCI
- 3.05
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Statistical Science (publishedVersion)
- Project Euclid (Cornell University) (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Markov Chains and Monte Carlo Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- Bayesian neural networks, feedforward neural networks, Gaussian approximation, Markov chain Monte Carlo, variational approximation, computational intractability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review