Bayesian methods for adaptive models
The DOI matches its OpenAlex record; 68 of the 68 references it cites were found in a registry or an index.
Abstract
The Bayesian framework for model comparison and regularisation is demonstrated by studying interpolation and classification problems modelled with both linear and non-linear models. This framework quantitatively embodies `Occam's razor'. Over-complex and under-regularised models are automatically inferred to be less probable, even though their flexibility allows them to fit the data better. When applied to `neural networks', the Bayesian framework makes possible (1) objective comparison of solutions using alternative network architectures; (2) objective stopping rules for network pruning or growing procedures; (3) objective choice of type of weight decay terms (or regularisers); (4) on--line techniques for optimising weight decay (or regularisation constant) magnitude; (5) a measure of the effective number of well--determined parameters in a model; (6) quantified estimates of the error bars on network parameters and on network output. In the case of classification models, it is sho...
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
(1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895
Introduction to Statistical Pattern Recognition
(1990) · Elsevier eBooks · doi:10.1016/c2009-0-27872-x · Cited by 11,031
Statistical Decision Theory and Bayesian Analysis
Berger, J. O. (1985) · Springer Series in Statistics · doi:10.1007/978-1-4757-4286-2 · Cited by 7,722
Modeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.415 · Cited by 4,388
Self-citationBayesian Inference in Statistical Analysis.
Kadane, J. B., Box, G. E. P., & Tiao, G. C. (1975) · Journal of the American Statistical Association · doi:10.2307/2285412 · Cited by 3,824
AN ESSAY TOWARDS SOLVING A PROBLEM IN THE DOCTRINE OF CHANCES
BAYES (1958) · Biometrika · doi:10.1093/biomet/45.3-4.296 · Cited by 2,911
Stastical Decision Theory and Bayesian Analysis.
Ghosh, M., & Berger, J. (1988) · Journal of the American Statistical Association · doi:10.2307/2288950 · Cited by 2,905
Cited by
A tutorial on support vector regression
Smola, A. J., & Schölkopf, B. (2004) · Statistics and Computing · doi:10.1023/b:stco.0000035301.49549.88 · Cited by 13,387
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
Support Vector Data Description
Tax, D. M. J., & Duin, R. P. W. (2004) · Machine Learning · doi:10.1023/b:mach.0000008084.60811.49 · Cited by 3,512
Nonlinear black-box modeling in system identification: a unified overview
Sjöberg, J., Zhang, Q., Ljung, L., Benveniste, A., Delyon, B., Glorennec, P.-Y., Hjalmarsson, H., & Juditsky, A. (1995) · Automatica · doi:10.1016/0005-1098(95)00120-8 · Cited by 1,942
GAUSSIAN PROCESSES FOR MACHINE LEARNING
SEEGER, M. (2004) · International Journal of Neural Systems · doi:10.1142/s0129065704001899 · Cited by 1,488
A rational analysis of the selection task as optimal data selection.
Oaksford, M., & Chater, N. (1994) · Psychological Review · doi:10.1037//0033-295x.101.4.608 · Cited by 1,052
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989
Generalized Out-of-Distribution Detection: A Survey
Yang, J., Zhou, K., Li, Y., & Liu, Z. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02117-4 · Cited by 884
Decision theoretic generalizations of the PAC model for neural net and other learning applications
Haussler, D. (1992) · Information and Computation · doi:10.1016/0890-5401(92)90010-d · Cited by 843
Experiences with an interactive museum tour-guide robot
Burgard, W., Cremers, A. B., Fox, D., Hähnel, D., Lakemeyer, G., Schulz, D., Steiner, W., & Thrun, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00070-3 · Cited by 818
Bayesian system identification based on probability logic
Beck, J. L. (2010) · Structural Control and Health Monitoring · doi:10.1002/stc.424 · Cited by 605
Application-independent evaluation of speaker detection
Brümmer, N., & du Preez, J. (2006) · Computer Speech & Language · doi:10.1016/j.csl.2005.08.001 · Cited by 595
Trusted Multi-View Classification With Dynamic Evidential Fusion
Han, Z., Zhang, C., Fu, H., & Zhou, J. T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3171983 · Cited by 536
Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
Hernández-Lobato, J. M., & Adams, R. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.05336 · Cited by 504
Learning from Labeled and Unlabeled Data
(2017) · Encyclopedia of Machine Learning and Data Mining · doi:10.1007/978-1-4899-7687-1_100253 · Cited by 462
Active learning for logistic regression: an evaluation
Schein, A. I., & Ungar, L. H. (2007) · Machine Learning · doi:10.1007/s10994-007-5019-5 · Cited by 346
Predictive Uncertainty Estimation via Prior Networks
Malinin, A., & Gales, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10501 · Cited by 337
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
Papernot, N., & McDaniel, P. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04765 · Cited by 335
Mackay, D. J. C. (1995) · Network Computation in Neural Systems · doi:10.1088/0954-898x/6/3/011 · Cited by 915
Deep Ensembles: A Loss Landscape Perspective
Fort, S., Hu, H., & Lakshminarayanan, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02757 · Cited by 311
A Bayesian approach to identification of hybrid systems
Juloski, A. L., Weiland, S., & Heemels, W. P. M. H. (2005) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2005.856649 · Cited by 300
Muto, M., & Beck, J. L. (2008) · Journal of Vibration and Control · doi:10.1177/1077546307079400 · Cited by 272
Uncertainty Estimation Using a Single Deep Deterministic Neural Network
van Amersfoort, J., Smith, L., Teh, Y. W., & Gal, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02037 · Cited by 263
Using Bayesian Deep Learning to Capture Uncertainty for Residential Net Load Forecasting
Sun, M., Zhang, T., Wang, Y., Strbac, G., & Kang, C. (2020) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2019.2924294 · Cited by 262
Damianou, A., & Lawrence, N. D. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.0358 · Cited by 249
“nalized allowing a variational lower bound on the likelihood to be computed. The appropriate size of the latent space can be computed using automatic relevance determination (ARD) priors [Neal, 1996, MacKay et al., 1992]. Damianou et al. [2011] extended this approach by placing a Gaussian process prior over the latent…”
MethodComparison of Approximate Methods for Handling Hyperparameters
MacKay, D. J. C. (1999) · Neural Computation · doi:10.1162/089976699300016331 · Cited by 244
“Problems for which models can be written in the form (2) include linear interpolation with a xed basisset (Gull 1988; MacKay 1992a), nonlinear regression with a neural network (MacKay 1992c), nonlinearclassi cation (MacKay 1992b), and image deconvolution (Gull 1989).”
BackgroundMethodInfluentialSelf-citationAutonomous exploration: driven by uncertainty
Whaite, P., & Ferrie, F. P. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.584097 · Cited by 240
Bayesian neural network learning for repeat purchase modelling in direct marketing
Baesens, B., Viaene, S., Van den Poel, D., Vanthienen, J., & Dedene, G. (2002) · European Journal of Operational Research · doi:10.1016/s0377-2217(01)00129-1 · Cited by 218
$L1$ -Norm Batch Normalization for Efficient Training of Deep Neural Networks
Wu, S., Li, G., Deng, L., Liu, L., Wu, D., Xie, Y., & Shi, L. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876179 · Cited by 166
Engelhardt, B. E., & Stephens, M. (2010) · PLoS Genetics · doi:10.1371/journal.pgen.1001117 · Cited by 128
Deep Deterministic Uncertainty: A New Simple Baseline
Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P. H. S., & Gal, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02336 · Cited by 116
Huang, Y., Beck, J. L., & Li, H. (2017) · Structural Safety · doi:10.1016/j.strusafe.2016.09.001 · Cited by 111
Neural Learning from Unbalanced Data
Murphey, Y. L., Guo, H., & Feldkamp, L. A. (2004) · Applied Intelligence · doi:10.1023/b:apin.0000033632.42843.17 · Cited by 82
Uncertainty Sets for Image Classifiers using Conformal Prediction
Angelopoulos, A., Bates, S., Malik, J., & Jordan, M. I. (2020) · arXiv · doi:10.48550/arxiv.2009.14193 · Cited by 79
Huang, Y., & Beck, J. L. (2018) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.12358 · Cited by 53
Model-Based Policy Search Using Monte Carlo Gradient Estimation With Real Systems Application
Amadio, F., Dalla Libera, A., Antonello, R., Nikovski, D., Carli, R., & Romeres, D. (2022) · IEEE Transactions on Robotics · doi:10.1109/tro.2022.3184837 · Cited by 29
Uncertainty-Aware Machine Translation Evaluation
Glushkova, T., Zerva, C., Rei, R., & Martins, A. F. T. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.330 · Cited by 17
The benefit of the doubt: Uncertainty aware sensing for edge computing platforms
Qendro, L., Chauhan, J., Ramos, A. G. C. P., & Mascolo, C. (2021) · arXiv · doi:10.48550/arxiv.2102.05956 · Cited by 6
Unsupervised Quality Estimation for Neural Machine Translation
Fomicheva, M., Sun, S., Yankovskaya, L., Blain, F., Guzmán, F., Fishel, M., Aletras, N., Chaudhary, V., & Specia, L. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00330 · Cited by 10
Metadata
- Authors
- Mackay, D.
- Published
- 1992-01-01
- OpenAlex
- W2117670920
- Cited by
- 610
- References
- 68
- FWCI
- 11.63
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.