The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Simplifying Neural Networks by Soft Weight-Sharing
Abstract
One way of simplifying neural networks so they generalize better is to add an extra term to the error function that will penalize complexity. Simple versions of this approach include penalizing the sum of the squares of the weights or penalizing the number of nonzero weights. We propose a more complicated penalty term in which the distribution of weight values is modeled as a mixture of multiple gaussians. A set of weights is simple if the weights have high probability density under the mixture model. This can be achieved by clustering the weights into subsets with the weights in each cluster having very similar values. Since we do not know the appropriate means or variances of the clusters in advance, we allow the parameters of the mixture model to adapt at the same time as the network learns. Simulations on two different problems demonstrate that this complexity term is more effective than previous complexity terms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Information Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer Series in Statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 17,781
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1093/oso/9780198503682.001.0001 · Cited by 7,878
Modeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011
Spectral Analysis and Time Series
Preston, D. B. (1983) · Technometrics · doi:10.1080/00401706.1983.10487866 · Cited by 5,011
(1989) · neural information processing systems · Cited by 3,736
(1989) · Cited by 3,214
Advances in Neural Information Processing Systems 37
(2024) · doi:10.52202/079017 · Cited by 2,791
Reed, R. (1993) · IEEE Transactions on Neural Networks · doi:10.1109/72.248452 · Cited by 1,708
What Size Net Gives Valid Generalization?
Baum, E. B., & Haussler, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.1.151 · Cited by 1,541
Threshold Autoregression, Limit Cycles and Cyclical Data
Tong, H., & Lim, K. S. (1980) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1980.tb01126.x · Cited by 1,309
Progress in supervised neural networks
Hush, D. R., & Horne, B. G. (1993) · IEEE Signal Processing Magazine · doi:10.1109/79.180705 · Cited by 1,247
Non-linear and non-stationary time series analysis
Priestley, M. (1988) · Medical Entomology and Zoology · Cited by 1,045
Generalization and network design strategies
LeCun, Y. (1989) · Cited by 885
Learning distributed representations of concepts.
Hinton, G. E. (1989) · eScholarship (California Digital Library) · Cited by 827
Self-citationPREDICTING THE FUTURE: A CONNECTIONIST APPROACH
Weigend, A. S., Huberman, B. A., & Rumelhart, D. E. (1990) · International Journal of Neural Systems · doi:10.1142/s0129065790000102 · Cited by 721
A time-delay neural network architecture for isolated word recognition
Lang, K. J., Waibel, A. H., & Hinton, G. E. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90044-l · Cited by 596
Self-citationBuntine, W., & Weigend, A. S. (1991) · Complex Systems · Cited by 361
Using Relevance to Reduce Network Size Automatically
MOZER, M. C., & SMOLENSKY, P. (1989) · Connection Science · doi:10.1080/09540098908915626 · Cited by 339
Skilling, J. (1989) · Springer eBooks · doi:10.1007/978-94-015-7860-8_3 · Cited by 266
Developments in Maximum Entropy Data Analysis
Gull, S. F. (1989) · Maximum Entropy and Bayesian Methods · doi:10.1007/978-94-015-7860-8_4 · Cited by 262
Bayesian Inductive Inference and Maximum Entropy
Gull, S. F. (1988) · Maximum-Entropy and Bayesian Methods in Science and Engineering · doi:10.1007/978-94-009-3049-0_4 · Cited by 246
Generalization and Parameter Estimation in Feedforward Nets: Some Experiments
Morgan, N., & Bourlard, H. (1989) · neural information processing systems · Cited by 225
JETNET 3.0—A versatile artificial neural network package
Peterson, C., Rögnvaldsson, T., & Lönnblad, L. (1994) · Computer Physics Communications · doi:10.1016/0010-4655(94)90120-1 · Cited by 204
Modèles connexionnistes de l'apprentissage
Le Cun, Y., & Fogelman-Soulié, F. (1987) · Intellectica. Revue de l'Association pour la Recherche Cognitive · doi:10.3406/intel.1987.1804 · Cited by 185
Bayesian Methods: General Background
Jaynes, E. T. (1986) · Maximum Entropy and Bayesian Methods in Applied Statistics · doi:10.1017/cbo9780511569678.003 · Cited by 185
Learning translation invariant recognition in a massively parallel networks
Hinton, G. E. (1987) · Lecture Notes in Computer Science · doi:10.1007/3-540-17943-7_117 · Cited by 179
Self-citationEvolving recurrent perceptrons for time-series modeling
McDonnell, J. R., & Waagen, D. E. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.265958 · Cited by 142
Nowlan, S. J. (1991) · Cited by 124
Self-citationIntrator, N. (1993) · Neural Computation · doi:10.1162/neco.1993.5.3.443 · Cited by 37
Optimal Network Construction by Minimum Description Length
Kendall, G. D., & Hall, T. J. (1993) · Neural Computation · doi:10.1162/neco.1993.5.2.210 · Cited by 11
On the combination of supervised and unsupervised learning
Intrator, N. (1993) · Physica A Statistical Mechanics and its Applications · doi:10.1016/0378-4371(93)90572-l · Cited by 5
Cited by
(2014) · Cited by 33,419
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503
Albumentations: fast and flexible image augmentations
Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A., Druzhinin, M., & Kalinin, A. A. (2020) · Information · doi:10.3390/info11020125 · Cited by 2,714
Reed, R. (1993) · IEEE Transactions on Neural Networks · doi:10.1109/72.248452 · Cited by 1,708
Metadata
- Authors
- Nowlan, S. J., & Hinton, G. E.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 1992-07
- Volume
- 4
- Issue
- 4
- Pages
- 473-493
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2156297475
- Cited by
- 595
- References
- 32
- FWCI
- 25.49
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.