The DOI matches its Crossref record; 21 of the 24 references it cites were found in a registry or an index.
Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units
Abstract
We generalize recent theoretical work on the minimal number of layers of narrow deep belief networks that can approximate any probability distribution on the states of their visible units arbitrarily well. We relax the setting of binary units (Sutskever & Hinton, 2008 ; Le Roux & Bengio, 2008 , 2010 ; Montúfar & Ay, 2011 ) to units with arbitrary finite state spaces and the vanishing approximation error to an arbitrary approximation error tolerance. For example, we show that a q-ary deep belief network with L > or = 2 + (q[m-delta]-1 / (q-1)) layers of width n < or = + log(q) (m) + 1 for some [Formula : see text] can approximate any probability distribution on {0, 1, ... , q-1}n without exceeding a Kullback-Leibler divergence of delta. Our analysis covers discrete restricted Boltzmann machines and naive Bayes models as special cases.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While the original theorems showed that two layers were sufficient for universal function approximation, deeper networks can be more powerful and more efficient than shallower networks with the same number of parameters [57][58][59].”
From Continuous-variable quantum neural networks (2019) · cites this paper as [58]
In the citing paper
“Every Gaussian transformation on N modes [Eq.( 7)] can be decomposed into a CV circuit containing only the basic gates mentioned above.”
From Continuous-variable quantum neural networks (2019) · cites this paper as [48]
Citations
Cites
Multilayer feedforward networks are universal approximators
Hornik, K., Stinchcombe, M., & White, H. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90020-8 · Cited by 21,728
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Advances in Neural Information Processing Systems 19: Proceedings of the 2006 Conference
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,044
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 7,329
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 5,044
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Advances in Neural Information Processing Systems 18
Murray, I., Mackay, D., Zoubin Ghahramani, A., & Skilling, J. (2006) · Cited by 1,927
Information processing in dynamical systems: foundations of harmony theory
Smolensky, P. (1986) · The MIT Press eBooks · Cited by 1,673
Research works printed but not linked: 7
- Ay N., 2006, Kybernetika, 42, 517
- Long P. M., 2010, Proceedings of the 27th ICML, 703
- Montúfar G., 2013, Kybernetika, 49, 23
- Montúfar G., 2012, Proceedings of the Ninth Workshop on Uncertainty Processing, 137
- Rauh J., 2013, Kybernetika, 49, 199
- Smolensky P., 1986, Parallel distributed processing: Explorations in the microstructure of cognition, 1, 194
- Welling M., 2005, Advances in neural information processing systems, 17, 1481
Cited by
On the Number of Linear Regions of Deep Neural Networks
Montúfar, G., Pascanu, R., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1402.1869 · Cited by 840
Self-citationNonparametric regression using deep neural networks with ReLU activation function
Schmidt-Hieber, J. (2020) · The Annals of Statistics · doi:10.1214/19-aos1875 · Cited by 663
Continuous-variable quantum neural networks
Killoran, N., Bromley, T. R., Arrazola, J. M., Schuld, M., Quesada, N., & Lloyd, S. (2019) · Physical Review Research · doi:10.1103/physrevresearch.1.033063 · Cited by 418
Continuous-variable quantum neural networks
Killoran, N., Bromley, T. R., Arrazola, J. M., Schuld, M., Quesada, N., & Lloyd, S. (2018) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 358
Classification of TrashNet Dataset Based on Deep Learning Models
Aral, R. A., Keskin, Ş. R., Kaya, M., & Haciömeroğlu, M. (2018) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2018.8622212 · Cited by 251
Metadata
- Authors
- Montúfar, G. F.
- Affiliations
- Pennsylvania State University (US)
- Corresponding author
- Montúfar, Guido F.
- ORCID
- Montúfar, Guido F.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2014-07
- Volume
- 26
- Issue
- 7
- Pages
- 1386-1407
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2039661710
- Cited by
- 38
- References
- 24
- Citations by year
- 2024: 1 · 2023: 1 · 2022: 4 · 2021: 3 · 2020: 8 · 2019: 5 · 2018: 8 · 2017: 1 · 2016: 2 · 2015: 1 · 2014: 2
- FWCI
- 1.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Max Planck Institute for Plasma Physics (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- deep belief network, universal approximation, finite state space, Kullback-Leibler divergence, restricted Boltzmann machine, naive Bayes model, probability distribution approximation
- MeSH terms
- Algorithms, Probability, Stochastic Processes, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
