The DOI matches its Crossref record; 21 of the 24 references it cites were found in a registry or an index.
Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units
Abstract
We generalize recent theoretical work on the minimal number of layers of narrow deep belief networks that can approximate any probability distribution on the states of their visible units arbitrarily well. We relax the setting of binary units (Sutskever & Hinton, 2008 ; Le Roux & Bengio, 2008 , 2010 ; Montúfar & Ay, 2011 ) to units with arbitrary finite state spaces and the vanishing approximation error to an arbitrary approximation error tolerance. For example, we show that a q-ary deep belief network with L > or = 2 + (q[m-delta]-1 / (q-1)) layers of width n < or = + log(q) (m) + 1 for some [Formula : see text] can approximate any probability distribution on {0, 1, ... , q-1}n without exceeding a Kullback-Leibler divergence of delta. Our analysis covers discrete restricted Boltzmann machines and naive Bayes models as special cases.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While the original theorems showed that two layers were sufficient for universal function approximation, deeper networks can be more powerful and more efficient than shallower networks with the same number of parameters [57][58][59].”
From Continuous-variable quantum neural networks (2019) · cites this paper as [58]
In the citing paper
“Every Gaussian transformation on N modes [Eq.( 7)] can be decomposed into a CV circuit containing only the basic gates mentioned above.”
From Continuous-variable quantum neural networks (2019) · cites this paper as [48]
Citations
Cites
Multilayer feedforward networks are universal approximators
Hornik, K., Stinchcombe, M., & White, H. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90020-8 · Cited by 21,728
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Advances in Neural Information Processing Systems 19: Proceedings of the 2006 Conference
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,044
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 7,329
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 5,044
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Advances in Neural Information Processing Systems 18
Murray, I., Mackay, D., Zoubin Ghahramani, A., & Skilling, J. (2006) · Cited by 1,927
Information processing in dynamical systems: foundations of harmony theory
Smolensky, P. (1986) · The MIT Press eBooks · Cited by 1,673
Research works printed but not linked: 7
- Ay N., 2006, Kybernetika, 42, 517
- Long P. M., 2010, Proceedings of the 27th ICML, 703
- Montúfar G., 2013, Kybernetika, 49, 23
- Montúfar G., 2012, Proceedings of the Ninth Workshop on Uncertainty Processing, 137
- Rauh J., 2013, Kybernetika, 49, 199
- Smolensky P., 1986, Parallel distributed processing: Explorations in the microstructure of cognition, 1, 194
- Welling M., 2005, Advances in neural information processing systems, 17, 1481
Cited by
On the Number of Linear Regions of Deep Neural Networks
Montúfar, G., Pascanu, R., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1402.1869 · Cited by 840
Self-citationNonparametric regression using deep neural networks with ReLU activation function
Schmidt-Hieber, J. (2020) · The Annals of Statistics · doi:10.1214/19-aos1875 · Cited by 663
Continuous-variable quantum neural networks
Killoran, N., Bromley, T. R., Arrazola, J. M., Schuld, M., Quesada, N., & Lloyd, S. (2019) · Physical Review Research · doi:10.1103/physrevresearch.1.033063 · Cited by 418
Continuous-variable quantum neural networks
Killoran, N., Bromley, T. R., Arrazola, J. M., Schuld, M., Quesada, N., & Lloyd, S. (2018) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 358
Classification of TrashNet Dataset Based on Deep Learning Models
Aral, R. A., Keskin, Ş. R., Kaya, M., & Haciömeroğlu, M. (2018) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2018.8622212 · Cited by 251
Universal Approximation with Deep Narrow NetworksPreprint
Kidger, P., & Lyons, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08539 · Cited by 182
Applications of deep learning in water quality management: A state-of-the-art review
Wai, K. P., Chia, M. Y., Koo, C. H., Huang, Y. F., & Chong, W. C. (2022) · Journal of Hydrology · doi:10.1016/j.jhydrol.2022.128332 · Cited by 151
Approximating smooth functions by deep neural networks with sigmoid activation function
Langer, S. (2020) · Journal of Multivariate Analysis · doi:10.1016/j.jmva.2020.104696 · Cited by 83
An Introduction to Neural Networks
Aggarwal, C. C. (2018) · Springer eBooks · doi:10.1007/978-3-319-94463-0_1 · Cited by 72
Restricted Boltzmann Machines: Introduction and Review
Montúfar, G. (2018) · Springer proceedings in mathematics & statistics · doi:10.1007/978-3-319-97798-0_4 · Cited by 55
Self-citationInvestigating ANN architectures and training to estimate snow water equivalent from snow depth
Ntokas, K., Odry, J., Boucher, M., & Garnaud, C. (2021) · Hydrology and earth system sciences · doi:10.5194/hess-25-3017-2021 · Cited by 48
Tang, X., Zhong, B., Peng, J., Hao, B., & Li, J. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106353 · Cited by 42
A Novel Deep Convolutional Neural Network Model for COVID-19 Disease Detection
Irmak, E. (2020) · Medical Technologies Congress (TIPTEKNO) · doi:10.1109/tiptekno50054.2020.9299286 · Cited by 41
Intelligent robust PI adaptive control strategy for speed control of EV(s)
Khooban, M. H., Almasi, O. N., Niknam, T., & Shasadeghi, M. (2016) · IET Science Measurement & Technology · doi:10.1049/iet-smt.2015.0190 · Cited by 34
Adaptation to criticality through organizational invariance in embodied agents
Aguilera, M., & Bedia, M. G. (2018) · Scientific Reports · doi:10.1038/s41598-018-25925-4 · Cited by 13
Almasi, O. N., & Khooban, M. H. (2016) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2015.11.009 · Cited by 11
Discrete Restricted Boltzmann MachinesPreprint
Montúfar, G., & Morton, J. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3529 · Cited by 8
Self-citationDeep Narrow Boltzmann Machines are Universal ApproximatorsPreprint
Montúfar, G. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.3784 · Cited by 8
Self-citationMaximal Information Divergence from Statistical Models Defined by Neural Networks
Montúfar, G., Rauh, J., & Ay, N. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40020-9_85 · Cited by 6
Self-citationNeural network modeling and prediction of HfO 2 thin film properties tuned by thermal annealing
Gao, M., Yin, C., Shao, J., & Zhu, M. (2024) · High Power Laser Science and Engineering · doi:10.1017/hpl.2024.6 · Cited by 5
Estimation of a regression function on a manifold by fully connected deep neural networks
Köhler, M., Langer, S., & Reif, U. (2022) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2022.05.008 · Cited by 6
Powers of layers for image-to-image translationPreprint
Touvron, H., Douze, M., Cord, M., & Jégou, H. (2020) · arXiv.org · doi:10.48550/arxiv.2008.05763 · Cited by 3
Enhancement Of Breast Cancer Diagnosis Accuracy With Deep Learning
Yıldız, İ., & Karadeniz, A. T. (2019) · European Journal of Science and Technology · doi:10.31590/ejosat.638428 · Cited by 4
Factorized mutual information maximization
Merkh, T., & Montúfar, G. (2020) · Kybernetika · doi:10.14736/kyb-2020-5-0948 · Cited by 3
Self-citationDeep Learning: Principles and Training Algorithms
Aggarwal, C. C. (2023) · Springer eBooks · doi:10.1007/978-3-031-29642-0_4 · Cited by 3
Exploring Criticality as a Generic Adaptive Mechanism
Aguilera, M., & Bedía, M. G. (2018) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2018.00055 · Cited by 3
Stochastic Feedforward Neural Networks: Universal Approximation
Merkh, T., & Montúfar, G. (2022) · Cambridge University Press eBooks · doi:10.1017/9781009025096.007 · Cited by 3
Self-citationDenoising Deep Boltzmann Machines: Compression for Deep Learning
Li, Q., & Chen, Y. (2020) · DCC · doi:10.1109/dcc47342.2020.00038 · Cited by 2
Aguilera, M., & Bedía, M. G. (2017) · European Conference on Artificial Life · doi:10.7551/ecal_a_009 · Cited by 2
Compression by and for Deep Boltzmann Machines
Li, Q., Chen, Y., & Kim, Y. (2020) · IEEE Transactions on Communications · doi:10.1109/tcomm.2020.3020796 · Cited by 2
WITHDRAWN: Approximation properties of fully connected deep neural networks
Köhler, M., & Langer, S. (2019) · Journal of Multivariate Analysis · doi:10.1016/j.jmva.2019.06.005 · Cited by 1
Network Horizon Dynamics I: Qualitative AspectsPreprint
Dribus, B. F., Sumner, A. C., Bist, K., Regmi, N., Sircar, J., & Upreti, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.10268
Cao, J., Zhou, Q. C., Chen, Y., Yin, L., & Zhang, F. (2021) · doi:10.36227/techrxiv.16522011.v1
Qahmash, A. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-19-4193-1_47
Learning from electrophysiology time series during sleep : from scoring to event detection
Chambon, S. (2018) · theses.fr (ABES) · doi:10.70675/9bc9b3a7z17c4z400ez9f55za529aad4f8f1
Méthodes numériques pour la résolution de problèmes inverses en électrocardiographie
Karoui, A. (2021) · theses.fr (ABES) · doi:10.70675/cd6e5212zb67cz46eaza6e4z6422e1f8e0ee
Metadata
- Authors
- Montúfar, G. F.
- Affiliations
- Pennsylvania State University (US)
- Corresponding author
- Montúfar, Guido F.
- ORCID
- Montúfar, Guido F.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2014-07
- Volume
- 26
- Issue
- 7
- Pages
- 1386-1407
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2039661710
- Cited by
- 38
- References
- 24
- Citations by year
- 2024: 1 · 2023: 1 · 2022: 4 · 2021: 3 · 2020: 8 · 2019: 5 · 2018: 8 · 2017: 1 · 2016: 2 · 2015: 1 · 2014: 2
- FWCI
- 1.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Max Planck Institute for Plasma Physics (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- deep belief network, universal approximation, finite state space, Kullback-Leibler divergence, restricted Boltzmann machine, naive Bayes model, probability distribution approximation
- MeSH terms
- Algorithms, Probability, Stochastic Processes, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
