The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures
Abstract
Recently, researchers in the artificial neural network field have focused their attention on connectionist models composed by several hidden layers. In fact, experimental results and heuristic considerations suggest that deep architectures are more suitable than shallow ones for modern applications, facing very complex problems, e.g., vision and human language understanding. However, the actual theoretical results supporting such a claim are still few and incomplete. In this paper, we propose a new approach to study how the depth of feedforward neural networks impacts on their ability in implementing high complexity functions. First, a new measure based on topological concepts is introduced, aimed at evaluating the complexity of the function implemented by a neural network, used for classification purposes. Then, deep and shallow neural architectures with common sigmoidal activation functions are compared, by deriving upper and lower bounds on their complexity, and studying how the complexity depends on the number of hidden units and the used activation function. The obtained results seem to support the idea that deep networks actually implements functions of higher complexity, so that they are able, with the same number of resources, to address more difficult problems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
The graph neural network model
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005605 · Cited by 9,582
Self-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Approximation capabilities of multilayer feedforward networks
Hornik, K. (1991) · Neural Networks · doi:10.1016/0893-6080(91)90009-t · Cited by 6,091
(2001) · Cited by 5,374
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841 · cited 1×
“…networks, researchers have proposed and experimented several models exploiting multilayer architectures, including convolutional neural networks [9], cascade correlation networks [10], deep neural networks [4], [5], layered graph neural networks [11], and networks with adaptive activations [12].”
Convolutional networks for images, speech, and time series
LeCun, Y., & Bengio, Y. (1998) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.5555/303568.303704 · Cited by 4,231
On the approximate realization of continuous mappings by neural networks
Funahashi, K.-I. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90003-8 · Cited by 4,206
Universal Approximation Using Radial-Basis-Function Networks
Park, J., & Sandberg, I. W. (1991) · Neural Computation · doi:10.1162/neco.1991.3.2.246 · Cited by 4,112
The Cascade-Correlation learning architecture
Fahlman, S. E., & Lebiere, C. J. (2018) · doi:10.1184/r1/6610403 · Cited by 2,372
Neural Networks: A Systematic Introduction
Rojas, R. (1996) · Medical Entomology and Zoology · Cited by 2,138
Learnability and the Vapnik-Chervonenkis dimension
Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M. K. (1989) · Journal of the ACM · doi:10.1145/76359.76371 · Cited by 1,839
Geometry, Topology and Physics
Nakahara, M. (2018) · CRC Press eBooks · doi:10.1201/9781315275826 · Cited by 1,505
Approximation of dynamical systems by continuous time recurrent neural networks
Funahashi, K.-I., & Nakamura, Y. (1993) · Neural Networks · doi:10.1016/s0893-6080(05)80125-x · Cited by 980
The Complexity of Boolean Functions
Wegener, I. (1987) · Cited by 882
Learning State Space Trajectories in Recurrent Neural Networks
Pearlmutter, B. A. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.263 · Cited by 656
Almost optimal lower bounds for small depth circuits
Hastad, J. (1986) · Proceedings of the eighteenth annual ACM symposium on Theory of computing - STOC '86 · doi:10.1145/12130.12132 · Cited by 620
Scarselli, F., & Chung Tsoi, A. (1998) · Neural Networks · doi:10.1016/s0893-6080(97)00097-x · Cited by 571
Self-citationSeparating the polynomial-time hierarchy by oracles
Yao, A. C.-C. (1985) · 26th Annual Symposium on Foundations of Computer Science (sfcs 1985) · doi:10.1109/sfcs.1985.49 · Cited by 548
The role of the primary visual cortex in higher level vision
Lee, T. S., Mumford, D., Romero, R., & Lamme, V. A. F. (1998) · Vision Research · doi:10.1016/s0042-6989(97)00464-1 · Cited by 534
On the Betti numbers of real varieties
Milnor, J. (1964) · Proceedings of the American Mathematical Society · doi:10.1090/s0002-9939-1964-0161339-9 · Cited by 500
A general framework for adaptive processing of data structures
Frasconi, P., Gori, M., & Sperduti, A. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/72.712151 · Cited by 471
The Complexity of Boolean Functions
(2005) · Complexity Theory · doi:10.1007/3-540-27477-4_16 · Cited by 376
Kaczynski, T., Mischaikow, K., & Mrozek, M. (2004) · Applied Mathematical Sciences · doi:10.1007/b97315 · Cited by 372
Coverage and hole-detection in sensor networks via homology
Ghrist, R., & Muhammad, A. (2005) · IPSN 2005. Fourth International Symposium on Information Processing in Sensor Networks, 2005. · doi:10.1109/ipsn.2005.1440933 · Cited by 289
On the power of small-depth threshold circuits
H�stad, J., & Goldmann, M. (1991) · Computational Complexity · doi:10.1007/bf01272517 · Cited by 247
Computational Capabilities of Graph Neural Networks
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005141 · Cited by 225
Self-citationA quantitative theory of immediate visual recognition
Serre, T., Kreiman, G., Kouh, M., Cadieu, C., Knoblich, U., & Poggio, T. (2007) · Progress in Brain Research · doi:10.1016/s0079-6123(06)65004-8 · Cited by 219
On the Expressive Power of Deep Architectures
Bengio, Y., & Delalleau, O. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24477-3_1 · Cited by 186
On the Expressive Power of Deep Architectures
Bengio, Y., & Delalleau, O. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24412-4_3 · Cited by 182
Natural language grammatical inference with recurrent neural networks
Lawrence, S., Giles, C. L., & Fong, S. (2000) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/69.842255 · Cited by 168
Feedforward nets for interpolation and classification
Sontag, E. D. (1992) · Journal of Computer and System Sciences · doi:10.1016/0022-0000(92)90039-l · Cited by 158
Polynomial Bounds for VC Dimension of Sigmoidal and General Pfaffian Neural Networks
Karpiński, M., & Macintyre, A. J. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1477 · Cited by 138
VC dimension of neural networks
Sontag, E. D. (1998) · Cited by 126
Real Solutions to Equations from Geometry
Sottile, F. (2011) · University lecture series · doi:10.1090/ulect/057 · Cited by 63
Vapnik-Chervonenkis dimension of neural nets
Bartlett, P. L., & Maass, W. (2003) · Cited by 54
Recursive neural networks for processing graphs with labelled edges: theory and applications
Bianchini, M., Maggini, M., Sarti, L., & Scarselli, F. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.07.003 · Cited by 48
Self-citationIdentification of Chemical Processes using Recurrent Networks
Su, H.-T., & McAvoy, T. J. (1991) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.1991.4791818 · Cited by 43
Learning long-term dependencies using layered graph neural networks
Bandinelli, N., Bianchini, M., & Scarselli, F. (2010) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2010.5596634 · Cited by 35
Self-citationRecursive neural networks learn to localize faces
Bianchini, M., Maggini, M., Sarti, L., & Scarselli, F. (2005) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.03.010 · Cited by 26
Self-citationBetti numbers of semi-Pfaffian sets
Zell, T. (1999) · Journal of Pure and Applied Algebra · doi:10.1016/s0022-4049(99)00017-1 · Cited by 18
Approximation capabilities of folding networks
Hammer, B. (1999) · Cited by 17
Computing customized page ranks
Tsoi, A. C., Hagenbuchner, M., & Scarselli, F. (2006) · ACM Transactions on Internet Technology · doi:10.1145/1183463.1183466 · Cited by 16
Self-citationInvestigation into the application of graph neural networks to large-scale recommender systems
Pucci, A., Gori, M., Hagenbuchner, M., Scarselli, F., & Tsoi, A. C. (2006) · Systems Science · Cited by 9
Self-citation
Cited by
NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results
Timofte, R., Agustsson, E., Gool, L. V., Yang, M.-H., Zhang, L., Lim, B., Son, S., Kim, H., Nah, S., Lee, K. M., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.149 · Cited by 1,441
Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06228 · Cited by 1,246
Error bounds for approximations with deep ReLU networks
Yarotsky, D. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.07.002 · Cited by 1,085
Deep neural networks, gradient-boosted trees, random forests: Statistical arbitrage on the S&P 500
Krauss, C., Do, X. A., & Huck, N. (2017) · European Journal of Operational Research · doi:10.1016/j.ejor.2016.10.031 · Cited by 750
Resnet in Resnet: Generalizing Residual Architectures
Targ, S., Almeida, D., & Lyman, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08029 · Cited by 627
Metadata
- Authors
- Bianchini, M., & Scarselli, F.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2014-08
- Volume
- 25
- Issue
- 8
- Pages
- 1553-1565
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W2089947415
- Cited by
- 500
- References
- 47
- FWCI
- 25.09
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.