Theoretical issues in deep networks
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
While deep learning is successful in a number of applications, it is not yet well understood theoretically. A theoretical characterization of deep learning should answer questions about their approximation power, the dynamics of optimization, and good out-of-sample performance, despite overparameterization and the absence of explicit regularization. We review our recent results toward this goal. In approximation theory both shallow and deep networks are known to approximate any continuous functions at an exponential cost. However, we proved that for certain types of compositional functions, deep networks of the convolutional type (even without weight sharing) can avoid the curse of dimensionality. In characterizing minimization of the empirical exponential loss we consider the gradient flow of the weight directions rather than the weights themselves, since the relevant function underlying classification corresponds to normalized networks. The dynamics of normalized weights turn out to be equivalent to those of the constrained problem of minimizing the loss subject to a unit norm constraint. In particular, the dynamics of typical gradient descent have the same critical points as the constrained problem. Thus there is implicit regularization in training deep networks under exponential-type loss functions during gradient flow. As a consequence, the critical points correspond to minimum norm infima of the loss. This result is especially relevant because it has been recently shown that, for overparameterized models, selection of a minimum norm solution optimizes cross-validation leave-one-out stability and thereby the expected error. Thus our results imply that gradient descent in deep networks minimize the expected error.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Advances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 8,890
Differential equations with discontinuous right hand sides
(1989) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(89)90171-7 · Cited by 4,893
Approximation theory of the MLP model in neural networks
Pinkus, A. (1999) · Acta Numerica · doi:10.1017/s0962492900002919 · Cited by 1,529
Error bounds for approximations with deep ReLU networks
Yarotsky, D. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.07.002 · Cited by 1,085
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
Salimans, T., & Kingma, D. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07868 · Cited by 883
On the Number of Linear Regions of Deep Neural Networks
Montúfar, G., Pascanu, R., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1402.1869 · Cited by 840
Proceedings of the 28th International Conference on Neural Information Processing Systems
Cortes, C., Lee, D., Sugiyama, M., & Garnett, R. (2015) · Cited by 747
Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review
Poggio, T., Mhaskar, H., Rosasco, L., Miranda, B., & Liao, Q. (2017) · International Journal of Automation and Computing · doi:10.1007/s11633-017-1054-2 · Cited by 559
Self-citationThe generalization error of random features regression: Precise asymptotics and double descent curve
Mei, S., & Montanari, A. (2019) · arXiv · doi:10.48550/arxiv.1908.05355 · Cited by 537
Optimal approximation of piecewise smooth functions using deep ReLU neural networks
Petersen, P., & Voigtlaender, F. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.08.019 · Cited by 432
The Mathematics of Learning: Dealing With Data
Smale, S. (2005) · 2005 International Conference on Neural Networks and Brain · doi:10.1109/icnnb.2005.1614860 · Cited by 398
Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks
Soltanolkotabi, M., Javanmard, A., & Lee, J. D. (2019) · IEEE Transactions on Information Theory · doi:10.1109/tit.2018.2854560 · Cited by 356
Deep vs. shallow networks: An approximation theory perspective
Mhaskar, H. N., & Poggio, T. (2016) · Analysis and Applications · doi:10.1142/s0219530516400042 · Cited by 322
Self-citationNeural Networks for Optimal Approximation of Smooth and Analytic Functions
Mhaskar, H. N. (1996) · Neural Computation · doi:10.1162/neco.1996.8.1.164 · Cited by 321
Gradient Descent Maximizes the Margin of Homogeneous Neural NetworksPreprint
Lyu, K., & Li, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05890 · Cited by 284
Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically BalancedPreprint
Du, S. S., Hu, W. P., & Lee, J. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00900 · Cited by 258
Arora, S., Du, S. S., Hu, W. P., Li, Z., & Wang, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08584 · Cited by 210
Just interpolate: Kernel “Ridgeless” regression can generalizePreprint
Liang, T., & Rakhlin, A. (2020) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 196
Niyogi, P., & Girosi, F. (1996) · Neural Computation · doi:10.1162/neco.1996.8.4.819 · Cited by 172
Approximation properties of a multilayered feedforward artificial neural network
Mhaskar, H. N. (1993) · Advances in Computational Mathematics · doi:10.1007/bf02070821 · Cited by 157
Representation Benefits of Deep Feedforward Networks
Telgarsky, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.08101 · Cited by 121
Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
Liang, T., Poggio, T., Rakhlin, A., & STOKES, J. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01530 · Cited by 109
Self-citationNeural Networks for Localized Approximation
Chui, C. K., Li, X., & Mhaskar, H. N. (1994) · Mathematics of Computation · doi:10.1090/s0025-5718-1994-1240656-2 · Cited by 106
Nonlinear Approximation and (Deep) ReLU NetworksPreprint
Daubechies, I. C., DeVore, R., Foucart, S., Hanin, B., & Petrova, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02199 · Cited by 93
Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep ModelsPreprint
Nacson, M. S., Gunasekar, S., Lee, J. D., Srebro, N., & Soudry, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07325 · Cited by 91
Consistency of Interpolation with Laplace Kernels is a High-Dimensional PhenomenonPreprint
Rakhlin, A., & Zhai, X.-Y. (2018) · Annual Conference Computational Learning Theory · doi:10.48550/arxiv.1812.11167 · Cited by 89
Limitations of the approximation capabilities of neural networks with one hidden layer
Chui, C. K., Li, X., & Mhaskar, H. N. (1996) · Advances in Computational Mathematics · doi:10.1007/bf02124745 · Cited by 81
SGD Learns the Conjugate Kernel Class of the Network
Daniely, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08503 · Cited by 64
The Mathematics of Learning: Dealing with Data
Poggio, T., & Smale, S. (2006) · International Conference on Neural Networks and Brain · doi:10.1109/icnnb.2005.1614546 · Cited by 59
Self-citationOn the Margin Theory of Feedforward Neural NetworksPreprint
Wei, C., Lee, J. D., Liu, Q., & Ma, T. (2018) · arXiv (Cornell University) · Cited by 59
Deep Convolutional Networks are Hierarchical Kernel MachinesPreprint
Anselmi, F., Rosasco, L., Tan, C., & Poggio, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.01084 · Cited by 47
Self-citationLearning Real and Boolean Functions: When Is Deep Better Than Shallow
Mhaskar, H. N., Liao, Q., & Poggio, T. (2016) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 45
Self-citationI-theory on depth vs width: hierarchical function composition
Poggio, T., Anselmi, F., & Rosasco, L. (2015) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 37
Self-citationNeural networks for localized approximation of real functions
Mhaskar, H. N. (2002) · Machine learning for signal processing ... · doi:10.1109/nnsp.1993.471870 · Cited by 27
A Provably Efficient Algorithm for Training Deep NetworksPreprint
Livni, R., Shalev‐Shwartz, S., & Shamir, O. (2013) · arXiv (Cornell University) · Cited by 17
Notes on Hierarchical Splines, DCLNs and i-theory
Poggio, T., Rosasco, L., Shashua, A., Cohen, N., & Anselmi, F. (2015) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 17
Self-citationOn the Risk of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels.Preprint
Liang, T., Rakhlin, A., & Zhai, X. (2019) · arXiv (Cornell University) · Cited by 17
Depth Separation in ReLU Networks for Approximating Smooth Non-Linear Functions.Preprint
Safran, I., & Shamir, O. (2016) · arXiv (Cornell University) · Cited by 13
Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of DimensionalityPreprint
Poggio, T., Mhaskar, H. N., Rosasco, L., et al. (2016) · arXiv (Cornell University) · Cited by 6
Self-citation
Cited by
Tabular data: Deep learning is not all you need
Shwartz-Ziv, R., & Armon, A. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.011 · Cited by 2,172
Deep physical neural networks trained with backpropagation
Wright, L. G., Onodera, T., Stein, M. M., Wang, T., Schachter, D. T., Hu, Z., & McMahon, P. L. (2022) · Nature · doi:10.1038/s41586-021-04223-6 · Cited by 727
The unreasonable effectiveness of deep learning in artificial intelligence
Sejnowski, T. J. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907373117 · Cited by 352
Semantic Communication: A Survey on Research Landscape, Challenges, and Future Directions
Getu, T. M., Kaddoum, G., & Bennis, M. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3520707 · Cited by 110
Implicit Gradient Regularization
Barrett, D. G. T., & Dherin, B. (2020) · arXiv · doi:10.48550/arxiv.2009.11162 · Cited by 95
Metadata
- Authors
- Poggio, T., Banburski, A., & Liao, Q.
- Venue
- Proceedings of the National Academy of Sciences
- Published
- 2020-06-09
- Volume
- 117
- Issue
- 48
- Pages
- 30039-30045
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- OpenAlex
- W2995896684
- Cited by
- 159
- References
- 41
- FWCI
- 8.43
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.