The Loss Surfaces of Multilayer Networks
Open accessAbstract
We study the connection between the highly non-convex loss function of a simple model of the fully-connected feed-forward neural network and the Hamiltonian of the spherical spin-glass model under the assumptions of: i) variable independence, ii) redundancy in network parametrization, and iii) uniformity. These assumptions enable us to explain the complexity of the fully decoupled neural network through the prism of the results from random matrix theory. We show that for large-size decoupled networks the lowest critical values of the random loss function form a layered structure and they are located in a well-defined band lower-bounded by the global minimum. The number of local minima outside that band diminishes exponentially with the size of the network. We empirically verify that the mathematical model exhibits similar behavior as the computer simulations, despite the presence of high dependencies in real networks. We conjecture that both simulated annealing and SGD converge to the band of low critical points, and that all critical points found there are local minima of high quality measured by the test error. This emphasizes a major difference between large- and small-size networks where for the latter poor quality local minima have non-zero probability of being recovered. Finally, we prove that recovering the global minimum becomes harder as the network size increases and that it is in practice irrelevant as global minimum often leads to overfitting.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In short time, SG-MCMC suffers from a bounded estimation error and it loses surface when applied to the multi-layer networks.
“However, in short time, SG-MCMC suffers from a bounded estimation error [78] and it loses surface when applied to the multi-layer networks [79].”
From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [79]
The Introduction (p. 1) analyzes loss surfaces of multilayer networks, so the cited paper supports that topic. It does not say that SG-MCMC ‘loses surface’ when applied to those networks. The separate bounded-estimation-error clause is attached to [78], not this source. (LLM, conf=0.77)
Experimental — accuracy not yet measuredRecent work indicates that in practice, the loss surfaces of sufficiently over-parameterized NNs are surprisingly well-behaved and less prone to bad local minima than previously thought.
“Recent work indicates that in practice, the loss surfaces of sufficiently over-parameterized NNs are surprisingly wellbehaved and in particular less prone to bad local minima than previously thought [11, 17, 9].”
From Communication-Efficient Learning of Deep Networks from Decentralized Data (2016) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019) · Journal of Computational Physics · doi:10.1016/j.jcp.2018.10.045 · Cited by 20,384 · see citing works
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576 · see citing works
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · see citing works
“Recent work indicates that in practice, the loss surfaces of sufficiently over-parameterized NNs are surprisingly wellbehaved and in particular less prone to bad local minima than previously thought [11, 17, 9].”
BackgroundGeometric Deep Learning: Going beyond Euclidean data
Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., & Vandergheynst, P. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2693418 · Cited by 3,661 · see citing works
Self-citationEfficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Kamnitsas, K., Ledig, C., Newcombe, V. F. J., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., & Glocker, B. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.10.004 · Cited by 3,498 · see citing works
“Due to randomness during training the lo-cal minima where a network converges are different between training sessions and some errors they produce differ (Choromanska et al. (2015)).”
BackgroundInfluentialDeep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503 · see citing works
Survey on deep learning with class imbalance
Johnson, J. M., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0192-5 · Cited by 3,032 · see citing works
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 2,720 · see citing works
“However, in short time, SG-MCMC suffers from a bounded estimation error [73] and it loses surface when applied to the multi-layer networks [74].”
BackgroundUnderstanding deep learning (still) requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021) · Communications of the ACM · doi:10.1145/3446776 · Cited by 2,400 · see citing works
Towards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548 · see citing works
“A trillion-dimensional manifold may have a huge, unknown number of minima [160].”
BackgroundKraus, M., Feuerriegel, S., & Oztekin, A. (2020) · European Journal of Operational Research · doi:10.1016/j.ejor.2019.09.018 · Cited by 392 · see citing works
Non-convex Optimization for Machine Learning
Jain, P., & Kar, P. (2017) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000058 · Cited by 364 · see citing works
Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery
Kim, S., Lu, P. Y., Mukherjee, S., Gilbert, M., Jing, L., Ceperic, V., & Soljacic, M. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3017010 · Cited by 215 · see citing works
Mattson, P., Cheng, C., Coleman, C., Diamos, G., Micikevicius, P., Patterson, D., Tang, H., Wei, G.-Y., Bailis, P., Bittorf, V., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.01500 · Cited by 206 · see citing works
“DNN training involves many stochastic influences that manifest in substantial run-to-run variation (Choromanska et al., 2015; Gori & Tesi, 1992; Auer et al., 1996; Coleman et al., 2019 selection and the non-commutative nature of floating-point addition).”
BackgroundFast Yet Effective Machine Unlearning
Tarun, A. K., Chundawat, V. S., Mandal, M., & Kankanhalli, M. (2021) · arXiv · doi:10.48550/arxiv.2111.08947 · Cited by 190 · see citing works
Deep Neural Network Ensembles for Time Series Classification
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · 2019 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852316 · Cited by 103 · see citing works
Problems and opportunities in training deep learning software systems: an analysis of variance
Pham, H. V., Qian, S., Wang, J., Lutellier, T., Rosenthal, J., Tan, L., Yu, Y., & Nagappan, N. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416545 · Cited by 100 · see citing works
Gossip training for deep learning
Blot, M., Picard, D., Cord, M., & Thome, N. (2016) · arXiv · doi:10.48550/arxiv.1611.09726 · Cited by 77 · see citing works
Wireless Network Intelligence at the Edge
Park, J., Samarakoon, S., Bennis, M., & Debbah, M. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2941458 · Cited by 92 · see citing works
Quantifying and Localizing Usable Information Leakage from Neural Network Gradients
Mo, F., Borovykh, A., Malekzadeh, M., Demetriou, S., Gündüz, D., & Haddadi, H. (2021) · arXiv · doi:10.48550/arxiv.2105.13929 · Cited by 1 · see citing works
- Authors
- Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., & LeCun, Y.
- Affiliations
- Courant Institute of Mathematical Sciences (US)
- New York University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2014
- Volume
- 38
- Pages
- 192-204
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1412.0233
- OpenAlex
- W1899249567
- Cited by
- 733
- References
- 1
- Citations by year
- 2026: 3 · 2025: 18 · 2024: 32 · 2023: 38 · 2022: 61 · 2021: 123 · 2020: 114 · 2019: 132 · 2018: 99 · 2017: 63 · 2016: 35 · 2015: 13 · 2014: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Artificial Intelligence and Statistics
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Physics and Astronomy › Condensed Matter Physics › Theoretical and Computational Physics
- Keywords
- loss landscape, multilayer perceptron, non-convex optimization, local minima, critical points, random matrix theory, simulated annealing, stochastic gradient descent, overfitting, network size scaling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
