Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Open accessAbstract
A central challenge to many fields of science and engineering involves minimizing non-convex error functions over continuous, high dimensional spaces. Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for these local methods to find the global minimum is the proliferation of local minima with much higher error than the global minimum. Here we argue, based on results from statistical physics, random matrix theory, neural network theory, and empirical evidence, that a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems of practical interest. Such saddle points are surrounded by high error plateaus that can dramatically slow down learning, and give the illusory impression of the existence of a local minimum. Motivated by these arguments, we propose a new approach to second-order optimization, the saddle-free Newton method, that can rapidly escape high dimensional saddle points, unlike gradient descent and quasi-Newton methods. We apply this algorithm to deep or recurrent neural network training, and provide numerical evidence for its superior optimization performance.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Poor loss disrupts NN training, especially in the early phase of training.
“This poor loss disrupts NN training (Dauphin et al., 2014), especially in the early phase of training (Jastrzebski et al., 2020; 2021).”
From How Do Vision Transformers Work? (2022) · cites this paper as Dauphin et al., 2014
The loss landscapes of MSAs are non-convex, and this non-convexity disturbs NN optimization.
“This means that the loss landscapes of MSAs are non-convex, and this non-convexity disturbs NN optimization (Dauphin et al., 2014).”
From How Do Vision Transformers Work? (2022) · cites this paper as Dauphin et al., 2014
Recent work indicates that in practice, the loss surfaces of sufficiently over-parameterized NNs are surprisingly well-behaved and less prone to bad local minima than previously thought.
“Recent work indicates that in practice, the loss surfaces of sufficiently over-parameterized NNs are surprisingly wellbehaved and in particular less prone to bad local minima than previously thought [11, 17, 9].”
From Communication-Efficient Learning of Deep Networks from Decentralized Data (2016) · cites this paper as [11]
Cited by
Sorted by citations
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Self-citationCommunication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · see citing works
“Recent work indicates that in practice, the loss surfaces of sufficiently over-parameterized NNs are surprisingly wellbehaved and in particular less prone to bad local minima than previously thought [11, 17, 9].”
BackgroundAn Introduction to Deep Learning for the Physical Layer
O'Shea, T., & Hoydis, J. (2017) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2017.2758370 · Cited by 2,980 · see citing works
Norouzzadeh, M. S., Nguyen, A., Kosmala, M., Swanson, A., Palmer, M. S., Packer, C., & Clune, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1719367115 · Cited by 1,230 · see citing works
f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
Nowozin, S., Cseke, B., & Tomioka, R. (2016) · arXiv · doi:10.48550/arxiv.1606.00709 · Cited by 1,183 · see citing works
A deep learning framework for unsupervised affine and deformable image registration
de Vos, B. D., Berendsen, F. F., Viergever, M. A., Sokooti, H., Staring, M., & Išgum, I. (2019) · Medical Image Analysis · doi:10.1016/j.media.2018.11.010 · Cited by 818 · see citing works
Recent Advances in Adversarial Training for Adversarial Robustness
Bai, T., Luo, J., Zhao, J., Wen, B., & Wang, Q. (2021) · arXiv · doi:10.48550/arxiv.2102.01356 · Cited by 490 · see citing works
Recent Advances in Adversarial Training for Adversarial Robustness
Bai, T., Luo, J., Zhao, J., Wen, B., & Wang, Q. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/591 · Cited by 490 · see citing works
If deep learning is the answer, what is the question?
Saxe, A., Nelli, S., & Summerfield, C. (2021) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-00395-8 · Cited by 397 · see citing works
Non-convex Optimization for Machine Learning
Jain, P., & Kar, P. (2017) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000058 · Cited by 364 · see citing works
TFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098021 · Cited by 350 · see citing works
Song, H. F., Yang, G. R., & Wang, X.-J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004792 · Cited by 281 · see citing works
Lipton, Z. C., & Steinhardt, J. (2019) · Queue · doi:10.1145/3317287.3328534 · Cited by 231 · see citing works
How Do Vision Transformers Work?
Park, N., & Kim, S. (2022) · arXiv · doi:10.48550/arxiv.2202.06709 · Cited by 356 · see citing works
“This poor loss disrupts NN training (Dauphin et al., 2014), especially in the early phase of training (Jastrzebski et al.”
BackgroundProblems and opportunities in training deep learning software systems: an analysis of variance
Pham, H. V., Qian, S., Wang, J., Lutellier, T., Rosenthal, J., Tan, L., Yu, Y., & Nagappan, N. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416545 · Cited by 100 · see citing works
Universal discriminative quantum neural networks
Chen, H., Wossnig, L., Severini, S., Neven, H., & Mohseni, M. (2021) · Quantum Machine Intelligence · doi:10.1007/s42484-020-00025-7 · Cited by 89 · see citing works
Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models
Rocks, J. W., & Mehta, P. (2022) · Physical Review Research · doi:10.1103/physrevresearch.4.013201 · Cited by 74 · see citing works
Learning Weight Uncertainty with Stochastic Gradient MCMC for Shape Classification
Li, C., Stevens, A., Chen, C., Pu, Y., Gan, Z., & Carin, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.611 · Cited by 39 · see citing works
aSGD: Stochastic Gradient Descent with Adaptive Batch Size for Every Parameter
Shi, H., Yang, N., Tang, H., & Yang, X. (2022) · Mathematics · doi:10.3390/math10060863 · Cited by 11 · see citing works
- Authors
- Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y.
- Affiliations
- Université de Montréal (CA)
- Stanford University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2014
- Volume
- 27
- Pages
- 2933-2941
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1406.2572
- OpenAlex
- W2146989110
- Cited by
- 827
- References
- 0
- Citations by year
- 2026: 3 · 2025: 20 · 2024: 44 · 2023: 63 · 2022: 86 · 2021: 110 · 2020: 107 · 2019: 121 · 2018: 110 · 2017: 77 · 2016: 48 · 2015: 30 · 2014: 7 · 2012: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Mathematics › Numerical Analysis › Advanced Optimization Algorithms Research
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- saddle points, non-convex optimization, high-dimensional optimization, second-order optimization, local minima, deep neural network training, recurrent neural network training, optimization landscape
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
