Qualitatively characterizing neural network optimization problems
Open accessAbstract
Training neural networks involves solving large-scale non-convex optimization problems. This task has long been believed to be extremely difficult, with fear of local minima and other obstacles motivating a variety of schemes to improve optimization, such as unsupervised pretraining. However, modern neural networks are able to achieve negligible training error on complex tasks, using only direct training with stochastic gradient descent. We introduce a simple analysis technique to look for evidence that such networks are overcoming local optima. We find that, in fact, on a straight path from initialization to solution, a variety of state of the art neural networks never encounter any significant obstacles.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Averaging two MNIST digit-recognition models trained from different initial conditions produces bad behavior, with loss worse than the parent models.
“Following the approach of Goodfellow et al. [17], we see exactly this bad behavior when we average two MNIST digit-recognition models 3 trained from different initial conditions (Figure 1, left).”
From Communication-Efficient Learning of Deep Networks from Decentralized Data (2016) · cites this paper as [17]
Recent work indicates that in practice, the loss surfaces of sufficiently over-parameterized NNs are surprisingly well-behaved and less prone to bad local minima than previously thought.
“Recent work indicates that in practice, the loss surfaces of sufficiently over-parameterized NNs are surprisingly wellbehaved and in particular less prone to bad local minima than previously thought [11, 17, 9].”
From Communication-Efficient Learning of Deep Networks from Decentralized Data (2016) · cites this paper as [17]
Cites
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
Self-citationWhat is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · see citing works
“Recent work indicates that in practice, the loss surfaces of sufficiently over-parameterized NNs are surprisingly wellbehaved and in particular less prone to bad local minima than previously thought [11, 17, 9].”
BackgroundArtificial intelligence: A powerful paradigm for scientific research
Xu, Y., Liu, X., Cao, X., Huang, C., Liu, E., Qian, S., Liu, X., Wu, Y., Dong, F., Qiu, C.-W., et al. (2021) · The Innovation · doi:10.1016/j.xinn.2021.100179 · Cited by 1,726 · see citing works
Robust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428 · see citing works
Lipton, Z. C., & Steinhardt, J. (2019) · Queue · doi:10.1145/3317287.3328534 · Cited by 231 · see citing works
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
Ghorbani, B., Krishnan, S., & Xiao, Y. (2019) · arXiv · doi:10.48550/arxiv.1901.10159 · Cited by 144 · see citing works
“…explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6], qualitative studies and visualizations of the loss surface [11], and characterization of the intrinsic dimensionality of the loss [18].”
BackgroundUsing transfer learning to detect galaxy mergers
Ackermann, S., Schawinski, K., Zhang, C., Weigel, A. K., & Turp, M. D. (2018) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/sty1398 · Cited by 137 · see citing works
Toward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119 · see citing works
Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
McKinstry, J. L., Esser, S. K., Appuswamy, R., Bablani, D., Arthur, J. V., Yildiz, I. B., & Modha, D. S. (2018) · arXiv · doi:10.48550/arxiv.1809.04191 · Cited by 74 · see citing works
Decentralized Bayesian Learning over Graphs
Lalitha, A., Wang, X., Kilinc, O., Lu, Y., Javidi, T., & Koushanfar, F. (2019) · arXiv · doi:10.48550/arxiv.1905.10466 · Cited by 20 · see citing works
- Authors
- Goodfellow, I. J., Vinyals, O., & Saxe, A. M.
- Affiliations
- Google (United States) (US)
- Stanford University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2014
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1412.6544
- OpenAlex
- W1850240193
- Cited by
- 341
- References
- 2
- Citations by year
- 2025: 3 · 2024: 25 · 2023: 24 · 2022: 33 · 2021: 58 · 2020: 58 · 2019: 59 · 2018: 41 · 2017: 23 · 2016: 17
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- neural network training, non-convex optimization, local optima, stochastic gradient descent, unsupervised pre-training, optimization landscape
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
