The DOI matches its DataCite record; 30 of the 36 references it cites were found in a registry or an index.
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
Open accessAbstract
To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Using this, we study a number of hypotheses concerning smoothness, curvature, and sharpness in the deep learning literature. We then thoroughly analyze a crucial structural feature of the spectra: in non-batch normalized networks, we observe the rapid appearance of large isolated eigenvalues in the spectrum, along with a surprising concentration of the gradient in the corresponding eigenspaces. In batch normalized networks, these two effects are almost absent. We characterize these effects, and explain how they affect optimization speed through both theory and experiments. As part of this work, we adapt advanced tools from numerical linear algebra that allow scalable and accurate estimation of the entire Hessian spectrum of ImageNet-scale neural networks; this technique may be of independent interest in other applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While large eigenvalues impede NN training (Ghorbani et al., 2019), MSAs can help NNs learn better representations by suppressing large Hessian eigenvalues .”
From How Do Vision Transformers Work? (2022) · cites this paper as Ghorbani et al., 2019
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“For our analysis, we study a variety of Resnet and VGG [26] architectures on both CIFAR-10 and ImageNet.”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 2×
“The original batch normalization paper [14] observed that the normalization parameters of BN, σ B and µ B , have to be computed (and back-propagated through) using the mini-batch.”
MethodLanczos, C. (1950) · doi:10.1201/9781351069397-23 · Cited by 4,041 · cited 1×
“We use Lanczos algorithm (Lanczos, 1950) (with full re-orthogonalization) to perform this computation in a numerically stable manner.”
Visualizing the Loss Landscape of Neural NetsPreprint
Li, H., Xu, Z., Taylor, G., Studer, C., & Goldstein, T. (2017) · Neural Information Processing Systems · doi:10.48550/arxiv.1712.09913 · Cited by 2,497 · cited 2×
“Finally, Li et al. [19] recently hypothesized that adding residual connections significantly smooths the optimization landscape, producing a series of compelling two-dimensional visualizations.”
BackgroundOn Large-Batch Training for Deep Learning: Generalization Gap and Sharp MinimaPreprint
Keskar, N. S., Mudigere, D., Nocedal, J. M., Smelyanskiy, M., & Tang, P. T. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.04836 · Cited by 2,270 · cited 1×
“Examples of such work, where some surrogate is defined for the curvature, include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6],…”
BackgroundCalculation of Gauss quadrature rules
Golub, G., & Welsch, J. H. (1967) · Milestones in Matrix Computation · doi:10.1090/s0025-5718-69-99647-1 · Cited by 1,464 · cited 2×
“The algorithm we use is due to Golub and Welsch [10].”
MethodHow Does Batch Normalization Help Optimization?Preprint
Santurkar, S., Tsipras, D., Ilyas, A., & Mądry, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11604 · Cited by 1,075 · cited 3×
“…include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6], qualitative studies and visualizations of the loss surface [11], and…”
BackgroundResultInfluentialRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.00567 · Cited by 741 · cited 1×
“Scaling it to run on a 27 million parameter Inception V3 [27] on ImageNet (where we performed our largest scale experiments) requires a significant engineering effort.”
BackgroundThree Factors Influencing Minima in SGDPreprint
Jastrzebski, S., Kenton, Z., Arpit, D., Ballas, N., Fischer, A., Bengio, Y., & Storkey, A. (2017) · arXiv.org · doi:10.48550/arxiv.1711.04623 · Cited by 560 · cited 2×
“Examples of such work, where some surrogate is defined for the curvature, include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6], qualitative studies and…”
Background
Cited by
Sorted by citations
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
“Making this connection quantitative will require a characterization of the Hessian spectrum [Pap18, GKX19, GARD18].”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application
Huang, L., Qin, J., Zhou, Y., Zhu, F., Liu, L., & Shao, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3250241 · Cited by 656
Towards Understanding Convergence and Generalization of AdamW
Zhou, P., Xie, X., Lin, Z., & Yan, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3382294 · Cited by 390
“Meanwhile, generally, we have x of Hessian matrix in deep networks are much smaller than one which is well observed in many works, e.g., fully connected networks, AlexNet, VGG and ResNet [49], [59], [60], [61] and our experimental results on ResNet50 and ViT-small in Fig.”
BackgroundResultHow Do Vision Transformers Work?
Park, N., & Kim, S. (2022) · arXiv · doi:10.48550/arxiv.2202.06709 · Cited by 356
“Ghorbani et al. (2019) visualized the Hessian eigenvalue spectrum by using the Lanczos quadrature algorithm for full batch.”
BackgroundMethodOptimization for Deep Learning: An Overview
Sun, R. (2020) · Journal of the Operations Research Society of China · doi:10.1007/s40305-020-00309-6 · Cited by 207
“Ghorbani et al. [32] numerically found that for networks without BatchNorm, there are large isolated eigenvalues, while for networks with BatchNorm this phenomenon does not occur.”
Background
Metadata
- Authors
- Ghorbani, B., Krishnan, S., & Xiao, Y.
- Affiliations
- Stanford University (US)
- Google (United States) (US)
- ORCID
- Krishnan, Shankar
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Pages
- 2232-2241
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1901.10159
- OpenAlex
- W2912713668
- Cited by
- 144
- References
- 36
- Citations by year
- 2025: 8 · 2024: 14 · 2023: 17 · 2022: 17 · 2021: 36 · 2020: 35 · 2019: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- Hessian spectrum, deep neural network optimization, batch normalization, isolated eigenvalues, curvature, sharpness, optimization speed, numerical linear algebra
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
