The DOI matches its DataCite record; 30 of the 36 references it cites were found in a registry or an index.
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
Open accessAbstract
To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Using this, we study a number of hypotheses concerning smoothness, curvature, and sharpness in the deep learning literature. We then thoroughly analyze a crucial structural feature of the spectra: in non-batch normalized networks, we observe the rapid appearance of large isolated eigenvalues in the spectrum, along with a surprising concentration of the gradient in the corresponding eigenspaces. In batch normalized networks, these two effects are almost absent. We characterize these effects, and explain how they affect optimization speed through both theory and experiments. As part of this work, we adapt advanced tools from numerical linear algebra that allow scalable and accurate estimation of the entire Hessian spectrum of ImageNet-scale neural networks; this technique may be of independent interest in other applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While large eigenvalues impede NN training (Ghorbani et al., 2019), MSAs can help NNs learn better representations by suppressing large Hessian eigenvalues .”
From How Do Vision Transformers Work? (2022) · cites this paper as Ghorbani et al., 2019
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“For our analysis, we study a variety of Resnet and VGG [26] architectures on both CIFAR-10 and ImageNet.”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 2×
“The original batch normalization paper [14] observed that the normalization parameters of BN, σ B and µ B , have to be computed (and back-propagated through) using the mini-batch.”
MethodLanczos, C. (1950) · doi:10.1201/9781351069397-23 · Cited by 4,041 · cited 1×
“We use Lanczos algorithm (Lanczos, 1950) (with full re-orthogonalization) to perform this computation in a numerically stable manner.”
Visualizing the Loss Landscape of Neural NetsPreprint
Li, H., Xu, Z., Taylor, G., Studer, C., & Goldstein, T. (2017) · Neural Information Processing Systems · doi:10.48550/arxiv.1712.09913 · Cited by 2,497 · cited 2×
“Finally, Li et al. [19] recently hypothesized that adding residual connections significantly smooths the optimization landscape, producing a series of compelling two-dimensional visualizations.”
BackgroundOn Large-Batch Training for Deep Learning: Generalization Gap and Sharp MinimaPreprint
Keskar, N. S., Mudigere, D., Nocedal, J. M., Smelyanskiy, M., & Tang, P. T. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.04836 · Cited by 2,270 · cited 1×
“Examples of such work, where some surrogate is defined for the curvature, include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6],…”
BackgroundCalculation of Gauss quadrature rules
Golub, G., & Welsch, J. H. (1967) · Milestones in Matrix Computation · doi:10.1090/s0025-5718-69-99647-1 · Cited by 1,464 · cited 2×
“The algorithm we use is due to Golub and Welsch [10].”
MethodHow Does Batch Normalization Help Optimization?Preprint
Santurkar, S., Tsipras, D., Ilyas, A., & Mądry, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11604 · Cited by 1,075 · cited 3×
“…include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6], qualitative studies and visualizations of the loss surface [11], and…”
BackgroundResultInfluentialRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.00567 · Cited by 741 · cited 1×
“Scaling it to run on a 27 million parameter Inception V3 [27] on ImageNet (where we performed our largest scale experiments) requires a significant engineering effort.”
BackgroundThree Factors Influencing Minima in SGDPreprint
Jastrzebski, S., Kenton, Z., Arpit, D., Ballas, N., Fischer, A., Bengio, Y., & Storkey, A. (2017) · arXiv.org · doi:10.48550/arxiv.1711.04623 · Cited by 560 · cited 2×
“Examples of such work, where some surrogate is defined for the curvature, include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6], qualitative studies and…”
Background
Cited by
Sorted by citations
Items 101–143 of 143
Spherical perspective on learning with normalization layers
Roburin, S., de Mont-Marin, Y. D., Bursuc, A., Marlet, R., Pérez, P., & Aubry, M. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.02.021 · Cited by 3
“The argument of Internal Covariate Shift reduction [2] has been challenged and shown to be secondary to smoothing of optimization landscape [7, 23] or its modification by creating a different objective function [24], enabling of high learning rates through improved conditioning [9], or alleviating…”
Incremental Trainable Parameter Selection in Deep Neural Networks
Thakur, A., Abrol, V., Sharma, P., Zhu, T., & Clifton, D. A. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3210297 · Cited by 3
“The Hessian matrix H ∈ R N × N is a symmetric matrix that can provide information about the learning dynamics of the neural networks [29], [32], [33].”
BackgroundFlatness is a False FriendPreprint
Granziol, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09091 · Cited by 3
“The Low rank nature of the Hessian: Previous empirical works [8, 30, 34, 35, 4] have noted a large portion of zero or near zero eigenvalues in the Hessian of deep neural networks.”
BackgroundSpherical Perspective on Learning with Batch Norm.Preprint
Roburin, S., de Mont-Marin, Y. D., Bursuc, A., Marlet, R., Pérez, P., & Aubry, M. (2020) · arXiv (Cornell University) · Cited by 3
Spectral Analysis and Stability of Deep Neural Dynamics.Preprint
Drgoňa, J., Skomski, E., Vasisht, S., Tuor, A. R., & Vrabie, D. (2020) · arXiv (Cornell University) · Cited by 3
A study on the plasticity of neural networks
Berariu, T., Czarnecki, W., De, S., Bornschein, J., Smith, S., Pascanu, R., & Clopath, C. (2021) · arXiv · doi:10.48550/arxiv.2106.00042 · Cited by 3
“(Ghorbani et al., 2019) computes the full spectrum of the Hessian showing that Jastrzebski et al.’s claims about smaller learning rates guiding the network into sharper minima doesn’t hold empirically.”
ConvNets Landscape Convergence: Hessian-Based Analysis of Matricized Networks
Meshkov, V., Kiselev, N., & Grabovoy, A. (2024) · Ivannikov ISPRAS Open Conference (ISPRAS) · doi:10.1109/ispras64596.2024.10899113 · Cited by 2
“Authors of the work [16] develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process.”
Review of optimization methods for supervised deep learning
Jiang, L., Yifeng, Z., Chen, C., Li, G., & Zhang, W. (2023) · Journal of Image and Graphics · doi:10.11834/jig.211139 · Cited by 2
Constituent Attention for Vision Transformers
Li, H., Xue, M., Song, J., Zhang, H., Huang, W., Liang, L., & Song, M. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103838 · Cited by 2
Spectral alignment of stochastic gradient descent for high-dimensional classification tasks
Arous, G. B., Gheissari, R., Huang, J., & Jagannath, A. (2025) · The Annals of Applied Probability · doi:10.1214/25-aap2187 · Cited by 17
Roulette: A Pruning Framework to Train a Sparse Neural Network From Scratch
Zhong, Q., Zhang, Z., Qiu, Q., & Cheng, X. (2021) · IEEE Access · doi:10.1109/access.2021.3065406 · Cited by 2
Heo, J. P., Im, C. G., Ryu, K. H., Sung, S. W., Yoo, C., & Yang, D. R. (2022) · Processes · doi:10.3390/pr10061157 · Cited by 2
A random matrix theory approach to damping in deep learning
Granziol, D., & Baskerville, N. P. (2022) · Journal of Physics Complexity · doi:10.1088/2632-072x/ac730c · Cited by 5
“…and Grosse, 2015]) can be interpreted as a trust region [Dauphin et al., 2014] that is required to stop the optimiser moving too far in directions deemed flat ( λ i ≈ 0 ), known to dominate the spectrum in practice [Granziol et al., 2020b, Papyan, 2018, Ghorbani et al., 2019], and hence diverging.”
BackgroundCreate and Find Flatness: Building Flat Training Spaces in Advance for Continual Learning
Shi, W., Chen, Y., Zhao, Z., Lu, W., Yan, K., & Du, X. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230509 · Cited by 3
A Simple and Efficient Tensor Calculus for Machine Learning
Laue, S., Mitterreiter, M., & Giesen, J. (2020) · Fundamenta Informaticae · doi:10.3233/fi-2020-1984 · Cited by 3
“Furthermore, some differences in the convergence behavior can be explained by looking at the spectrum of the Hessian of the objective function [21, 22, 23].”
BackgroundA Reparameterization-Invariant Flatness Measure for Deep Neural NetworksPreprint
Petzka, H., Adilova, L., Kamp, M., & Sminchisescu, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.00058 · Cited by 2
“An Averaging Alternative Experimental work [2] suggests that the spectrum of the Hessian has a lot of small values and only a few large outliers.”
BackgroundA spherical analysis of Adam with Batch NormalizationPreprint
Roburin, S., de Mont-Marin, Y. D., Bursuc, A., Marlet, R., Pérez, P., & Aubry, M. (2021) · arXiv (Cornell University) · Cited by 2
Rao, V., & Sohl‐Dickstein, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10687 · Cited by 2
“We estimate the Hessian for the cross-entropy loss using the Lanczos al-gorithm as described in Ghorbani et al. (2019) using 512 samples of Cifar-10.”
BackgroundMethodInfluentialRidge Rider: Finding Diverse Solutions by Following Eigenvectors of the HessianPreprint
Parker-Holder, J., Metz, L., Resnick, C., Hu, H., Lerer, A. K., Letcher, A., Peysakhovich, A., Pacchiano, A., & Foerster, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06505 · Cited by 4
Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic ModelsPreprint
Paquette, C., & Paquette, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03696 · Cited by 2
“…spectrum of the Hessians of neural networks share characteristics with the limiting spectral distributions found in classical random matrix theory [Behrooz et al., 2019, Dauphin et al., 2014, Granziol et al., 2020, Liao et al., 2020, Martin and Mahoney, 2018, Papyan, 2018, Pennington and B.,…”
BackgroundLow Rank Saddle Free Newton: A Scalable Method for Stochastic Nonconvex OptimizationPreprint
O’Leary-Roseberry, T., Alger, N., & Ghattas, O. N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02881 · Cited by 6
“Recent empirical studies of their spectra show that neural network training Hessians are typically numerically low rank, and when away from local minima often have at least one large magnitude negative eigenvalue (Alain et al., 2019; Ghorbani et al., 2019; Sagun et al., 2016).”
BackgroundMethodArjevani, Y., & Field, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.10370 · Cited by 2
Sublinear Time Eigenvalue Approximation via Random Sampling
Bhattacharjee, R., Dexter, G., Petros Drineas, B., Musco, C., & Ray, A. (2024) · Algorithmica · doi:10.1007/s00453-024-01208-5 · Cited by 1
A Dynamic Sampling Adaptive-SGD Method for Machine LearningPreprint
Bahamou, A., & Goldfarb, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.13357 · Cited by 1
“Although the loss function corresponding to neural network models is generally non-convex, we were motivated to apply our framework to training deep learning models based on the recent experimental observations in (Ghorbani et al., 2019).”
MethodExplaining the Adaptive Generalisation Gap.Preprint
Granziol, D., Albanie, S., Wan, X., & Roberts, S. (2020) · arXiv (Cornell University) · Cited by 1
Federated Few-Shot Learning with Adversarial LearningPreprint
Fan, C., & Huang, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.00365 · Cited by 1
Generalisation under gradient descent via deterministic PAC-BayesPreprint
Clerico, E., Farghly, T., Deligiannidis, G., Guedj, B., & Arnaud Doucet, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.02525 · Cited by 9
“As for the Hessian-dependent term, although in general it is difficult to characterise the spectrum of the Hessian, empirical studies on deep neural networks suggest that in the later stages of training, the Hessian has only a few dominant positive Hessian eigenvalues (Sagun et al., 2018; Ghorbani et al., 2019; Sankar…”
BackgroundLiu, Y., Li, X., & Liu, Z. (2022) · Research Square · doi:10.21203/rs.3.rs-2192513/v1 · Cited by 1
Quantum many-body physics with ultracold atoms and molecules: exact dynamics and machine learning
Lekowska, A. M. D. (2022) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.5821/dissertation-2117-379455 · Cited by 1
DGSAM: Domain Generalization via Individual Sharpness-Aware MinimizationPreprint
Song, Y., Hwang, Y., Lee, J.-H., Lee, H., & Lim, D. (2025) · Qeios · doi:10.32388/clu2qf · Cited by 4
The Probabilistic Fault Tolerance of Neural Networks in the Continuous LimitPreprint
El-Mhamdi, E.-M., Guerraoui, R., Kucharavy, A., & Volodin, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.01686
Group Whitening: Balancing Learning Efficiency and Representational CapacityPreprint
Huang, L., Zhou, Y., Liu, L., Zhu, F., & Shao, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13333
“This argument has been further investigated by computing the spectrum of the Hessian for a large-scale dataset [13].”
Finding Critical and Gradient-Flat Points of Deep Neural Network Loss Functions
Frye, C. G. (2020) · DigitalCommons-IMSA (Illinois Mathematics and Science Academy)
ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurePreprint
Dangel, F., Tatzel, L., & Hennig, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02624
“…Newton steps (Martens, 2010; Zhang et al., 2017; Gargiani et al., 2020) and spectral properties (Sagun et al., 2017; 2018; Adams et al., 2018; Ghorbani et al., 2019; Papyan, 2019b; Yao et al., 2019; Granziol et al., 2021) on arbitrary architectures thanks to the generality of automatic…”
BackgroundTakagi, S., Nagano, Y., Yoshida, Y., & Okada, M. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533904
“Yet, it is known that negative eigenvalues of Hessian in neural network training are scarce and very small [31], [32].”
BackgroundThe Equilibrium Hypothesis: Rethinking implicit regularization in Deep Neural Networks.Preprint
Lou, Y., Mingard, C., & Hayou, S. (2021) · arXiv (Cornell University)
Huang, L. (2022) · Synthesis lectures on computer vision · doi:10.1007/978-3-031-14595-7_9
Adaptive Low-Rank Gradient Descent
Jadbabaie, A., Makur, A., & Reisizadeh, A. (2023) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc49753.2023.10383982
Towards a Geometric Understanding of Spatiotemporal Graph Convolution Networks
Das, P., Shekkizhar, S., & Ortega, A. (2024) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2024.3396635
“Function approximation methods are based on the inductive bias of the loss function [10], the ability of the optimization to achieve good minima [11], or consider the study of classifier margins [12].”
MethodKireeva, N. V., & Tsivadze, A. Y. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-4c82v
Structured Directional Pruning via Perturbation Orthogonal Projection
Liu, X., Wang, Q., Shao, Y., Geng, Y., & Li, Y. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3501674 · Cited by 4
“Since ∇ f ( w (cid:63) ) ≈ 0 , the Hessian ∇ 2 (cid:96) ( w (cid:63) ) has multiple nearly zero eigenvalues [25, 26].”
BackgroundThe role of the time-dependent Hessian in high-dimensional optimization
Bonnaire, T., Biroli, G., & Cammarota, C. (2025) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/adf4bc
“…provides a showcase for a strong advantage of spectral initializations and, more generally, of spectral properties to improve optimization in non-convex and high-dimensional landscape – a research direction that received a lot of attention recently in the context of deep learning [62–64].”
Sharpness-Aware Minimization for Efficiently Improving Generalization
Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. (2020) · arXiv · doi:10.48550/arxiv.2010.01412 · Cited by 521
“Due to the parameter space’s dimensionality, we approximate the Hessian spectrum using the Lanczos algorithm of Ghorbani et al. (2019).”
Method
Metadata
- Authors
- Ghorbani, B., Krishnan, S., & Xiao, Y.
- Affiliations
- Stanford University (US)
- Google (United States) (US)
- ORCID
- Krishnan, Shankar
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Pages
- 2232-2241
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1901.10159
- OpenAlex
- W2912713668
- Cited by
- 144
- References
- 36
- Citations by year
- 2025: 8 · 2024: 14 · 2023: 17 · 2022: 17 · 2021: 36 · 2020: 35 · 2019: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- Hessian spectrum, deep neural network optimization, batch normalization, isolated eigenvalues, curvature, sharpness, optimization speed, numerical linear algebra
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
