An Investigation into Neural Net Optimization via Hessian Eigenvalue Density

“For our analysis, we study a variety of Resnet and VGG [26] architectures on both CIFAR-10 and ImageNet.”

“The original batch normalization paper [14] observed that the normalization parameters of BN, σ B and µ B , have to be computed (and back-propagated through) using the mini-batch.”

“We use Lanczos algorithm (Lanczos, 1950) (with full re-orthogonalization) to perform this computation in a numerically stable manner.”

“Finally, Li et al. [19] recently hypothesized that adding residual connections significantly smooths the optimization landscape, producing a series of compelling two-dimensional visualizations.”

“Examples of such work, where some surrogate is defined for the curvature, include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6],…”

“The algorithm we use is due to Golub and Welsch [10].”

“…include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6], qualitative studies and visualizations of the loss surface [11], and…”

“Scaling it to run on a 27 million parameter Inception V3 [27] on ImageNet (where we performed our largest scale experiments) requires a significant engineering effort.”

“Examples of such work, where some surrogate is defined for the curvature, include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6], qualitative studies and…”

“Explicit representation of the Hessian matrix is infeasible when n is large, but using Pearlmutter’s trick [22] we are able to compute Hessian-vector products for any chosen vector.”

“Sagun et al. [23, 24] study the spectrum of the Hessian for small two-layer feed-forward networks.”

“Examples of such work, where some surrogate is defined for the curvature, include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6],…”

“It turns out that if the integrand were a polynomial g of degree d , with d small enough compared to m , it is possible to compute the integral exactly, Theorem 2.1 ([9] Chapter 6) .”

“This is in accordance with the ideas of Li et al. [18], who hypothesize that the loss surface has low intrinsic dimensionality, and also with results of Sagun et al. on toy models.”

“…explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6], qualitative studies and visualizations of the loss surface [11], and characterization of the intrinsic dimensionality of the loss [18].”

“Sagun et al. [23, 24] study the spectrum of the Hessian for small two-layer feed-forward networks.”

“Throughout the optimization, we observe that almost all of the gradient energy is concentrated in these outlier subspaces (Figure 11), reproducing an observation of Gur-Ari et al. [12] 7 .”

“The underlying algorithm is extremely elegant, and has been known in the numerical analysis literature for decades (Bai et al., 1996) (based on foundational work by Golub and Welsch (1969)); here we introduce it to the machine learning community, and build (and release) a system to run it at modern deep learning scale.”

“…Resnet-32 with no batch normalization, a small ball around 0 accounts for almost 50% of the total L 1 energy of the Hessian eigenvalues for a converged model (the L 1 reflects the loss function Thus to achieve successful optimization, we are forced to optimize these slowly converging directions 6 .”

“Examples of such work, where some surrogate is defined for the curvature, include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6],…”

“There is an extensive literature regarding estimating the eigenvalues distribution of large matrices (for a small survey, see [20]).”

“There exists an extensive literature on bounding this error; [28] prove that under suitable conditions that where ρ > 1 .”

“Previous studies on the Hessian have focused on small models, or are limited to computing only a few eigenvalues [23, 24, 30].”

“We perform a thorough comparison of our work to the recent proposal of [2] in Appendix D. Batch Normalization (BN) [14] is one of the most influential innovations in optimizing deep neural networks as it substantially reduces the training time and the dependence of the training on initialization.”

“During the preparation of this paper, [21] appeared on Arxiv which briefly introduces the same spectrum estimation methodology and studies the Hessian on small subsamples of MNIST and CIFAR-10 at the end of the training.”

“The following lemma is one result on the concentration of quadratic forms: Lemma A.1 (Concentration of Quadratic Forms, [3]) .”

“Making this connection quantitative will require a characterization of the Hessian spectrum [Pap18, GKX19, GARD18].”

“Meanwhile, generally, we have x of Hessian matrix in deep networks are much smaller than one which is well observed in many works, e.g., fully connected networks, AlexNet, VGG and ResNet [49], [59], [60], [61] and our experimental results on ResNet50 and ViT-small in Fig.”

“Ghorbani et al. (2019) visualized the Hessian eigenvalue spectrum by using the Lanczos quadrature algorithm for full batch.”

“Ghorbani et al. [32] numerically found that for networks without BatchNorm, there are large isolated eigenvalues, while for networks with BatchNorm this phenomenon does not occur.”