CoChat
Sign Up
PreprintCited by 13Stochastic Gradient Optimization TechniquesMachine Learning and Data ClassificationDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record.

Fisher SAM: Information Geometry and Sharpness Aware Minimisation

Open access
Kim, Minyoung, Li, Da, Hu, Shell Xu, Hospedales, Timothy M.·2022·arXiv
DOI 10.48550/arxiv.2206.04920arXiv 2206.04920
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent sharpness-aware minimisation (SAM) is known to find flat minima which is beneficial for better generalisation with improved robustness. SAM essentially modifies the loss function by reporting the maximum loss value within the small neighborhood around the current iterate. However, it uses the Euclidean ball to define the neighborhood, which can be inaccurate since loss functions for neural networks are typically defined over probability distributions (e.g., class predictive probabilities), rendering the parameter space non Euclidean. In this paper we consider the information geometry of the model parameter space when defining the neighborhood, namely replacing SAM's Euclidean balls with ellipsoids induced by the Fisher information. Our approach, dubbed Fisher SAM, defines more accurate neighborhood structures that conform to the intrinsic metric of the underlying statistical manifold. For instance, SAM may probe the worst-case loss value at either a too nearby or inappropriately distant point due to the ignorance of the parameter space geometry, which is avoided by our Fisher SAM. Another recent Adaptive SAM approach stretches/shrinks the Euclidean ball in accordance with the scale of the parameter magnitudes. This might be dangerous, potentially destroying the neighborhood structure. We demonstrate improved performance of the proposed Fisher SAM on several benchmark datasets/tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2206.04920

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 53

  1. Amari, S. Natural gradient works efficiently in learning. Neural Computation , 10(2):251-276, 1998.
  2. Arora, S., Ge, R., Neyshabur, B., and Zhang, Y. Stronger Generalization Bounds for Deep Nets via a Compression Approach. In International Conference on Machine Learning , 2018.
  3. Bottou, L., Curtis, F. E., and Nocedal, J. Optimization methods for large-scale machine learning. Siam Review , 60(2):223-311, 2018.
  4. Boyd, S. and Vandenberghe, L. Convex Optimization . Cambridge: Cambridge University Press, 2004.
  5. Cha, J., Chun, S., Lee, K., Cho, H.-C., Park, S., Lee, Y ., and Park, S. Swad: Domain generalization by seeking flat minima. NeurIPS , 2021.
  6. Chatterji, N. S., Neyshabur, B., and Sedghi, H. The intriguing role of module criticality in the generalization of deep networks. In International Conference on Learning Representations , 2020.
  7. Chaudhar, P., Choromansk, A., Soatt, S., LeCun, Y., Baldass, C., Borg, C., Chays, J., Sagun, L., and Zecchina, R. Entropy-sgd: Biasing gradient descent into wide valleys. In ICLR , 2017.
  8. Chen, X., Liu, C., Li, B., Lu, K., and Song, D. Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. In arXiv preprint arXiv:1712.05526 , 2017. · arXiv
  9. Cochran, W. G. Sampling Techniques . Wiley, Palo Alto, CA, 1977.
  10. Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation strategies from data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019.
  11. Dai, J., Chen, C., and Li, Y. A Backdoor Attack Against LSTM-Based Text Classification Systems. In IEEE Access 7 , 2019.
  12. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition , 2009.
  13. DeVries, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017. · arXiv
  14. Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. Sharp minima can generalize for deep nets. In International Conference on Machine Learning , 2017.
  15. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR , 2021.
  16. Dziugaite, G. K. and Roy, D. M. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. UAI , 2017.
  17. Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.
  18. Graves, A. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems , 2011.
  19. Gu, T., Liu, K., Dolan-Gavitt, B., and Garg, S. BadNets: Evaluating Backdooring Attacks on Deep Neural Networks. In IEEE Access 7 , 2019.
  20. Han, D., Kim, J., and Kim, J. Deep pyramidal residual networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2017.
  21. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2016.
  22. Hochreiter, S. and Schmidhuber, J. Simplifying neural nets by discovering flat minima. In Advances in Neural Information Processing Systems , 1995.
  23. Hochreiter, S. and Schmidhuber, J. Flat minima. Neural Computation , 9(1):1-42, 1997.
  24. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In CVPR , 2017.
  25. Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. In UAI , 2018.
  26. Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. In ICLR , 2020.
  27. Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR , 2017.
  28. Khan, M. E., Nielsen, D., Tangkaratt, V., Lin, W., Gal, Y., and Srivastava, A. Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam. In International Conference on Machine Learning , 2018.
  29. Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) , Sydney, Australia, 2013.
  30. Kwon, J., Kim, J., Park, H., and Choi, I. K. ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks. In International Conference on Machine Learning , 2021.
  31. Langford, J. and Caruana, R. (Not) Bounding the true error. In Advances in Neural Information Processing Systems , 2002.
  32. Laurent, B. and Massart, P. Adaptive estimation of a quadratic functional by model selection. Annals of Statistics , 28(5):1302-1338, 2000.
  33. Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. Visualizing the loss landscape of neural nets. In NIPS . 2018.
  34. Loshchilov, I. and Hutter, F. SGDR: Stochastic gradient descent with warm restarts. In arXiv preprint arXiv:1608.03983 , 2016. · arXiv
  35. Martens, J. New insights and perspectives on the natural gradient method. arXiv preprint arXiv:1412.1193 , 2014. · arXiv
  36. McAllester, D. A. PAC-Bayesian model averaging. In In Proceedings of the Twelfth Annual Conference on Computational Learning Theory , 1999.
  37. M¨ uller, R., Kornblith, S., and Hinton, G. E. When does label smoothing help? In Advances in Neural Information Processing Systems , 2019.
  38. Nagel, M., van Baalen, M., Blankevoort, T., and Welling, M. Data-Free Quantization Through Weight Equalization and Bias Correction. In IEEE/CVF International Conference on Computer Vision , 2019.
  39. Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems , 2017.
  40. Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing , 2008.
  41. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 . 2019.
  42. Rakin, A. S.; He, Z. and Fan, D. TBT: Targeted Neural Network Attack With Bit Trojan. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020.
  43. Rooyen, B. v., Menon, A. K., and Williamson, R. C. Learning with symmetric label noise: the importance of being unhinged. In Advances in Neural Information Processing Systems , 2015.
  44. Schraudolph, N. N. Fast curvature matrix-vector products for second-order gradient descent. Neural computation , 14(7):1723-1738, 2002.
  45. Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations , 2015.
  46. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. JMLR , 2014.
  47. Sun, X., Zhang, Z., Ren, X., Luo, R., and Li, L. Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption. In Proc. of AAAI Conference on Artificial Intelligence , 2021.
  48. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jegou, H. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning , 2021.
  49. Weng, T., Zhao, P., Liu, S., Chen, P., Lin, X., and Daniel, L. Towards Certificated Model Robustness Against Weight Perturbations. In AAAI Conference on Artificial Intelligence , 2020.
  50. Xie, S.and Girshick, R., Doll´ ar, P., Tu, Z., and He, K. Aggregated residual transformations for deep neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2017.
  51. Zagoruyko, S. and Komodakis, N. Wide residual networks. In BMVC , 2016.
  52. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. In ICLR , 2017.
  53. Zhu, Z., Wu, J., Yu, B., Wu, L., and Ma, J. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In International Conference on Machine Learning , 2019.

Cited by

Sorted by citations

See all 13

Metadata

Authors
Kim, M., Li, D., Hu, S. X., & Hospedales, T. M.
Affiliations
University of Edinburgh (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2206.04920
arXiv
2206.04920
OpenAlex
W4282813352
Cited by
13
References
0
Citations by year
2025: 3 · 2024: 6 · 2023: 4
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
Fisher information, information geometry, statistical manifolds, flat minima, generalization, robustness, neural network training, neighborhood structure

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review