CoChat
Sign Up
PreprintCited by 1Stochastic Gradient Optimization TechniquesDomain Adaptation and Few-Shot LearningNeural Networks and Applications

The DOI matches its DataCite record.

Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It

Open access
da Silva, Marvin F., Dangel, Felix, Oore, Sageev·2025·arXiv
DOI 10.48550/arxiv.2505.05409arXiv 2505.05409
Open this paper in CoChat — free accountOpen PDF

Abstract

The concept of sharpness has been successfully applied to traditional architectures like MLPs and CNNs to predict their generalization. For transformers, however, recent work reported weak correlation between flatness and generalization. We argue that existing sharpness measures fail for transformers, because they have much richer symmetries in their attention mechanism that induce directions in parameter space along which the network or its loss remain identical. We posit that sharpness must account fully for these symmetries, and thus we redefine it on a quotient manifold that results from quotienting out the transformer symmetries, thereby removing their ambiguities. Leveraging tools from Riemannian geometry, we propose a fully general notion of sharpness, in terms of a geodesic ball on the symmetry-corrected quotient manifold. In practice, we need to resort to approximating the geodesics. Doing so up to first order yields existing adaptive sharpness measures, and we demonstrate that including higher-order terms is crucial to recover correlation with generalization. We present results on diagonal networks with synthetic data, and show that our geodesic sharpness reveals strong correlation for real-world transformers on both text and image classification tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.05409

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 32

  1. Adilova, L., Abourayya, A., Li, J., Dada, A., Petzka, H., Egger, J., Kleesiek, J., and Kamp, M. Fam: Relative flatness aware min- imization, 2023. URL https://arxiv.org/abs/2307. 02337.
  2. Boumal, N. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023.
  3. Croce, F. and Hein, M. Reliable evaluation of adversarial robust- ness with an ensemble of diverse parameter-free attacks, 2020. URL https://arxiv.org/abs/2003.01690. · arXiv
  4. Dagréou, M., Ablin, P., Vaiter, S., and Moreau, T. How to compute hessian-vector products? In ICLR Blog- posts 2024, 2024. URL https://iclr-blogposts. github.io/2024/blog/bench-hvp/. https://iclr- blogposts.github.io/2024/blog/bench-hvp/.
  5. Dziugaite, G. K., Drouin, A., Neal, B., Rajkumar, N., Caballero, E., Wang, L., Mitliagkas, I., and Roy, D. M. In search of robust mea- sures of generalization. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Informa- tion Processing Systems, volume 33, pp. 11723-11733.
  6. Curran Associates, Inc., 2020. URL https://proceedings. neurips.cc/paper_files/paper/2020/file/ 86d7c8a08b4aaa1bc7c599473f5dddda-Paper. pdf.
  7. Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness- aware minimization for efficiently improving generalization. In International Conference on Learning Representations (ICLR), 2021.
  8. Hochreiter, S. and Schmidhuber, J. Simplifying neural nets by discovering flat minima. In Advances in Neural Information Processing Systems (NIPS), 1994.
  9. Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
  10. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015.
  11. Kendall, M. G. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
  12. Kirrinnis, P. Fast algorithms for the sylvester equation. Theo- retical Computer Science, 259(1):623-638, 2001. ISSN 0304- 3975. doi: https://doi.org/10.1016/S0304-3975(00)00322-4. URL https://www.sciencedirect.com/science/ article/pii/S0304397500003224. · DOI
  13. Kristiadi, A., Dangel, F., and Hennig, P. The geometry of neural nets' parameter spaces under reparametrization. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
  14. Kunin, D., Sagastuy-Brena, J., Ganguli, S., Yamins, D. L., and Tanaka, H. Symmetry, conservation laws, and learning dynam- ics in neural networks. In International Conference on Learning Representations (ICLR), 2021.
  15. Kwon, J., Kim, J., Park, H., and Choi, I. K. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning (ICML), 2021.
  16. Lee, J. Introduction to Smooth Manifolds. Graduate Texts in Math- ematics. Springer, 2003. ISBN 9780387954486. URL https: //books.google.ca/books?id=eqfgZtjQceYC.
  17. Luo, Y., Li, X., and Zhang, A. R. On geometric connections of embedded and quotient geometries in riemannian fixed-rank matrix optimization, 2023. URL https://arxiv.org/ abs/2110.12121. · arXiv
  18. McCoy, R. T., Min, J., and Linzen, T. Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance, 2020. URL https: //arxiv.org/abs/1911.02969.
  19. Pesme, S., Pillaud-Vivien, L., and Flammarion, N. Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity, 2021. URL https://arxiv.org/abs/ 2106.09524.
  20. Petzka, H., Kamp, M., Adilova, L., Sminchisescu, C., and Boley, M. Relative flatness and generalization. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  21. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agar- wal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. URL https://arxiv.org/abs/2103.00020. · arXiv
  22. Rangamani, A., Nguyen, N. H., Kumar, A., Phan, D., Chin, S. H., and Tran, T. D. A Scale Invariant Flatness Measure for Deep Network Minima, February 2019. URL http: //arxiv.org/abs/1902.02434. arXiv:1902.02434 [cs, stat]. · arXiv
  23. Salimans, T. and Kingma, D. P. Weight normalization: A simple reparameterization to accelerate training of deep neural net- works. Advances in neural information processing systems (NeurIPS), 29, 2016.
  24. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
  25. Walter, N. P., Adilova, L., Vreeken, J., and Kamp, M. The uncanny valley: Exploring adversarial robustness from a flatness per- spective, 2025. URL https://arxiv.org/abs/2405. 16918.
  26. Williams, A., Nangia, N., and Bowman, S. A broad-coverage challenge corpus for sentence understanding through inference. In Walker, M., Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1101. URL https:// aclanthology.org/N18-1101. · DOI · Paper page
  27. Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., and Srebro, N. Kernel and rich regimes in overparametrized models, 2020. URL https://arxiv. org/abs/2002.09277.
  28. Wortsman, M., Ilharco, G., Yitzhak Gadre, S., Roelofs, R., Gontijo- Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Car- mon, Y., Kornblith, S., and Schmidt, L. Model soups: av- eraging weights of multiple fine-tuned models improves ac- curacy without increasing inference time. arXiv e-prints, art. arXiv:2203.05482, March 2022. doi: 10.48550/arXiv.2203. 05482. · DOI · Paper page
  29. Yen, J.-N., Si, S., Meng, Z., Yu, F., Surya Duvvuri, S., Dhillon, I. S., Hsieh, C.-J., and Kumar, S. LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization. arXiv e-prints, art. arXiv:2410.20625, October 2024. doi: 10. 48550/arXiv.2410.20625. · DOI
  30. Yu, H. and Wu, J. Compressing transformers: Features are low- rank, but weights are not! Proceedings of the AAAI Confer- ence on Artificial Intelligence, 37(9):11007-11015, Jun. 2023. doi: 10.1609/aaai.v37i9.26304. URL https://ojs.aaai. org/index.php/AAAI/article/view/26304. · DOI
  31. Zhao, B., Dehmamy, N., Walters, R., and Yu, R. Symmetry tele- portation for accelerated optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2022a.
  32. Zhao, Y., Zhang, H., and Hu, X. Penalizing gradient norm for efficiently improving generalization in deep learning. In ICML, 2022b.

Cited by

Sorted by citations

Metadata

Authors
da Silva, M. F., Dangel, F., & Oore, S.
Affiliations
  • Dalhousie University (CA)
  • Vector Institute (CA)
ORCID
Oore, Sageev
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2505.05409
arXiv
2505.05409
OpenAlex
W4416091725
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
  • Government of Canada
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
sharpness, transformer models, generalization, quotient manifolds, Riemannian geometry, geodesic balls, flatness, text classification, image classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review