CoChat
Sign Up
PreprintCited by 156Stochastic Gradient Optimization TechniquesNeural Networks and ApplicationsMachine Learning and Data Classification

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

Open access
Entezari, Rahim, Sedghi, Hanie, Saukh, Olga, Neyshabur, Behnam·2021·arXiv
DOI 10.48550/arxiv.2110.06296arXiv 2110.06296
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them. Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We further provide a preliminary theoretical result to support our conjecture. Our conjecture has implications for lottery ticket hypothesis, distributed training, and ensemble methods.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2110.06296

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Different minima of neural networks can hardly be merged into a generalized one even if trained on the same datasets from the same initialization but with different random seeds.

    “different minima of neural networks can hardly be merged into a generalized one even if trained on the same datasets from the same initialization (but with different random seeds) [71, 72].”

    From WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (2024) · cites this paper as [71]

    The Introduction reports that networks trained from the same initialization are connected by linear paths of constant test error, contrary to the claim that such solutions can hardly be merged. The Discussion and Conclusion further argue that accounting for permutation invariance can remove loss barriers between different solutions. The paper does not present merging as generally impossible. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • The main reason different minima cannot be merged is the permutation invariance property of deep neural networks, meaning positions of neurons can be permuted without affecting network function; as a result, different minima reside in different loss basins.

    “The main reason is considered to be the permutation invariance property of deep neural networks, which means that the positions of neurons can be permuted without affecting the network function [71]; as a result, different minima reside in different loss basins [72].”

    From WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (2024) · cites this paper as [71]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. Baldassi, C., Pittorino, F., and Zecchina, R. (2020). Shaping the learning landscape in neural networks around wide flat minima. Proceedings of the National Academy of Sciences, 117(1):161-170.
  2. Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-15854.
  3. Brea, J., Simsek, B., Illing, B., and Gerstner, W. (2019a). Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape. arXiv preprint arXiv:1907.02911. · arXiv
  4. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
  5. Draxler, F., Veschgini, K., Salmhofer, M., and Hamprecht, F. (2018). Essentially no barriers in neural network energy landscape. In International conference on machine learning, pages 1309-1318.
  6. PMLR.
  7. Fort, S., Hu, H., and Lakshminarayanan, B. (2019). Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757. · arXiv
  8. Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M. (2020). Linear mode connectivity and the lottery ticket hypothesis. In ICML, volume 119, pages 3259-3269.
  9. PMLR.
  10. Freeman, C. D. and Bruna, J. (2016). Topology and geometry of half-rectified network optimization. arXiv preprint arXiv:1611.01540. · arXiv
  11. Geiger, M., Spigler, S., d'Ascoli, S., Sagun, L., Baity-Jesi, M., Biroli, G., and Wyart, M. (2019). Jamming transition as a paradigm to understand the loss landscape of deep neural networks. Physical Review E, 100(1):012115.
  12. He, X., Zhou, Z., and Thiele, L. (2018). Multi-task zipping via layer-wise neuron sharing. arXiv preprint arXiv:1805.09791. · arXiv
  13. Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. arXiv preprint arXiv:1806.07572. · arXiv
  14. Li, D., Ding, T., and Sun, R. (2018). Over-parameterized deep neural networks have no strict local minima for any continuous activations. arXiv preprint arXiv:1812.11039. · arXiv
  15. Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. (2017). Visualizing the loss landscape of neural nets. arXiv preprint arXiv:1712.09913. · arXiv
  16. Liu, C., Zhu, L., and Belkin, M. (2020). Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. arXiv preprint arXiv:2003.00307. · arXiv
  17. Mei, S., Montanari, A., and Nguyen, P.-M. (2018). A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665-E7671.
  18. Nagarajan, V. and Kolter, J. Z. (2019). Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems, 32.
  19. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. (2011). Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011. License: CC0: Public Domain.
  20. Neyshabur, B. (2020). Towards learning convolutions from scratch. In Advances in Neural Information Processing Systems.
  21. Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. (2017). Exploring generalization in deep learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 5949-5958.
  22. Neyshabur, B., Salakhutdinov, R., and Srebro, N. (2015). Path-SGD: Path-normalized optimization in deep neural networks. arXiv preprint arXiv:1506.02617. · arXiv
  23. Neyshabur, B., Sedghi, H., and Zhang, C. (2020). What is being transferred in transfer learning? In NeurIPS.
  24. Nguyen, Q., Mukkamala, M. C., and Hein, M. (2018). On the loss landscape of a class of deep neural networks with no bad local valleys. arXiv preprint arXiv:1809.10749. · arXiv
  25. Ritchie, S., Slone, A., and Ramasesh, V. (2020). Caliban: Docker-based job manager for reproducible workflows. Journal of Open Source Software, 5(53):2403.
  26. Scaman, K., Bach, F., Bubeck, S., Lee, Y., and Massoulié, L. (2019). Optimal convergence rates for convex distributed optimization in networks. Journal of Machine Learning Research, 20:1-31.
  27. Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations.
  28. Şimşek, B., Ged, F., Jacot, A., Spadaro, F., Hongler, C., Gerstner, W., and Brea, J. (2021). Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. arXiv preprint arXiv:2105.12221. · arXiv
  29. Tatro, N. J., Chen, P.-Y., Das, P., Melnyk, I., Sattigeri, P., and Lai, R. (2020). Optimizing mode connectivity via neuron alignment. arXiv preprint arXiv:2009.02439. · arXiv
  30. Wortsman, M., Horton, M., Guestrin, C., Farhadi, A., and Rastegari, M. (2021). Learning neural network subspaces. arXiv preprint arXiv:2102.10472. · arXiv
  31. Wu, X., Dobriban, E., Ren, T., Wu, S., Li, Z., Gunasekar, S., Ward, R., and Liu, Q. (2019). Implicit regularization and convergence for weight normalization. arXiv preprint arXiv:1911.07956. · arXiv
  32. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In Proceedings of the International Conference on Learning Representations.

Cited by

Sorted by citations

Metadata

Authors
Entezari, R., Sedghi, H., Saukh, O., & Neyshabur, B.
Affiliations
  • Graz University of Technology (AT)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2110.06296
arXiv
2110.06296
OpenAlex
W3200668485
Cited by
156
References
1
Citations by year
2026: 39 · 2025: 33 · 2024: 43 · 2023: 27 · 2022: 13 · 2021: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
permutation invariance, artificial neural networks, stochastic gradient descent, linear interpolation, lottery ticket hypothesis, distributed training, ensemble methods, generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review