The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Deep Learning Is Singular, and That’s Good
Open accessAbstract
In singular models, the optimal set of parameters forms an analytic set with singularities, and a classical statistical inference cannot be applied to such models. This is significant for deep learning as neural networks are singular, and thus, "dividing" by the determinant of the Hessian or employing the Laplace approximation is not appropriate. Despite its potential for addressing fundamental issues in deep learning, a singular learning theory appears to have made little inroads into the developing canon of a deep learning theory. Via a mix of theory and experiment, we present an invitation to the singular learning theory as a vehicle for understanding deep learning and suggest an important future work to make the singular learning theory directly applicable to how deep learning is performed in practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
It has been observed that new model capabilities can emerge as a function of scale.
“For instance, it has been observed that new model capabilities can emerge as a function of scale (Wei et al. (2022), though also see Schaeffer et al. (2023)).”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Wei et al. (2022)
Section 7 (Future directions; page unavailable) mentions power laws in neural language model training, but does not say that new capabilities emerge as models scale. Searches for emergent abilities and capabilities as a function of scale found no passage addressing that claim. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“The existence of power laws in neural language model training [43], [44] is one of the most remarkable experimental results in deep learning.”
BackgroundUnderstanding deep learning (still) requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021) · Communications of the ACM · doi:10.1145/3446776 · Cited by 2,400
Keeping the neural networks simple by minimizing the description length of the weights
Hinton, G. E., & van Camp, D. (1993) · Proceedings of the sixth annual conference on Computational learning theory - COLT '93 · doi:10.1145/168304.168306 · Cited by 994
Flagged first, then by citations
Research works printed but not linked: 45
- 10.1002/scj.10353 · DOI
- 10.1109/FOCI.2007.371500 · DOI
- 10.1017/CBO9780511800474 · DOI
- Smith, 2017, A Bayesian perspective on generalization and stochastic gradient descent, arXiv:1710.06451 · arXiv
- 10.1080/00268976.2018.1483535 · DOI
- 10.1007/978-3-540-28650-9_8 · DOI
- Du, Gradient descent provably optimizes over-parameterized neural networks
- Allen-Zhu, A convergence theory for deep learning via over-parameterization, 242
- Neyshabur, Norm-based capacity control in neural networks, 1376
- Bartlett, Spectrally-normalized margin bounds for neural networks, 6240
- Neyshabur, Towards understanding the role of over-parametrization in generalization of neural networks
- Arora, Stronger generalization bounds for deep nets via a compression approach, 254
- Brutzkus, SGD learns over-parameterized networks that provably generalize on linearly separable data, 1
- Li, Learning overparameterized neural networks via stochastic gradient descent on structured data, 8157
- Allen-Zhu, Learning and generalization in overparameterized neural networks, going beyond two layers, 6155
- Daniely, SGD learns the conjugate kernel class of the network, 2422
- Arora, Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks, 322
- Yehudai, On the power and limitations of random features for understanding neural networks, 6594
- Cao, Generalization bounds of stochastic gradient descent for wide and deep neural networks, 10835
- 10.1145/3406325.3465355 · DOI
- 10.1201/9781315373010 · DOI
- Watanabe, 2013, A widely applicable Bayesian information criterion, J. Mach. Learn. Res., 14, 867
- 10.1162/neco.1997.9.1.1 · DOI
- 10.1088/1742-5468/ab39d9 · DOI
- Jastrzebski, 2017, Three factors influencing minima in SGD, arXiv:1711.04623 · arXiv
- 10.1162/neco.1997.9.2.349 · DOI
- Poggio, 2018, Theory of deep learning III: Explaining the non-overfitting puzzle, arXiv:1801.00173 · arXiv
- Neyshabur, Exploring generalization in deep learning, 5947
- Thomas, 2019, On the interplay between noise and curvature and its effect on optimization and generalization, arXiv:1906.07774 · arXiv
- Maddox, 2020, Rethinking parameter counting in deep models: Effective dimensionality revisited, arXiv:2003.02139 · arXiv
- 10.1016/j.neunet.2005.03.014 · DOI
- Aoyagi, Resolution of singularities and the generalization error with Bayesian estimation for layered neural network, 2112
- 10.1162/neco.2007.19.4.1112 · DOI
- Kristiadi, 2020, Being Bayesian, even just a bit, fixes overconfidence in ReLU networks, arXiv:2002.10118 · arXiv
- 10.5555/2627435.2638586 · DOI
- Hendrycks, 2016, Gaussian error linear units (GELUs), arXiv:1606.08415 · arXiv
- Ramachandran, 2017, Searching for activation functions, arXiv:1710.05941 · arXiv
- ŞimŠekli, Fractional Langevin Monte Carlo: Exploring Levy driven stochastic differential equations for Markov Chain Monte Carlo, 70, 3200
- Mandt, 2017, Stochastic gradient descent as approximate Bayesian inference, J. Mach. Learn. Res., 18, 4873
- Smith, 2018, Stochastic natural gradient descent draws posterior samples in function space, arXiv:1806.09597 · arXiv
- Hestness, 2017, Deep learning scaling is predictable, empirically, arXiv:1712.00409 · arXiv
- Sagun, 2016, Singularity of the Hessian in deep learning, arXiv:1611.07476 · arXiv
- Pennington, The spectrum of the Fisher information matrix of a single-hidden-layer neural network, 5410
- Phuong, Functional vs. parametric equivalence of ReLU networks
- Boothby, 1986, An Introduction to Differentiable Manifolds and Riemannian Geometry
Cited by
Sorted by citations
Open Problems in Mechanistic Interpretability
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.16496 · Cited by 291
Self-citation
Metadata
- Authors
- Wei, S., Murfet, D., Gong, M., Li, H., Gell-Redman, J., & Quella, T.
- Affiliations
- School of Mathematics and Statistics, The University of Melbourne, Melbourne, VIC, Australia
- The University of Melbourne (AU)
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2023-12
- Volume
- 34
- Issue
- 12
- Pages
- 10473-10486
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W3093826057
- Cited by
- 26
- References
- 3
- Citations by year
- 2026: 10 · 2025: 4 · 2024: 4 · 2023: 3
- FWCI
- 0.74
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion)
- arXiv (Cornell University)
- Funders
- ARC (DE200101253, DE210101624)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- singular learning theory, singular models, artificial neural networks, Hessian determinant, Laplace approximation, statistical inference, deep learning theory, analytic sets, singularities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
