The DOI matches its Crossref record.
Implicit Regularization in Matrix Factorization
Open accessAbstract
We study implicit regularization when optimizing an underdetermined quadratic objective over a matrix X with gradient descent on a factorization of X. We conjecture and provide empirical and theoretical evidence that with small enough step sizes and initialization close enough to the origin, gradient descent on a full dimensional factorization converges to the minimum nuclear norm solution.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is significant empirical and some theoretical evidence that a similar minimum norm inductive bias is present for more general neural networks.
“While the behavior of SGD for more general neural networks is not fully understood, there is significant empirical and some theoretical evidence (e.g., [20]) that a similar minimum norm inductive bias is present.”
From Reconciling modern machine-learning practice and the classical bias–variance trade-off (2019) · cites this paper as [20]
The Introduction reports empirical and theoretical evidence for a minimum-nuclear-norm bias in matrix factorization, which it describes as a two-layer network with linear transfer. It also discusses implicit regularization in deep models, but does not establish a similar minimum-norm bias for more general neural networks or specifically for SGD. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe improvement of test risk beyond the interpolation threshold is compatible with the conjectured 'small norm' inductive biases of common training algorithms for neural networks.
“That the test risk improves beyond the interpolation threshold is compatible with the conjectured 'small norm' inductive biases of the common training algorithms for neural networks [20, 25].”
From Reconciling modern machine-learning practice and the classical bias–variance trade-off (2019) · cites this paper as [20]
The Introduction says that gradient-based training can favor well-generalizing solutions in underdetermined models and reports a conjectured minimum-nuclear-norm bias for matrix factorization. This supports the proposed compatibility, but the paper does not establish that test risk improves beyond an interpolation threshold or establish a small-norm bias for neural networks generally. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 17
- raghunandan, 2012, Efficient Algorithms for Collaborative Filtering
- nitish shirish, 2017, On large-batch training for deep learning: Generalization gap and sharp minima, International Conference on Learning Representations
- lee, 2016, and Benjamin Recht. Gradient descent only converges to minimizers, 29th Annual Conference on Learning Theory
- behnam, 2015, In search of the real inductive bias: On the role of implicit regularization in deep learning, International Conference on Learning Representations
- benjamin, 2010, Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization, SIAM Review, 52, 471, 10.1137/070697835 · DOI
- nathan, 2005, Rank, trace-norm and max-norm, Conference on Computational Learning Theory, 545
- nathan, 2005, Generalization error bounds for collaborative prediction with low-rank matrices, Advances in neural information processing systems, 1321
- chiyuan, 2017, Understanding deep learning requires rethinking generalization, International Conference on Learning Representations
- samuel, 2003, A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization, Mathematical Programming, 329
- srinadh, 2016, Global optimality of local search for low rank matrix recovery, Advances in neural information processing systems
- rina, 2011, Concentration-based guarantees for low-rank matrix reconstruction, COLT, 315
- 10.1007/s10208-009-9045-5 · DOI
- eric, 2001, SciPy Open source scientific tools for Python
- rong, 2016, completion has no spurious local minimum, Advances in neural information processing systems, 2973
- andreas, 2007, Multi-task feature learning, Advances in neural information processing systems, 19
- yonatan, 2007, Uncovering shared structures in multiclass classification, Proceedings of the 24th International Conference on Machine Learning, 17
- michel, 2010, and Rodolphe Sepulchre. Low-rank optimization on the cone of positive semidefinite matrices, SIAM Journal on Optimization, 20, 2327, 10.1137/080731359 · DOI
Cited by
Sorted by citations
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595
Koçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168
Chang, X., Li, Y., Oymak, S., & Thrampoulidis, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16859 · Cited by 24
How Memory in Optimization Algorithms Implicitly Modifies the Loss
Cattaneo, M., & Shigida, B. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-5215
Metadata
- Authors
- Gunasekar, S., Woodworth, B., Bhojanapalli, S., Neyshabur, B., & Srebro, N.
- Affiliations
- Kyushu University (JP)
- Toyota Technological Institute at Chicago (US)
- Venue
- 2018 Information Theory and Applications Workshop (ITA)
- Publisher
- IEEE
- Published
- 2018-02
- Volume
- 30
- Pages
- 1-10
- OpenAlex
- W2618069152
- Cited by
- 170
- References
- 0
- Citations by year
- 2026: 16 · 2025: 19 · 2024: 32 · 2023: 19 · 2022: 15 · 2021: 28 · 2020: 20 · 2019: 13 · 2018: 5
- FWCI
- 14.62
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Information Theory and Applications Workshop (ITA) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- neural information processing systems
- Topics
- Physical Sciences › Computer Science › Signal Processing › Sparse and Compressive Sensing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Matrix Theory and Algorithms
- Keywords
- implicit regularization, matrix factorization, gradient descent, nuclear norm minimization, low-rank approximation, step size, initialization, convergence analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
