Train faster, generalize better: Stability of stochastic gradient descent
Open accessThe DOI matches its OpenAlex record; 30 of the 30 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
(2014) · Cited by 33,419
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Parikh, N., & Boyd, S. (2014) · Foundations and Trends® in Optimization · doi:10.1561/2400000003 · Cited by 4,102
Monotone Operators and the Proximal Point Algorithm
Rockafellar, R. T. (1976) · SIAM Journal on Control and Optimization · doi:10.1137/0314056 · Cited by 3,559
Signal Recovery by Proximal Forward-Backward Splitting
Combettes, P. L., & Wajs, V. R. (2005) · Multiscale Modeling & Simulation · doi:10.1137/050626090 · Cited by 2,678
Recurrent Neural Network Regularization
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 2,123
Adaptive Algorithms and Stochastic Approximations
Benveniste, A., Métivier, M., & Priouret, P. (1990) · Springer eBooks · doi:10.1007/978-3-642-75894-2 · Cited by 2,006
Robust Stochastic Approximation Approach to Stochastic Programming
Nemirovski, A., Juditsky, A., Lan, G., & Shapiro, A. (2009) · SIAM Journal on Optimization · doi:10.1137/070704277 · Cited by 1,945
Stochastic Approximation and Recursive Algorithms and Applications
(2003) · Stochastic Modelling and Applied Probability · doi:10.1007/b97441 · Cited by 1,838
(1989) · Choice Reviews Online · doi:10.5860/choice.26-5109 · Cited by 1,755
(1992) · Cited by 1,405
Almost Optimal Exploration in Multi-Armed Bandits
Karnin, Z. S., Koren, T., & Somekh, O. (2013) · International Conference on Machine Learning · Cited by 637
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244302760200704 · Cited by 636
Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
Rakhlin, A., Shamir, O., & Sridharan, K. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1109.5647 · Cited by 492
An optimal method for stochastic composite optimization
Lan, G. (2012) · Mathematical Programming · doi:10.1007/s10107-010-0434-y · Cited by 427 · cited 1×
“e update is a linear combination of the current iterate and the previous direction. For convex problems, momentum is known to decrease the number of iterations required by stochastic gradient descent [22]. For general nonlinear problems, is believed to decrease the number of iterations required to achieve low-training…”
BackgroundFirst-order methods of smooth convex optimization with inexact oracle
Devolder, O., Glineur, F., & Nesterov, Y. (2014) · Mathematical Programming · doi:10.1007/s10107-013-0677-5 · Cited by 422
Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation
Kearns, M., & Ron, D. (1999) · Neural Computation · doi:10.1162/089976699300016304 · Cited by 407
Beyond the regret minimization barrier
Hazan, E., & Kale, S. (2014) · Journal of Machine Learning Research · doi:10.5555/2627435.2670328 · Cited by 334
Learnability, Stability and Uniform Convergence
Shalev‐Shwartz, S., Shamir, O., Srebro, N., & Sridharan, K. (2010) · doi:10.5555/1756006.1953019 · Cited by 328
Logarithmic Regret Algorithms for Online Convex Optimization
Hazan, E., Kalai, A. T., Kale, S., & Agarwal, A. (2006) · Lecture notes in computer science · doi:10.1007/11776420_37 · Cited by 242
Non-stochastic Best Arm Identification and Hyperparameter Optimization
Jamieson, K., & Talwalkar, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.07943 · Cited by 232
Distribution-free performance bounds for potential function rules
Devroye, L., & Wagner, T. (1979) · IEEE Transactions on Information Theory · doi:10.1109/tit.1979.1056087 · Cited by 226
Mukherjee, S., Niyogi, P., Poggio, T., & Rifkin, R. (2006) · Advances in Computational Mathematics · doi:10.1007/s10444-004-7634-z · Cited by 198
Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor MethodsPreprint
Janzamin, M., Sedghi, H., & Anandkumar, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.08473 · Cited by 176
In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
Neyshabur, B., Tomioka, R., & Srebro, N. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6614 · Cited by 125
Stability of Randomized Learning Algorithms
Elisseeff, A., Evgeniou, T., & Pontil, M. (2005) · Cited by 116
Competing with the Empirical Risk Minimizer in a Single PassPreprint
Frostig, R., Ge, R., Kakade, S. M., & Sidford, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6606 · Cited by 78
On the Generalization Properties of Differential PrivacyPreprint
Kobbi Nissim, B., & Stemmer, U. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.05800 · Cited by 37
Cited by
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
“However, after extensive model exploration we identified a family of archi-2 tectures that resulted in stable training across a range of datasets and allowed for training higher resolution and deeper generative models.”
BackgroundMethodMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754
“Similar metrics are used in the literature to measure how overfitted a model is [18].”
BackgroundMethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
“A number of arguments support the idea that SGD-based learning imparts a regularization effect, especially with a small batch size (Wilson & Martinez, 2003) or a small number of epochs (Hardt et al., 2015).”
BackgroundReinforcement learning, fast and slow
Botvinick, M., Ritter, S., Wang, J. X., Kurth-Nelson, Z., Blundell, C., & Hassabis, D. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.02.006 · Cited by 667
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
John Xu, Z.-Q., Zhang, Y., Luo, T., Xiao, Y., & Ma, Z. (2020) · Communications in Computational Physics · doi:10.4208/cicp.oa-2020-0085 · Cited by 522
Metadata
- Authors
- Hardt, M., Recht, B., & Singer, Y.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2015-09-03
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1509.01240
- OpenAlex
- W1811750039
- Cited by
- 323
- References
- 30
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.