The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
Recent Advances in Stochastic Gradient Descent in Deep Learning
Open accessAbstract
In the age of artificial intelligence, the best approach to handling huge amounts of data is a tremendously motivating and hard problem. Among machine learning models, stochastic gradient descent (SGD) is not only simple but also very effective. This study provides a detailed analysis of contemporary state-of-the-art deep learning applications, such as natural language processing (NLP), visual data processing, and voice and audio processing. Following that, this study introduces several versions of SGD and its variant, which are already in the PyTorch optimizer, including SGD, Adagrad, adadelta, RMSprop, Adam, AdamW, and so on. Finally, we propose theoretical conditions under which these methods are applicable and discover that there is still a gap between theoretical conditions under which the algorithms converge and practical applications, and how to bridge this gap is a question for the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643
Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
Fan, J., & Li, R. (2001) · Journal of the American Statistical Association · doi:10.1198/016214501753382273 · Cited by 8,900
Ideal spatial adaptation by wavelet shrinkage
Donoho, D. L., & Johnstone, I. M. (1994) · Biometrika · doi:10.1093/biomet/81.3.425 · Cited by 8,760
On the limited memory BFGS method for large scale optimization
Liu, D. C., & Nocedal, J. (1989) · Mathematical Programming · doi:10.1007/bf01589116 · Cited by 8,560
(2010) · Cited by 8,261
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Conditioning of quasi-Newton methods for function minimization
Shanno, D. F. (1970) · Mathematics of Computation · doi:10.1090/s0025-5718-1970-0274029-x · Cited by 3,872
Nearly unbiased variable selection under minimax concave penalty
Zhang, C.-H. (2010) · The Annals of Statistics · doi:10.1214/09-aos729 · Cited by 3,835
(2013) · Cited by 3,740
A family of variable-metric methods derived by variational means
Goldfarb, D. (1970) · Mathematics of Computation · doi:10.1090/s0025-5718-1970-0258249-6 · Cited by 3,325
Optimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,128
Updating quasi-Newton matrices with limited storage
Nocedal, J. (1980) · Mathematics of Computation · doi:10.1090/s0025-5718-1980-0572855-7 · Cited by 2,958
Robust Stochastic Approximation Approach to Stochastic Programming
Nemirovski, A., Juditsky, A., Lan, G., & Shapiro, A. (2009) · SIAM Journal on Optimization · doi:10.1137/070704277 · Cited by 1,945
Accelerating Stochastic Gradient Descent using Predictive Variance Reduction
Johnson, R., & Zhang, T. (2013) · Cited by 1,889
Image Super-Resolution Via Iterative Refinement
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., & Norouzi, M. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3204461 · Cited by 1,829
On the Convergence of Adam and Beyond
Reddi, S. J., Kale, S., & Kumar, S. (2019) · arXiv · doi:10.48550/arxiv.1904.09237 · Cited by 1,771
Palette: Image-to-Image Diffusion Models
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., & Norouzi, M. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530757 · Cited by 1,717
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
The Tradeoffs of Large-Scale Learning
Bottou, L., & Bousquet, O. (2011) · Optimization for Machine Learning · doi:10.7551/mitpress/8996.003.0015 · Cited by 1,128
Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis
Litjens, G., Sánchez, C. I., Timofeeva, N., Hermsen, M., Nagtegaal, I., Kovacs, I., Hulsbergen - van de Kaa, C., Bult, P., van Ginneken, B., & van der Laak, J. (2016) · Scientific Reports · doi:10.1038/srep26286 · Cited by 1,069
$L_{1/2}$ Regularization: A Thresholding Representation Theory and a Fast Solver
Xu, Z., Chang, X., Xu, F., & Zhang, H. (2012) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2197412 · Cited by 1,062
Self-citationABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs
Yin, W., Schütze, H., Xiang, B., & Zhou, B. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00097 · Cited by 924
A method for unconstrained convex minimization problem with the rate of convergence o(1/k^2)
Nesterov, Y. E. (1983) · Medical Entomology and Zoology · Cited by 865
Speech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-57 · Cited by 794
A Survey of Optimization Methods From a Machine Learning Perspective
Sun, S., Cao, Z., Zhu, H., & Zhao, J. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2950779 · Cited by 789
Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection
Socher, R., Huang, E., Pennin, J., Manning, C. D., & Ng, A. Y. (2011) · neural information processing systems · Cited by 782
Efficient mini-batch training for stochastic optimization
Li, M., Zhang, T., Chen, Y., & Smola, A. J. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623612 · Cited by 742
DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement
Hu, Y., Liu, Y., Lv, S., Xing, M., Zhang, S., Fu, Y., Wu, J., Zhang, B., & Xie, L. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2537 · Cited by 737
Deep learning via Hessian-free optimization
Martens, J. (2010) · Cited by 722
A Proximal Stochastic Gradient Method with Progressive Variance Reduction
Xiao, L., & Zhang, T. (2014) · SIAM Journal on Optimization · doi:10.1137/140961791 · Cited by 592
Learning Recurrent Neural Networks with Hessian-Free Optimization
Martens, J., & Sutskever, I. (2011) · Cited by 525
A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
Roux, N. L., Schmidt, M., & Bach, F. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1202.6258 · Cited by 481
Advances in optimizing recurrent networks
Bengio, Y., Boulanger-Lewandowski, N., & Pascanu, R. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639349 · Cited by 469
A Stochastic Quasi-Newton Method for Large-Scale Optimization
Byrd, R. H., Hansen, S. L., Nocedal, J., & Singer, Y. (2016) · SIAM Journal on Optimization · doi:10.1137/140954362 · Cited by 459
iPiano: Inertial Proximal Algorithm for Nonconvex Optimization
Ochs, P., Chen, Y., Brox, T., & Pock, T. (2014) · SIAM Journal on Imaging Sciences · doi:10.1137/130942954 · Cited by 453
Analysis of Multi-stage Convex Relaxation for Sparse Regularization
Zhang, T. (2010) · Cited by 444
A Sufficient Condition for Convergences of Adam and RMSProp
Zou, F., Shen, L., Jie, Z., Zhang, W., & Liu, W. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01138 · Cited by 419
Maximum Entropy and the Nearly Black Object
Donoho, D. L., Johnstone, I. M., Hoch, J. C., & Stern, A. S. (1992) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1992.tb01864.x · Cited by 393
Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization
Ghadimi, S., Lan, G., & Zhang, H. (2016) · Mathematical Programming · doi:10.1007/s10107-014-0846-1 · Cited by 376
Self-citationA Stochastic Quasi-Newton Method for Online Convex Optimization
Schraudolph, N. N., Yu, J., & Günter, S. (2007) · ANU Open Research (Australian National University) · Cited by 324
Katyusha: the first direct acceleration of stochastic gradient methods
Allen-Zhu, Z. (2017) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/3055399.3055448 · Cited by 309
WAVESHRINK WITH FIRM SHRINKAGE
Gao, H., & Bruce, A. G. (1997) · Cited by 277
Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems
Meinhardt, T., Moeller, M., Hazirbas, C., & Cremers, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.198 · Cited by 257
Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting
Konecny, J., Liu, J., Richtarik, P., & Takac, M. (2016) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2015.2505682 · Cited by 254
Neumann, M., & Vu, N. T. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-917 · Cited by 243
On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning
Byrd, R. H., Chin, G. M., Neveitt, W., & Nocedal, J. (2011) · SIAM Journal on Optimization · doi:10.1137/10079923x · Cited by 242
Global convergence of the Heavy-ball method for convex optimization
Ghadimi, E., Feyzmahdavian, H. R., & Johansson, M. (2015) · 2015 European Control Conference (ECC) · doi:10.1109/ecc.2015.7330562 · Cited by 240
Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization
Shalev‐Shwartz, S., & Zhang, T. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1209.1873 · Cited by 240
Stochastic Variance Reduction for Nonconvex Optimization
Reddi, S. J., Hefny, A., Sra, S., Póczos, B., & Smola, A. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06160 · Cited by 215
A Linearly-Convergent Stochastic L-BFGS AlgorithmPreprint
Moritz, P., Nishihara, R., & Jordan, M. I. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.02087 · Cited by 188
An Asynchronous Parallel Stochastic Coordinate Descent AlgorithmPreprint
Liu, J., Wright, S. J., Ré, C., Bittorf, V., & Sridhar, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1311.1873 · Cited by 183
Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization
Wang, X., Ma, S., Goldfarb, D., & Liu, W. (2017) · SIAM Journal on Optimization · doi:10.1137/15m1053141 · Cited by 169
RES: Regularized Stochastic BFGS Algorithm
Mokhtari, A., & Ribeiro, A. (2014) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2014.2357775 · Cited by 158
Optimization with First-Order Surrogate FunctionsPreprint
Mairal, J. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1305.3120 · Cited by 134
Convergence of Stochastic Proximal Gradient Algorithm
Rosasco, L., Villa, S., & Vũ, B. C. (2019) · Applied Mathematics & Optimization · doi:10.1007/s00245-019-09617-7 · Cited by 131
A Unified Analysis of Stochastic Momentum Methods for Deep Learning
Yan, Y., Yang, T., Li, Z., Lin, Q., & Yang, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/410 · Cited by 117
A Generic Proximal Algorithm for Convex Optimization—Application to Total Variation Minimization
Condat, L. (2014) · IEEE Signal Processing Letters · doi:10.1109/lsp.2014.2322123 · Cited by 111
Proximal stochastic methods for nonsmooth nonconvex finite-sum optimization
Reddi, S. J., Sra, S., Póczos, B., & Smola, A. J. (2016) · neural information processing systems · Cited by 111
On the Insufficiency of Existing Momentum Schemes for Stochastic Optimization
Kidambi, R., Netrapalli, P., Jain, P., & Kakade, S. M. (2018) · Information Theory and Applications Workshop (ITA) · doi:10.1109/ita.2018.8503173 · Cited by 100
VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning
Shang, F., Zhou, K., Liu, H., Cheng, J. S.-C., Tsang, I. W.-H., Zhang, L., Tao, D., & Jiao, L. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2878765 · Cited by 81
A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex OptimizationPreprint
Li, Z., & Li, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04477 · Cited by 78
Information-theoretic lower bounds on the oracle complexity of convex optimizationPreprint
Agarwal, A., Wainwright, M. J., Bartlett, P. L., & Ravikumar, P. (2009) · arXiv (Cornell University) · Cited by 77
A Linearly Convergent Proximal Gradient Algorithm for Decentralized OptimizationPreprint
Alghunaim, S. A., Yuan, K., & Sayed, A. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07996 · Cited by 56
Combining cross-modal knowledge transfer and semi-supervised learning for speech emotion recognition
Zhang, S., Chen, M., Chen, J., Li, Y.-F., Wu, Y., Li, M., & Zhu, C. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107340 · Cited by 55
Skip-Convolutions for Efficient Video Processing
Habibian, A., Abati, D., Cohen, T., & Bejnordi, B. E. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00272 · Cited by 54
DTMT: A Novel Deep Transition Architecture for Neural Machine Translation
Meng, F., & Zhang, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301224 · Cited by 48
A Stochastic Quasi-Newton Method for Large-Scale Nonconvex Optimization With Applications
Chen, H., Wu, H. C., Chan, S. C., & Lam, W.-H. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2957843 · Cited by 45
On the Convergence of Stochastic Gradient Descent with Adaptive StepsizesPreprint
Li, X., & Orabona, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08114 · Cited by 32
Convergence Rates of the Heavy Ball Method for Quasi-strongly Convex Optimization
Aujol, J., Dossal, C., & Rondepierre, A. (2022) · SIAM Journal on Optimization · doi:10.1137/21m1403990 · Cited by 30
Towards stability and optimality in stochastic gradient descentPreprint
Toulis, P., Tran, D., & Airoldi, E. M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02417 · Cited by 22
Linear Convergence of Adaptive Stochastic Gradient DescentPreprint
Xie, Y., Wu, X., & Ward, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.10525 · Cited by 17
Hoffer, E., Hubara, I., & Soudry, D. (2017) · arXiv · doi:10.48550/arxiv.1705.08741
Metadata
- Authors
- Tian, Y., Zhang, Y., & Zhang, H.
- Venue
- Mathematics
- Publisher
- MDPI AG
- Published
- 2023-01-29
- Volume
- 11
- Issue
- 3
- Pages
- 682
- ISSN
- 2227-7390 (ISSN-L)
- OpenAlex
- W4318484896
- Cited by
- 277
- References
- 79
- FWCI
- 24.33
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.