Optimization Methods for Large-Scale Machine Learning
Open accessThe DOI matches its Crossref record; 151 of the 151 references it cites were found in a registry or an index.
Abstract
This paper provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications. Through case studies on text classification and the training of deep neural networks, we discuss how optimization problems arise in machine learning and what makes them challenging. A major theme of our study is that large-scale machine learning represents a distinctive setting in which the stochastic gradient (SG) method has traditionally played a central role while conventional gradient-based nonlinear optimization techniques typically falter. Based on this viewpoint, we present a comprehensive theory of a straightforward, yet versatile SG algorithm, discuss its practical behavior, and highlight opportunities for designing algorithms with improved performance. This leads to a discussion about the next generation of optimization methods for large-scale machine learning, including an investigation of two main streams of research on techniques that diminish noise in the stochastic directions and methods that make use of second-order derivative approximations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Self-citationRegression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Golub, G. (2012) · doi:10.56021/9781421407944 · Cited by 34,505
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
A New Approach to Linear Filtering and Prediction Problems
Kalman, R. E. (1960) · Journal of Basic Engineering · doi:10.1115/1.3662552 · Cited by 31,751
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686
(2021) · A First Course in Statistical Programming with R · doi:10.1017/9781108993456.011 · Cited by 13,724
A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
Beck, A., & Teboulle, M. (2009) · SIAM Journal on Imaging Sciences · doi:10.1137/080716542 · Cited by 11,952
Bertsekas, D. P. (1997) · Journal of the Operational Research Society · doi:10.1038/sj.jors.2600425 · Cited by 11,726
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643
De-noising by soft-thresholding
Donoho, D. L. (1995) · IEEE Transactions on Information Theory · doi:10.1109/18.382009 · Cited by 9,619
On the limited memory BFGS method for large scale optimization
Liu, D. C., & Nocedal, J. (1989) · Mathematical Programming · doi:10.1007/bf01589116 · Cited by 8,560
Self-citation(2010) · Cited by 8,261
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0026683 · Cited by 7,999
Numerical Methods for Unconstrained Optimization and Nonlinear Equations
Dennis, J. E., & Schnabel, R. B. (1996) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611971200 · Cited by 7,615
Iterative Solution of Nonlinear Equations in Several Variables
Ortega, J. M., & Rheinboldt, W. C. (2000) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9780898719468 · Cited by 6,860
Probability Inequalities for sums of Bounded Random Variables
Hoeffding, W. (1994) · Springer Series in Statistics · doi:10.1007/978-1-4612-0865-5_26 · Cited by 6,666
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795
Self-citationADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Parallel and Distributed Computation: Numerical Methods
Bertsekas, D. P., & Tsitsiklis, J. N. (1989) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 5,293
An iterative thresholding algorithm for linear inverse problems with a sparsity constraint.
Daubechies, I., Defrise, M., & De Mol, C. (2004) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.20042 · Cited by 4,958
Exact Matrix Completion via Convex Optimization
Candès, E. J., & Recht, B. (2009) · Foundations of Computational Mathematics · doi:10.1007/s10208-009-9045-5 · Cited by 4,877
Probability Inequalities for Sums of Bounded Random Variables
Hoeffding, W. (1963) · Journal of the American Statistical Association · doi:10.1080/01621459.1963.10500830 · Cited by 4,852
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451
A new approach to variable metric algorithms
Fletcher, R. (1970) · The Computer Journal · doi:10.1093/comjnl/13.3.317 · Cited by 4,068
Conditioning of quasi-Newton methods for function minimization
Shanno, D. F. (1970) · Mathematics of Computation · doi:10.1090/s0025-5718-1970-0274029-x · Cited by 3,872
(2013) · Cited by 3,740
(1989) · neural information processing systems · Cited by 3,736
Introductory Lectures on Convex Optimization: A Basic Course
Nesterov, Y. E. (2014) · Medical Entomology and Zoology · Cited by 3,557
Figueiredo, M. A. T., Nowak, R. D., & Wright, S. J. (2007) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2007.910281 · Cited by 3,524
Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization
Zhu, C., Byrd, R. H., Lu, P., & Nocedal, J. (1997) · ACM Transactions on Mathematical Software · doi:10.1145/279232.279236 · Cited by 3,462
Self-citationA family of variable-metric methods derived by variational means
Goldfarb, D. (1970) · Mathematics of Computation · doi:10.1090/s0025-5718-1970-0258249-6 · Cited by 3,325
On the Convergence Properties of the EM Algorithm
Wu, C. F. J. (1983) · The Annals of Statistics · doi:10.1214/aos/1176346060 · Cited by 3,269
On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities
Vapnik, V. N., & Chervonenkis, A. Y. (2015) · Measures of Complexity · doi:10.1007/978-3-319-21852-6_3 · Cited by 3,105
Some methods of speeding up the convergence of iteration methods
Polyak, B. T. (1964) · USSR Computational Mathematics and Mathematical Physics · doi:10.1016/0041-5553(64)90137-5 · Cited by 3,069
Conn, A. R., Gould, N. I. M., & Toint, P. L. (2000) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9780898719857 · Cited by 2,981
Updating quasi-Newton matrices with limited storage
Nocedal, J. (1980) · Mathematics of Computation · doi:10.1090/s0025-5718-1980-0572855-7 · Cited by 2,958
Self-citationThe Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations
BROYDEN, C. G. (1970) · IMA Journal of Applied Mathematics · doi:10.1093/imamat/6.1.76 · Cited by 2,874
Eckstein, J., & Bertsekas, D. P. (1992) · Mathematical Programming · doi:10.1007/bf01581204 · Cited by 2,791
Introductory Lectures on Convex Optimization
Nesterov, Y. (2004) · Applied Optimization · doi:10.1007/978-1-4419-8853-9 · Cited by 2,698
Natural Gradient Works Efficiently in Learning
Amari, S.-I. (1998) · Neural Computation · doi:10.1162/089976698300017746 · Cited by 2,681
A dual algorithm for the solution of nonlinear variational problems via finite element approximation
Gabay, D., & Mercier, B. (1976) · Computers & Mathematics with Applications · doi:10.1016/0898-1221(76)90003-1 · Cited by 2,514
RCV1: A New Benchmark Collection for Text Categorization Research
Lewis, D., Yang, Y., Rose, T. G., & Li, F. (2004) · Goldsmiths (University of London) · doi:10.5555/1005332.1005345 · Cited by 2,503
Estimation of Dependences Based on Empirical Data
Vapnik, V. (2006) · Information Science and Statistics · doi:10.1007/0-387-34239-7 · Cited by 2,238
Methods of Information Geometry
Amari, S.-I., & Nagaoka, H. (2007) · Translations of Mathematical Monographs · doi:10.1090/mmono/191 · Cited by 2,204
Adaptive Algorithms and Stochastic Approximations
Benveniste, A., Métivier, M., & Priouret, P. (1990) · Springer eBooks · doi:10.1007/978-3-642-75894-2 · Cited by 2,006
Robust Stochastic Approximation Approach to Stochastic Programming
Nemirovski, A., Juditsky, A., Lan, G., & Shapiro, A. (2009) · SIAM Journal on Optimization · doi:10.1137/070704277 · Cited by 1,945
Accelerating Stochastic Gradient Descent using Predictive Variance Reduction
Johnson, R., & Zhang, T. (2013) · Cited by 1,889
Blind signal separation: statistical principles
Cardoso, J.-F. (1998) · Proceedings of the IEEE · doi:10.1109/5.720250 · Cited by 1,885
Sparse Reconstruction by Separable Approximation
Wright, S. J., Nowak, R. D., & Figueiredo, M. A. T. (2009) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2009.2016892 · Cited by 1,854
Acceleration of Stochastic Approximation by Averaging
Polyak, B. T., & Juditsky, A. B. (1992) · SIAM Journal on Control and Optimization · doi:10.1137/0330046 · Cited by 1,827
Dembo, R. S., Eisenstat, S. C., & Steihaug, T. (1982) · SIAM Journal on Numerical Analysis · doi:10.1137/0719025 · Cited by 1,793
(2003) · Cited by 1,733
Pegasos: primal estimated sub-gradient solver for SVM
Shalev-Shwartz, S., Singer, Y., Srebro, N., & Cotter, A. (2011) · Mathematical Programming · doi:10.1007/s10107-010-0420-4 · Cited by 1,477
Inductive learning algorithms and representations for text categorization
Dumais, S., Platt, J., Heckerman, D., & Sahami, M. (1998) · Proceedings of the seventh international conference on Information and knowledge management · doi:10.1145/288627.288651 · Cited by 1,461
Stochastic variational inference
Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. W. (2013) · Journal of Machine Learning Research · doi:10.5555/2567709.2502622 · Cited by 1,333
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (1998) · neural information processing systems · Cited by 1,333
Self-citationProblem complexity and method efficiency in optimization
(1984) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(84)90101-0 · Cited by 1,329
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211
Iterative solution of nonlinear equations
Murphy, R. D. (1994) · Journal of Computers in Mathematics and Science Teaching · Cited by 1,192
Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems
Nesterov, Y. (2012) · SIAM Journal on Optimization · doi:10.1137/100802001 · Cited by 1,187
(2011) · Validated Numerics · doi:10.2307/j.ctvcm4g18.8 · Cited by 1,187
Deng, L., Hinton, G., & Kingsbury, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639344 · Cited by 1,178
The Tradeoffs of Large-Scale Learning
Bottou, L., & Bousquet, O. (2011) · Optimization for Machine Learning · doi:10.7551/mitpress/8996.003.0015 · Cited by 1,128
Self-citationGlowinski, R., & Marroco, A. (1975) · Revue française d'automatique, informatique, recherche opérationnelle. Analyse numérique · doi:10.1051/m2an/197509r200411 · Cited by 1,105
The convergence of a class of double-rank minimization algorithms
Broyden, C. G. (1970) · Medical Entomology and Zoology · Cited by 924
Minimizing finite sums with the stochastic average gradient
Schmidt, M., Le Roux, N., & Bach, F. (2017) · Mathematical Programming · doi:10.1007/s10107-016-1030-6 · Cited by 922
LeCun, Y., Bottou, L., Orr, G. B., & Müller, K. R. (1998) · Lecture Notes in Computer Science · doi:10.1007/3-540-49430-8_2 · Cited by 919
Self-citationCover, T. M. (1991) · Mathematical Finance · doi:10.1111/j.1467-9965.1991.tb00002.x · Cited by 898
Optimization for Machine Learning
(2011) · The MIT Press eBooks · doi:10.7551/mitpress/8996.001.0001 · Cited by 875
Primal-dual subgradient methods for convex problems
Nesterov, Y. (2009) · Mathematical Programming · doi:10.1007/s10107-007-0149-x · Cited by 835
SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
Defazio, A., Bach, F., & Lacoste-Julien, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.0202 · Cited by 834
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2572 · Cited by 827
The Conjugate Gradient Method and Trust Regions in Large Scale Optimization
Steihaug, T. (1983) · SIAM Journal on Numerical Analysis · doi:10.1137/0720042 · Cited by 824
Uniform Central Limit Theorems
Dudley, R. M. (1999) · Cambridge University Press eBooks · doi:10.1017/cbo9780511665622 · Cited by 785
What Is Mathematics?: An Elementary Approach To Ideas And Methods
Courant, R., & Robbins, H. (1996) · doi:10.1093/oso/9780195105193.001.0001 · Cited by 777
Feature Article: Optimization for simulation: Theory vs. Practice
Fu, M. C. (2002) · INFORMS Journal on Computing · doi:10.1287/ijoc.14.3.192.113 · Cited by 742
Optimization with Sparsity-Inducing Penalties
Bach, F., Jenatton, R., Mairal, J., & Obozinski, G. (2025) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000015 · Cited by 730
A coordinate gradient descent method for nonsmooth separable minimization
Tseng, P., & Yun, S. (2009) · Mathematical Programming · doi:10.1007/s10107-007-0170-0 · Cited by 728
Convex Optimization Algorithms
Bertsekas, D. P. (2015) · Cited by 689
Stochastic dual coordinate ascent methods for regularized lossPreprint
Shalev‐Shwartz, S., & Zhang, T. (2013) · Cited by 637
Templates for convex cone problems with applications to sparse signal recovery
Becker, S. R., Candès, E. J., & Grant, M. C. (2011) · Mathematical Programming Computation · doi:10.1007/s12532-011-0029-5 · Cited by 608
Stochastic Gradient Learning in Neural Networks
Bottou, L. (1991) · Cited by 584
Self-citationOptimal aggregation of classifiers in statistical learning
Tsybakov, A. B. (2004) · The Annals of Statistics · doi:10.1214/aos/1079120131 · Cited by 582
Fast Exact Multiplication by the Hessian
Pearlmutter, B. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.1.147 · Cited by 540
A Theory of Adaptive Pattern Classifiers
Amari, S. (1967) · IEEE Transactions on Electronic Computers · doi:10.1109/pgec.1967.264666 · Cited by 534
Learning Recurrent Neural Networks with Hessian-Free Optimization
Martens, J., & Sutskever, I. (2011) · International Conference on Machine Learning · Cited by 525
Scalable training of L1-regularized log-linear models
Andrew, G., & Gao, J. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273501 · Cited by 506
A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
Roux, N. L., Schmidt, M., & Bach, F. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1202.6258 · Cited by 481
A Stochastic Quasi-Newton Method for Online Convex Optimization
Schraudolph, N. N., Yu, J., & Günter, S. (2007) · ANU Open Research (Australian National University) · Cited by 430
On the Generalization Ability of On-Line Learning Algorithms
Cesa-Bianchi, N., Conconi, A., & Gentile, C. (2004) · IEEE Transactions on Information Theory · doi:10.1109/tit.2004.833339 · Cited by 426
A Convergence Theorem for Non Negative Almost Supermartingales and Some Applications
Robbins, H., & Siegmund, D. (1985) · Herbert Robbins Selected Papers · doi:10.1007/978-1-4612-5110-1_10 · Cited by 412
Sample size selection in optimization methods for machine learning
Byrd, R. H., Chin, G. M., Nocedal, J., & Wu, Y. (2012) · Mathematical Programming · doi:10.1007/s10107-012-0572-5 · Cited by 369
Self-citationOn a Stochastic Approximation Method
Chung, K. L. (1954) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728716 · Cited by 363
Randomized Methods for Linear Constraints: Convergence Rates and Conditioning
Leventhal, D., & Lewis, A. S. (2010) · Mathematics of Operations Research · doi:10.1287/moor.1100.0456 · Cited by 349
Large-Scale Object Classification Using Label Relation Graphs
Deng, J., Ding, N., Jia, Y., Frome, A., Murphy, K., Bengio, S., Li, Y., Neven, H., & Adam, H. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_4 · Cited by 347
Qualitatively characterizing neural network optimization problems
Goodfellow, I. J., Vinyals, O., & Saxe, A. M. (2014) · arXiv · doi:10.48550/arxiv.1412.6544 · Cited by 341
On search directions for minimization algorithms
Powell, M. J. D. (1973) · Mathematical Programming · doi:10.1007/bf01584660 · Cited by 338
A simulation-based approach to two-stage stochastic programming with recourse
Shapiro, A., & Homem‐de‐Mello, T. (1998) · Mathematical Programming · doi:10.1007/bf01580086 · Cited by 338
Improving the convergence of back-propagation learning with second-order methods
Becker, S., & LeCun, Y. (1989) · Cited by 331
LIBLINEAR: A Library for Large Linear Classification
FanRong-En, ChangKai-Wei, HsiehCho-Jui, WangXiang-Rui, & LinChih-Jen (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1442794 · Cited by 328
SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent
Bordes, A., Bottou, L., & Gallinari, P. (2009) · HAL · Cited by 324
Self-citationNewton's Method for Large Bound-Constrained Optimization Problems
Lin, C., & Morè, J. J. (1999) · SIAM Journal on Optimization · doi:10.1137/s1052623498345075 · Cited by 322
Hybrid deterministic-stochastic methods for data fitting
Friedlander, M. P., & Schmidt, M. (2011) · Cited by 314
A Convergent Incremental Gradient Method with a Constant Step Size
Blatt, D., Hero, A. O., & Gauchman, H. (2007) · SIAM Journal on Optimization · doi:10.1137/040615961 · Cited by 306
Probability and Finance: It's Only a Game!
Shafer, G. R., & Vovk, V. (2001) · Cited by 285
New insights and perspectives on the natural gradient method
Martens, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.1193 · Cited by 283
Efficient Estimations from a Slowly Convergent Robbins-Monro Process
Ruppert, D. (1988) · eCommons (Cornell University) · Cited by 277
Hybrid Deterministic-Stochastic Methods for Data Fitting
Friedlander, M. P., & Schmidt, M. (2012) · SIAM Journal on Scientific Computing · doi:10.1137/110830629 · Cited by 254
Sparse Inverse Covariance Matrix Estimation Using Quadratic Approximation
Hsieh, C., Sustik, M. A., Dhillon, I. S., & Ravikumar, P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1306.3212 · Cited by 251
On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning
Byrd, R. H., Chin, G. M., Neveitt, W., & Nocedal, J. (2011) · SIAM Journal on Optimization · doi:10.1137/10079923x · Cited by 242
Self-citationSome applications of concentration inequalities to statistics
Massart, P. (2000) · Annales de la Faculté des sciences de Toulouse : Mathématiques · doi:10.5802/afst.961 · Cited by 240
Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization
Schmidt, M., Roux, N. L., & Bach, F. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1109.2415 · Cited by 237
Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization
Agarwal, A., Bartlett, P. L., Ravikumar, P., & Wainwright, M. J. (2012) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2182178 · Cited by 225
BIG & QUIC: Sparse Inverse Covariance Estimation for a Million Variables
Hsieh, C., Sustik, M. A., Dhillon, I. S., Ravikumar, P., & Poldrack, R. A. (2013) · neural information processing systems · Cited by 219
Newton Sketch: A Near Linear-Time Optimization Algorithm with Linear-Quadratic Convergence
Pilanci, M., & Wainwright, M. J. (2017) · SIAM Journal on Optimization · doi:10.1137/15m1021106 · Cited by 185
(2015) · arXiv (Cornell University) · Cited by 182
A Universal Catalyst for First-Order Optimization
Lin, H., Mairal, J., & Harchaoui, Z. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.02186 · Cited by 165
Adaptive natural gradient learning algorithms for various stochastic models
Park, H., Amari, S.-I., & Fukumizu, K. (2000) · Neural Networks · doi:10.1016/s0893-6080(00)00051-4 · Cited by 164
RES: Regularized Stochastic BFGS Algorithm
Mokhtari, A., & Ribeiro, A. (2014) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2014.2357775 · Cited by 158
Incremental Least Squares Methods and the Extended Kalman Filter
Bertsekas, D. P. (1996) · SIAM Journal on Optimization · doi:10.1137/s1052623494268522 · Cited by 150
On‐line learning for very large data sets
Bottou, L., & LeCun, Y. (2005) · Applied Stochastic Models in Business and Industry · doi:10.1002/asmb.538 · Cited by 149
Self-citationTowards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient DescentPreprint
Xu, W. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1107.2490 · Cited by 143
Gladyshev, E. G. (1965) · Theory of Probability and Its Applications · doi:10.1137/1110031 · Cited by 131
Bousquet, O. (2002) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 122
On the Convergence Rate of Incremental Aggregated Gradient Algorithms
Gürbüzbalaban, M., Ozdaglar, A., & Parrilo, P. A. (2017) · SIAM Journal on Optimization · doi:10.1137/15m1049695 · Cited by 121
Graphical model structure learning using L₁-regularization
Schmidt, M. (2010) · cIRcle (University of British Columbia) · doi:10.14288/1.0051935 · Cited by 114
Second-Order Stochastic Optimization for Machine Learning in Linear TimePreprint
Agarwal, N., Bullins, B., & Hazan, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03943 · Cited by 104
The importance of convexity in learning with squared loss
Lee, W. S., Bartlett, P. L., & Williamson, R. C. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.705577 · Cited by 103
Osborne, M. R. (1992) · International Statistical Review · doi:10.2307/1403504 · Cited by 102
A Statistical Study of On-line Learning
Murata, N. (1999) · Cambridge University Press eBooks · doi:10.1017/cbo9780511569920.005 · Cited by 93
An inexact successive quadratic approximation method for L-1 regularized optimization
Byrd, R. H., Nocedal, J. M., & Öztoprak, F. (2015) · Mathematical Programming · doi:10.1007/s10107-015-0941-y · Cited by 88
Self-citationSub-Sampled Newton Methods II: Local Convergence RatesPreprint
Roosta, F., & Mahoney, M. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.04738 · Cited by 78
Practical inexact proximal quasi-Newton method with global complexity analysis
Scheinberg, K., & Tang, X. (2016) · Mathematical Programming · doi:10.1007/s10107-016-0997-3 · Cited by 78
Information-Based Complexity, Feedback and Dynamics in Convex Programming
Raginsky, M., & Rakhlin, A. (2011) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2154375 · Cited by 69
A family of second-order methods for convex $$\ell _1$$-regularized optimization
Byrd, R. H., Chin, G. M., Nocedal, J. M., & Öztoprak, F. (2015) · Mathematical Programming · doi:10.1007/s10107-015-0965-3 · Cited by 54
Self-citationOn Stochastic Proximal Gradient Algorithms
Atchadé, Y. F., Fort, G., & Moulines, É. (2017) · HAL · Cited by 36
Batch and online learning algorithms for nonconvex neyman-pearson classification
Gasso, G., Pappaioannou, A., Spivak, M., & Bottou, L. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961200 · Cited by 26
Self-citationOn adaptive sampling rules for stochastic recursions
Hashemi, F. S., Ghosh, S., & Pasupathy, R. (2014) · Winter Simulation Conference · doi:10.1109/wsc.2014.7020221 · Cited by 23
Erratum: SGD-QN is less careful than expected
Bordes, A., Université, L., Curie, P. M., Jussieu, P., Bottou, L., Gallinari, P., Chang, J., Smith, S. A., Sonnenburg, S., Franc, V., et al. (2010) · Cited by 14
Self-citationFast Curvature Matrix-Vector Products
Schraudolph, N. N. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44668-0_4 · Cited by 10
Cited by
Generalizing from a Few Examples: A Survey on Few-shot Learning
Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2021) · ACM Computing Surveys · doi:10.1145/3386252 · Cited by 3,165
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
Federated Optimization: Distributed Machine Learning for On-Device Intelligence
Konečný, J., McMahan, H. B., Ramage, D., & Richtárik, P. (2016) · arXiv · doi:10.48550/arxiv.1610.02527 · Cited by 1,563
Super-convergence: very fast training of neural networks using large learning rates
Smith, L. N., & Topin, N. (2019) · Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications · doi:10.1117/12.2520589 · Cited by 1,335
Blockchain for AI: review and open research challenges
Salah, K., Rehman, M. H. U., Nizamuddin, N., & Al-Fuqaha, A. (2019) · IEEE Access · doi:10.1109/access.2018.2890507 · Cited by 1,001
Metadata
- Authors
- Bottou, L., Curtis, F. E., & Nocedal, J.
- Venue
- SIAM Review
- Published
- 2018-01-01
- Volume
- 60
- Issue
- 2
- Pages
- 223-311
- ISSN
- 0036-1445 (ISSN-L)
- 1095-7200
- OpenAlex
- W2436219157
- Cited by
- 3,128
- References
- 151
- FWCI
- 199.30
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.