The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
A Proximal Stochastic Gradient Method with Progressive Variance Reduction
Open accessAbstract
We consider the problem of minimizing the sum of two convex functions: one is the average of a large number of smooth component functions, and the other is a general convex function that admits a simple proximal mapping. We assume the whole objective function is strongly convex. Such problems often arise in machine learning, known as regularized empirical risk minimization. We propose and analyze a new proximal stochastic gradient method, which uses a multistage scheme to progressively reduce the variance of the stochastic gradient. While each iteration of this algorithm has similar cost as the classical stochastic gradient method (or incremental gradient method), we show that the expected objective value converges to the optimum at a geometric rate. The overall complexity of this method is much lower than both the proximal full gradient method and the standard proximal stochastic gradient method.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
Beck, A., & Teboulle, M. (2009) · SIAM Journal on Imaging Sciences · doi:10.1137/080716542 · Cited by 11,952
Introductory Lectures on Convex Optimization: A Basic Course
Nesterov, Y. E. (2014) · Medical Entomology and Zoology · Cited by 3,557
Introductory Lectures on Convex Optimization
Nesterov, Y. (2004) · Applied Optimization · doi:10.1007/978-1-4419-8853-9 · Cited by 2,698
RCV1: A New Benchmark Collection for Text Categorization Research
Lewis, D., Yang, Y., Rose, T. G., & Li, F. (2004) · Goldsmiths (University of London) · doi:10.5555/1005332.1005345 · Cited by 2,503
Splitting Algorithms for the Sum of Two Nonlinear Operators
Lions, P. L., & Mercier, B. (1979) · SIAM Journal on Numerical Analysis · doi:10.1137/0716071 · Cited by 1,972
Accelerating Stochastic Gradient Descent using Predictive Variance Reduction
Johnson, R., & Zhang, T. (2013) · Cited by 1,889
Self-citationGradient methods for minimizing composite functions
Nesterov, Y. (2013) · Mathematical Programming · doi:10.1007/s10107-012-0629-5 · Cited by 1,298
A Modified Forward-Backward Splitting Method for Maximal Monotone Mappings
Tseng, P. (2000) · SIAM Journal on Control and Optimization · doi:10.1137/s0363012998338806 · Cited by 1,223
Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems
Nesterov, Y. (2012) · SIAM Journal on Optimization · doi:10.1137/100802001 · Cited by 1,187
Minimizing finite sums with the stochastic average gradient
Schmidt, M., Le Roux, N., & Bach, F. (2017) · Mathematical Programming · doi:10.1007/s10107-016-1030-6 · Cited by 922
Efficient Online and Batch Learning Using Forward Backward Splitting
Duchi, J. C., & Singer, Y. (2009) · Journal of Machine Learning Research · Cited by 687
Richtárik, P., & Takáč, M. (2014) · Mathematical Programming · doi:10.1007/s10107-012-0614-z · Cited by 654
Stochastic dual coordinate ascent methods for regularized lossPreprint
Shalev‐Shwartz, S., & Zhang, T. (2013) · Cited by 637
Self-citationDual Averaging Method for Regularized Stochastic Learning and Online Optimization
Xiao, L. (2009) · Cited by 574
Self-citationA Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
Roux, N. L., Schmidt, M., & Bach, F. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1202.6258 · Cited by 481
Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm
Needell, D., Srebro, N., & Ward, R. (2016) · Mathematical Programming · doi:10.1007/s10107-015-0864-7 · Cited by 432
Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey
Bertsekas, D. P. (2011) · Optimization for Machine Learning · doi:10.7551/mitpress/8996.003.0006 · Cited by 381
Sample size selection in optimization methods for machine learning
Byrd, R. H., Chin, G. M., Nocedal, J., & Wu, Y. (2012) · Mathematical Programming · doi:10.1007/s10107-012-0572-5 · Cited by 369
Hybrid deterministic-stochastic methods for data fitting
Friedlander, M. P., & Schmidt, M. (2011) · Cited by 314
A Convergent Incremental Gradient Method with a Constant Step Size
Blatt, D., Hero, A. O., & Gauchman, H. (2007) · SIAM Journal on Optimization · doi:10.1137/040615961 · Cited by 306
Incremental proximal methods for large scale convex optimization
Bertsekas, D. P. (2011) · Mathematical Programming · doi:10.1007/s10107-011-0472-0 · Cited by 303
Convergence Rates in Forward--Backward Splitting
Chen, G. H.-G., & Rockafellar, R. T. (1997) · SIAM Journal on Optimization · doi:10.1137/s1052623495290179 · Cited by 285
Hybrid Deterministic-Stochastic Methods for Data Fitting
Friedlander, M. P., & Schmidt, M. (2012) · SIAM Journal on Scientific Computing · doi:10.1137/110830629 · Cited by 254
Semi-Stochastic Gradient Descent Methods
Konečný, J., & Richtárik, P. (2017) · Frontiers in Applied Mathematics and Statistics · doi:10.3389/fams.2017.00009 · Cited by 213
Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems
Lee, Y. T., & Sidford, A. (2013) · 2013 IEEE 54th Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2013.24 · Cited by 167
Accelerated Gradient Methods for Stochastic Optimization and Online Learning
Hu, C., Pan, W., & Kwok, J. T.-Y. (2009) · Cited by 140
Linear Convergence with Condition Number Independent Access of Full Gradients
Zhang, L., Mahdavi, M., & Jin, R. (2013) · Cited by 99
Stochastic Optimization with Importance SamplingPreprint
Zhao, P., & Zhang, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.2753 · Cited by 77
Self-citationMixed Optimization for Smooth Functions
Mahdavi, M., Zhang, L., & Jin, R. (2013) · Cited by 35
Cited by
Minimizing finite sums with the stochastic average gradient
Schmidt, M., Le Roux, N., & Bach, F. (2017) · Mathematical Programming · doi:10.1007/s10107-016-1030-6 · Cited by 922
SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
Defazio, A., Bach, F., & Lacoste-Julien, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.0202 · Cited by 834
Problem formulations and solvers in linear SVM: a review
Chauhan, V. K., Dahiya, K., & Sharma, A. (2019) · Artificial Intelligence Review · doi:10.1007/s10462-018-9614-6 · Cited by 660
Tank, A., Covert, I., Foti, N., Shojaie, A., & Fox, E. B. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3065601 · Cited by 282
Recent Advances in Stochastic Gradient Descent in Deep Learning
Tian, Y., Zhang, Y., & Zhang, H. (2023) · Mathematics · doi:10.3390/math11030682 · Cited by 277
Metadata
- Authors
- Xiao, L., & Zhang, T.
- Venue
- SIAM Journal on Optimization
- Published
- 2014-01
- Volume
- 24
- Issue
- 4
- Pages
- 2057-2075
- ISSN
- 1052-6234 (ISSN-L)
- 1095-7189
- OpenAlex
- W2047152541
- Cited by
- 592
- References
- 31
- FWCI
- 60.74
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.