The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Adaptive Restart for Accelerated Gradient Schemes
Open accessAbstract
In this paper we introduce a simple heuristic adaptive restart technique that can dramatically improve the convergence rate of accelerated gradient schemes. The analysis of the technique relies on the observation that these schemes exhibit two modes of behavior depending on how much momentum is applied at each iteration. In what we refer to as the ‘high momentum’ regime the iterates generated by an accelerated gradient scheme exhibit a periodic behavior, where the period is proportional to the square root of the local condition number of the objective function. Separately, it is known that the optimal restart interval is proportional to this same quantity. This suggests a restart technique whereby we reset the momentum whenever we observe periodic behavior. We provide a heuristic analysis that suggests that in many cases adaptively restarting allows us to recover the optimal rate of convergence with no prior knowledge of function parameters.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Donoho, D. L. (2006) · IEEE Transactions on Information Theory · doi:10.1109/tit.2006.871582 · Cited by 23,994
Donoho, D. L. (2004) · Cited by 16,703
(2021) · A First Course in Statistical Programming with R · doi:10.1017/9781108993456.011 · Cited by 13,724
A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
Beck, A., & Teboulle, M. (2009) · SIAM Journal on Imaging Sciences · doi:10.1137/080716542 · Cited by 11,952
An Introduction To Compressive Sampling
Candes, E. J., & Wakin, M. B. (2008) · IEEE Signal Processing Magazine · doi:10.1109/msp.2007.914731 · Cited by 10,017
Methods of conjugate gradients for solving linear systems
Hestenes, M. R., & Stiefel, E. (1952) · Journal of Research of the National Bureau of Standards · doi:10.6028/jres.049.044 · Cited by 8,132
Stable signal recovery from incomplete and inaccurate measurements
Candès, E. J., Romberg, J. K., & Tao, T. (2006) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.20124 · Cited by 7,036
Self-citationAn iterative thresholding algorithm for linear inverse problems with a sparsity constraint.
Daubechies, I., Defrise, M., & De Mol, C. (2004) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.20042 · Cited by 4,958
Introductory Lectures on Convex Optimization: A Basic Course
Nesterov, Y. E. (2014) · Medical Entomology and Zoology · Cited by 3,557
Introductory Lectures on Convex Optimization
Nesterov, Y. (2004) · Applied Optimization · doi:10.1007/978-1-4419-8853-9 · Cited by 2,698
(1989) · Choice Reviews Online · doi:10.5860/choice.26-5109 · Cited by 1,755
Fundamental methods of mathematical economics
(1984) · Journal of Macroeconomics · doi:10.1016/0164-0704(84)90011-9 · Cited by 1,539
Problem complexity and method efficiency in optimization
(1984) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(84)90101-0 · Cited by 1,329
Numerical Optimization (Springer Series in Operations Research and Financial Engineering)
Nocedal, J., & Wright, S. J. (2000) · Cited by 1,112
Gradient methods for minimizing composite objective function
Nesterov, Y. E. (2007) · Cited by 867
Chambolle, A., De Vore, R. A., Lee, N.-Y., & Lucier, B. J. (1998) · IEEE Transactions on Image Processing · doi:10.1109/83.661182 · Cited by 751
Templates for convex cone problems with applications to sparse signal recovery
Becker, S. R., Candès, E. J., & Grant, M. C. (2011) · Mathematical Programming Computation · doi:10.1007/s12532-011-0029-5 · Cited by 608
Self-citationInterior Gradient and Proximal Methods for Convex and Conic Optimization
Auslender, A., & Teboulle, M. (2006) · SIAM Journal on Optimization · doi:10.1137/s1052623403427823 · Cited by 269
An efficient algorithm for a class of fused lasso problems
Liu, J., Yuan, L., & Ye, J. (2010) · Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1835804.1835847 · Cited by 163
Lan, G., Lu, Z., & Monteiro, R. D. C. (2009) · Mathematical Programming · doi:10.1007/s10107-008-0261-6 · Cited by 147
EFFICIENT METHODS IN CONVEX PROGRAMMING
Nemirovski, A. S. (2007) · Cited by 114
Iteration-complexity of first-order penalty methods for convex programming
Lan, G., & Monteiro, R. D. C. (2012) · Mathematical Programming · doi:10.1007/s10107-012-0588-x · Cited by 68
Cited by
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309
Parikh, N., & Boyd, S. (2014) · Foundations and Trends® in Optimization · doi:10.1561/2400000003 · Cited by 4,102
Fast Alternating Direction Optimization Methods
Goldstein, T., O'Donoghue, B., Setzer, S., & Baraniuk, R. (2014) · SIAM Journal on Imaging Sciences · doi:10.1137/120896219 · Cited by 785
Self-citationConic Optimization via Operator Splitting and Homogeneous Self-Dual Embedding
O’Donoghue, B., Chu, E., Parikh, N., & Boyd, S. (2016) · Journal of Optimization Theory and Applications · doi:10.1007/s10957-016-0892-3 · Cited by 664
“’ There are several other methods that share similar characteristics, such as [55–62], as well as some techniques for solving the split-feasibility problem [35].”
MethodSelf-citationAn introduction to continuous optimization for imaging
Chambolle, A., & Pock, T. (2016) · Acta Numerica · doi:10.1017/s096249291600009x · Cited by 545
Metadata
- Authors
- O’Donoghue, B., & Candès, E.
- Venue
- Foundations of Computational Mathematics
- Published
- 2013-07-03
- Volume
- 15
- Issue
- 3
- Pages
- 715-732
- ISSN
- 1615-3375 (ISSN-L)
- 1615-3383
- OpenAlex
- W2127447444
- Cited by
- 642
- References
- 24
- FWCI
- 41.60
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.