A Delay-Free Adaptive Stepsize for the Incremental Aggregated Gradient Method
Abstract
In most existing asynchronous methods, the stepsize depends on an upper bound on the delays and decreases as this bound increases. However, since the upper bound is usually unknown and large, the resulting stepsizes are not only difficult to determine in practice but also overly conservative, which leads to slow convergence. To solve this issue, we propose an adaptive stepsize strategy for a typical asynchronous optimization method—the Incremental Aggregated Gradient (IAG) method. Unlike existing methods in the literature, our stepsize does not rely on any delay information and is less conservative, which leads to easier stepsize determination and faster convergence. Under standard assumptions, we provide the convergence rate of IAG with the proposed stepsize. Numerical experiments demonstrate the superior performance of our stepsize over alternative stepsize strategies.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · see citing works
Optimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. M. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,233 · see citing works
Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition
Karimi, H., Nutini, J., & Schmidt, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46128-1_50 · Cited by 842 · see citing works
Dalcín, L., Paz, R. R., & Storti, M. A. (2005) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2005.03.010 · Cited by 418 · see citing works
A Convergent Incremental Gradient Method with a Constant Step Size
Blatt, D., Hero, A. O., & Gauchman, H. (2007) · SIAM Journal on Optimization · doi:10.1137/040615961 · Cited by 316 · see citing works
ARock: An Algorithmic Framework for Asynchronous Parallel Coordinate Updates
Peng, Z., Xu, Y., Yan, M., & Yin, W. (2016) · SIAM Journal on Scientific Computing · doi:10.1137/15m1024950 · Cited by 238 · see citing works
On the Convergence Rate of Incremental Aggregated Gradient Algorithms
Gürbüzbalaban, M., Ozdaglar, A., & Parrilo, P. A. (2017) · SIAM Journal on Optimization · doi:10.1137/15m1049695 · Cited by 121 · see citing works
Advances in Asynchronous Parallel and Distributed Optimization
Assran, M., Aytekin, A., Feyzmahdavian, H. R., Johansson, M., & Rabbat, M. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3026619 · Cited by 77 · see citing works
Global Convergence Rate of Proximal Incremental Aggregated Gradient Methods
Vanli, N. D., Gürbüzbalaban, M., & Ozdaglar, A. (2018) · SIAM Journal on Optimization · doi:10.1137/16m1094415 · Cited by 51 · see citing works
Analysis and Implementation of an Asynchronous Optimization Algorithm for the Parameter ServerPreprint
Aytekin, A., Feyzmahdavian, H. R., & Johansson, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.05507 · Cited by 31 · see citing works
PRIAG: Proximal Reweighted Incremental Aggregated Gradient Algorithm for Distributed Optimizations
Deng, X., Sun, T., Liu, F., & Huang, F. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60245-1_34 · Cited by 1 · see citing works
Bertsekas, D. P. (2017) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2600425
Flagged first, then by citations
Research works printed but not linked: 9
- Feyzmahdavian, 2023, Asynchronous iterations in optimization: New sequence results and sharper algorithmic guarantees, J. Mach. Learn. Res., 24, 1
- Bertsekas, 1989, Parallel and Distributed Computation: Numerical Methods
- Niu, HOGWILD!: A lock-free approach to parallelizing stochastic gradient descent, 693
- Sun, General proximal incremental aggregated gradient algorithms: Better and novel results under general scheme, 32, 1
- Li, Parameter server for distributed machine learning, 6, 1
- Gu, Fast federated learning in the presence of arbitrary device unavailability, 34, 12052
- Wu, Delay-adaptive step-sizes for asynchronous learning, 162, 24093
- Roux, A stochastic gradient method with an exponential convergence rate for finite training sets, 25, 1
- Mishchenko, A delay-tolerant proximal-gradient algorithm for distributed learning, 80, 3587
Cited by
Still being checked.
- Authors
- Deng, Z., Wu, X., & Lu, J.
- Affiliations
- ShanghaiTech University (CN)
- Southern University of Science and Technology (CN)
- Venue
- IEEE Control Systems Letters
- Published
- 2025
- Volume
- 9
- Pages
- 3095-3100
- ISSN
- 2475-1456 (ISSN-L)
- OpenAlex
- W7117783371
- Cited by
- 0
- References
- 12
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Control Systems Letters (publishedVersion)
- Funders
- Shenzhen Science and Technology Program (JCYJ20241202125309014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Distributed Control Multi-Agent Systems
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Optimization and Variational Analysis
- Keywords
- adaptive step size, asynchronous optimization, distributed optimization, convergence rate, delay bounds
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
