Acceleration of Stochastic Approximation by Averaging
Abstract
A new recursive algorithm of stochastic approximation type with the averaging of trajectories is investigated. Convergence with probability one is proved for a variety of classical optimization and identification problems. It is also demonstrated for these problems that the proposed algorithm achieves the highest possible rate of convergence.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The momentum teacher in DINO can be interpreted as a form of Polyak-Ruppert averaging with an exponential decay.
“We propose to interpret the momentum teacher in DINO as a form of Polyak-Ruppert averaging [51, 59] with an exponentially decay.”
From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [51]
Weight averaging usually produces a better model than the individual models from each iteration.
“Indeed, as motivated in Tarvainen et al. [65], weight averaging usually produces a better model than the individual models from each iteration [51].”
From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [51]
Cited by
Sorted by citations
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
“We include most of the augmentation and regularization strategies of [57] in training, except for repeated augmentation [28] and EMA [41], which do not enhance performance.”
MethodAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
“For ImageNet results in Table 2, we fine-tuned at higher resolution: 512 for ViT-L/16 and 518 for ViT-H/14, and also used Polyak & Juditsky (1992) averaging with a factor of 0 .”
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · see citing works
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01167 · Cited by 8,494 · see citing works
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795 · see citing works
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · see citing works
“Indeed, we observe that this teacher performs a form of model ensembling similar to Polyak-Ruppert averaging with an exponential decay [41, 48].”
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259 · see citing works
Going Deeper with Convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014) · arXiv · doi:10.48550/arxiv.1409.4842 · Cited by 1,461 · see citing works
Zhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891 · see citing works
An optimal method for stochastic composite optimization
Lan, G. (2012) · Mathematical Programming · doi:10.1007/s10107-010-0434-y · Cited by 427 · see citing works
Adaptive stochastic approximation by the simultaneous perturbation method
Spall, J. C. (2000) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2000.880982 · Cited by 424 · see citing works
Robust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428 · see citing works
A one-measurement form of simultaneous perturbation stochastic approximation
Spall, J. C. (1997) · Automatica · doi:10.1016/s0005-1098(96)00149-5 · Cited by 254 · see citing works
fMRI reveals language-specific predictive coding during naturalistic sentence comprehension
Shain, C., Blank, I. A., van Schijndel, M., Schuler, W., & Fedorenko, E. (2020) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2019.107307 · Cited by 215 · see citing works
Unsupervised speech representation learning using WaveNet autoencoders
Chorowski, J., Weiss, R. J., Bengio, S., & van den Oord, A. (2019) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2019.2938863 · Cited by 325 · see citing works
Xie, J., Ajagekar, A., & You, F. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121162 · Cited by 109 · see citing works
Noise-resilient variational hybrid quantum-classical optimization
Gentini, L., Cuccoli, A., Pirandola, S., Verrucchi, P., & Banchi, L. (2020) · Physical Review A · doi:10.1103/physreva.102.052414 · Cited by 46 · see citing works
- Authors
- Polyak, B. T., & Juditsky, A. B.
- Venue
- SIAM Journal on Control and Optimization
- Published
- 1992-07
- Volume
- 30
- Issue
- 4
- Pages
- 838-855
- ISSN
- 0363-0129 (ISSN-L)
- 1095-7138
- OpenAlex
- W2086161653
- Cited by
- 1,827
- References
- 0
- Citations by year
- 2026: 108 · 2025: 124 · 2024: 130 · 2023: 125 · 2022: 131 · 2021: 175 · 2020: 147 · 2019: 170 · 2018: 93 · 2017: 88 · 2016: 73 · 2015: 53 · 2014: 55 · 2013: 38 · 2012: 36
- FWCI
- 17.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- SIAM Journal on Control and Optimization (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Simulation Techniques and Applications
- Physical Sciences › Engineering › Control and Systems Engineering › Control Systems and Identification
- Physical Sciences › Engineering › Control and Systems Engineering › Extremum Seeking Control Systems
- Keywords
- stochastic approximation, recursive algorithm, almost sure convergence, convergence rate, optimization, system identification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
