CoChat
Sign Up
PreprintCited by 5,309Stochastic Gradient Optimization TechniquesMachine Learning and Data ClassificationDomain Adaptation and Few-Shot Learning

SGDR: Stochastic Gradient Descent with Warm Restarts

Open access
Loshchilov, Ilya, Hutter, Frank·2016·arXiv
DOI 10.48550/arxiv.1608.03983arXiv 1608.03983
Open this paper in CoChat — free accountOpen PDF

Abstract

Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1608.03983

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The warmup-constant learning rate schedule performs comparably to a half-cosine schedule.

    “Similarly to Hägele et al. (2024), we found that this schedule performs comparably to a half-cosine schedule (Loshchilov and Hutter, 2016); it also makes exploring long training runs more cost-effective, since multiple cooldown runs can be started from different checkpoints of the constant phase.”

    From V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025) · cites this paper as Loshchilov and Hutter, 2016

  • DINO uses centering to prevent collapse while UIC resorts to balance sampling techniques as in [8].

    “However, we use centering to prevent collapse while UIC resorts to balance sampling techniques as in [8].”

    From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [8]

    The cited paper’s Section 2.2 (p. 3) discusses warm restarts in gradient-based optimization, not UIC or balanced sampling. Searches for UIC, balanced sampling, centering, and collapse found no passage addressing the claim attributed to [8]. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 31

  1. Antoine Bordes, L´ eon Bottou, and Patrick Gallinari. Sgd-qn: Careful quasi-newton stochastic gradient descent. The Journal of Machine Learning Research , 10:1737-1754, 2009.
  2. Anna Choromanska, Mikael Henaff, Michael Mathieu, G´ erard Ben Arous, and Yann LeCun. The loss surface of multilayer networks. arXiv preprint arXiv:1412.0233 , 2014. · arXiv
  3. Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advances in Neural Information Processing Systems , pp. 2933-2941, 2014.
  4. Yann N Dauphin, Harm de Vries, Junyoung Chung, and Yoshua Bengio. Rmsprop and equilibrated adaptive learning rates for non-convex optimization. arXiv preprint arXiv:1502.04390 , 2015. · arXiv
  5. L. Deng, G. Hinton, and B. Kingsbury. New types of deep neural network learning for speech recognition and related applications: An overview. In Proc. of ICASSP'13 , 2013.
  6. Reeves Fletcher and Colin M Reeves. Function minimization by conjugate gradients. The computer journal , 7(2):149-154, 1964.
  7. Kenji Fukumizu and Shun-ichi Amari. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Networks , 13(3):317-327, 2000.
  8. Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. arXiv preprint arXiv:1610.02915 , 2016. · arXiv
  9. Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 , 2015. · arXiv
  10. Nikolaus Hansen. Benchmarking a BI-population CMA-ES on the BBOB-2009 function testbed. In Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers , pp. 2389-2396. ACM, 2009.
  11. Nikolaus Hansen and Stefan Kern. Evaluating the cma evolution strategy on multimodal test functions. In International Conference on Parallel Problem Solving from Nature , pp. 282-291. Springer, 2004.
  12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027 , 2016. · arXiv
  13. Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get m for free. ICLR 2017 submission , 2016a.
  14. Gao Huang, Zhuang Liu, and Kilian Q Weinberger. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993 , 2016b. · arXiv
  15. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. arXiv preprint arXiv:1603.09382 , 2016c. · arXiv
  16. A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Proc. of NIPS'12 , pp. 1097-1105, 2012a.
  17. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pp. 1097-1105, 2012b.
  18. Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming , 45(1-3):503-528, 1989.
  19. Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Restarts. arXiv preprint arXiv:1608.03983 , 2016.
  20. Ilya Loshchilov, Marc Schoenauer, and Michele Sebag. Alternative restart strategies for CMA-ES. In International Conference on Parallel Problem Solving from Nature , pp. 296-305. Springer, 2012.
  21. Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady , volume 27, pp. 372-376, 1983.
  22. Yurii Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2013.
  23. Brendan O'Donoghue and Emmanuel Candes. Adaptive restart for accelerated gradient schemes. arXiv preprint arXiv:1204.3982 , 2012. · arXiv
  24. Michael James David Powell. Restart procedures for the conjugate gradient method. Mathematical programming , 12(1):241-254, 1977.
  25. Mike Preuss. Niching the CMA-ES via nearest-better clustering. In Proceedings of the 12th annual conference companion on Genetic and evolutionary computation , pp. 1711-1718. ACM, 2010.
  26. Mike Preuss. Niching methods and multimodal optimization performance. In Multimodal Optimization by Means of Evolutionary Algorithms , pp. 115-137. Springer, 2015.
  27. Raymond Ros. Benchmarking the bfgs algorithm on the bbob-2009 function testbed. In Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers , pp. 2409-2414. ACM, 2009.
  28. Leslie N Smith. No more pesky learning rate guessing games. arXiv preprint arXiv:1506.01186 , 2015. · arXiv
  29. Leslie N Smith. Cyclical learning rates for training neural networks. arXiv preprint arXiv:1506.01186v3 , 2016. · arXiv
  30. Tianbao Yang and Qihang Lin. Stochastic subgradient methods with linear convergence for polyhedral convex optimization. arXiv preprint arXiv:1510.01444 , 2015. · arXiv
  31. K. Zhang, M. Sun, T. X. Han, X. Yuan, L. Guo, and T. Liu. Residual Networks of Residual Networks: Multilevel Residual Networks. ArXiv e-prints , August 2016.

Cited by

Sorted by citations

Authors
Loshchilov, I., & Hutter, F.
Affiliations
University of Freiburg (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2016
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1608.03983
arXiv
1608.03983
OpenAlex
W2518108298
Cited by
5,309
References
6
Citations by year
2026: 70 · 2025: 627 · 2024: 1056 · 2023: 1044 · 2022: 715 · 2021: 868 · 2020: 515 · 2019: 287 · 2018: 93 · 2017: 27 · 2016: 6 · 2015: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
stochastic gradient descent, deep neural network training, convergence rate, multimodal optimization, EEG classification, ImageNet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review