Super-convergence: very fast training of neural networks using large learning rates
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
In this paper, we describe a phenomenon, which we named “super-convergence”, where neural networks can be trained an order of magnitude faster than with standard training methods. The existence of super-convergence is relevant to understanding why deep networks generalize well. One of the key elements of super-convergence is training with one learning rate cycle and a large maximum learning rate. A insight that allows super-convergence training is that large learning rates regularize the training, hence requiring a reduction of all other forms of regularization in order to preserve an optimal regularization balance. We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate. Experiments demonstrate super-convergence for Cifar-10/100, MNIST and Imagenet datasets, and resnet, wide-resnet, densenet, and inception architectures. In addition, we show that super-convergence provides a greater boost in performance relative to standard training when the amount of labeled training data is limited. The architectures and code to replicate the figures in this paper are available at github.com/lnsmith54/super-convergence.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
(2010) · Cited by 8,261
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“Please note that the philosophy behind CLR is a combination of curriculum learning [Bengio et al., 2009] and simulated annealing [Aarts and Korst, 1988], both of which have a long history of use in deep learning.”
BackgroundADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309
(2013) · Cited by 3,740
Optimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,128
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Cyclical Learning Rates for Training Neural Networks
Smith, L. N. (2017) · 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2017.58 · Cited by 2,670
Self-citation
Cited by
AI Feynman: A physics-inspired method for symbolic regression
Udrescu, S.-M., & Tegmark, M. (2020) · Science Advances · doi:10.1126/sciadv.aay2631 · Cited by 1,080
Smith, L. N. (2018) · arXiv · doi:10.48550/arxiv.1803.09820 · Cited by 790
Self-citationAdaBins: Depth Estimation Using Adaptive Bins
Farooq Bhat, S., Alhashim, I., & Wonka, P. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00400 · Cited by 726
Fast is better than free: Revisiting adversarial training
Wong, E., Rice, L., & Kolter, J. Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.03994 · Cited by 544
RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching
Lipson, L., Teed, Z., & Deng, J. (2021) · 2021 International Conference on 3D Vision (3DV) · doi:10.1109/3dv53792.2021.00032 · Cited by 536
Mish: A Self Regularized Non-Monotonic Activation Function
Misra, D. (2020) · Proceedings of the British Machine Vision Conference 2020 · doi:10.5244/c.34.191 · Cited by 432
Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds
Zhang, Y., Hu, Q., Xu, G., Ma, Y., Wan, J., & Guo, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01838 · Cited by 420
DaViT: Dual Attention Vision Transformers
Ding, M., Xiao, B., Codella, N., Luo, P., Wang, J., & Yuan, L. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20053-3_5 · Cited by 382
Learning to Estimate Hidden Motions with Global Motion Aggregation
Jiang, S., Campbell, D., Lu, Y., Li, H., & Hartley, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00963 · Cited by 365
Chemformer: a pre-trained transformer for computational chemistry
Irwin, R., Dimitriadis, S., He, J., & Bjerrum, E. J. (2022) · Machine Learning: Science and Technology · doi:10.1088/2632-2153/ac3ffb · Cited by 343
Sambasivam, G., & Opiyo, G. D. (2021) · Egyptian Informatics Journal · doi:10.1016/j.eij.2020.02.007 · Cited by 338
Gaitgraph: Graph Convolutional Network for Skeleton-Based Gait Recognition
Teepe, T., Khan, A., Gilg, J., Herzog, F., Hormann, S., & Rigoll, G. (2021) · 2021 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip42928.2021.9506717 · Cited by 323
Scene Text Recognition with Permuted Autoregressive Sequence Models
Bautista, D., & Atienza, R. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19815-1_11 · Cited by 267
Mask3D: Mask Transformer for 3D Semantic Instance Segmentation
Schult, J., Engelmann, F., Hermans, A., Litany, O., Tang, S., & Leibe, B. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160590 · Cited by 261
Ismail, N., & Malik, O. A. (2022) · Information Processing in Agriculture · doi:10.1016/j.inpa.2021.01.005 · Cited by 253
Robust Multi-Modality Multi-Object Tracking
Zhang, W., Zhou, H., Sun, S., Wang, Z., Shi, J., & Loy, C. C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00245 · Cited by 250
LogFiT: Log Anomaly Detection Using Fine-Tuned Language Models
Almodovar, C., Sabrina, F., Karimi, S., & Azad, S. (2024) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2024.3358730 · Cited by 111
“Fine-tuning: To adapt the RoBERTa or Longformer model to the log anomaly detection task, LogFiT utilises fine-tuning methods based on super-convergence techniques [41] implemented in the FastAI/ULMFiT framework [42], [43].”
MethodRegression-based Deep-Learning predicts molecular biomarkers from pathology slides
El Nahhas, O. S. M., Loeffler, C. M. L., Carrero, Z. I., van Treeck, M., Kolbinger, F. R., Hewitt, K. J., Muti, H. S., Graziani, M., Zeng, Q., Calderaro, J., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-45589-1 · Cited by 108
Qi, X., de Almeida, G. A. M., & Maldonado, S. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131263 · Cited by 78
Hannan, M. A., How, D. N. T., Mansor, M. B., Hossain Lipu, M. S., Ker, P., & Muttaqi, K. (2021) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2021.3065194 · Cited by 70
Binh, L. M., & Woo, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19886 · Cited by 63
MemFlow: Optical Flow Estimation and Prediction with Memory
Dong, Q., & Fu, Y. (2024) · arXiv · doi:10.48550/arxiv.2404.04808 · Cited by 49
Exploration-exploitation in multi-agent learning: Catastrophe theory meets game theory
Leonardos, S., & Piliouras, G. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2021.103653 · Cited by 39
Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learning
Le, B. M., & Woo, S. S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02045 · Cited by 36
“The models are trained with the Adam optimizer [30] with a learning rate of 2e-3, scheduled by one-cycle strategy [54] in 32 epochs.”
MethodNetwork Pruning That Matters: A Case Study on Retraining Variants
Le, D. H., & Hua, B.-S. (2021) · arXiv · doi:10.48550/arxiv.2105.03193 · Cited by 28
Doeleman, T., Brussee, S., Hondelink, L. M., Westerbeek, D. W. F., Sequeira, A. M., Valkema, P. A., Jansen, P. M., He, J., Vermeer, M. H., Quint, K. D., et al. (2025) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2024.07.036 · Cited by 7
PS4: A Next-Generation Dataset for Protein Single-Sequence Secondary Structure Prediction
Peracha, O. (2024) · BioTechniques · doi:10.2144/btn-2023-0024 · Cited by 7
King, O. N. F., Levik, K. E., Sandy, J., & Basham, M. (2024) · Acta Crystallographica Section D Structural Biology · doi:10.1107/s2059798324009276 · Cited by 5
(2025) · doi:10.1016/j.cels.2025.101265 · Cited by 3
Genome-scale annotation of protein binding sites via language model and geometric deep learning
(2023) · doi:10.1101/2023.11.02.565344 · Cited by 2
Sinitsin, A., Plokhotnyuk, V., Pyrkin, D., Popov, S., & Babenko, A. (2020) · arXiv · doi:10.48550/arxiv.2004.00345
Metadata
- Authors
- Smith, L. N., & Topin, N.
- Venue
- Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications
- Publisher
- SPIE
- Published
- 2019-05-10
- Pages
- 36
- OpenAlex
- W2749581528
- Cited by
- 1,335
- References
- 32
- FWCI
- 43.37
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.