The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Cyclical Learning Rates for Training Neural Networks
Open accessAbstract
It is known that the learning rate is the most important hyper-parameter to tune for training deep neural networks. This paper describes a new method for setting the learning rate, named cyclical learning rates, which practically eliminates the need to experimentally find the best values and schedule for the global learning rates. Instead of monotonically decreasing the learning rate, this method lets the learning rate cyclically vary between reasonable boundary values. Training with cyclical learning rates instead of fixed values achieves improved classification accuracy without a need to tune and often in fewer iterations. This paper also describes a simple way to estimate "reasonable bounds" - linearly increasing the learning rate of the network for a few epochs. In addition, cyclical learning rates are demonstrated on the CIFAR-10 and CIFAR-100 datasets with ResNets, Stochastic Depth networks, and DenseNets, and the ImageNet dataset with the AlexNet and GoogLeNet architectures. These are practical tools for everyone who trains neural networks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Going Deeper with Convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014) · arXiv · doi:10.48550/arxiv.1409.4842 · Cited by 1,461
Cited by
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
Desplanques, B., Thienpondt, J., & Demuynck, K. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2650 · Cited by 1,568
Smith, L. N. (2018) · arXiv · doi:10.48550/arxiv.1803.09820 · Cited by 790
Self-citationDeep Fluids: A Generative Network for Parameterized Fluid Simulations
Kim, B., Azevedo, V. C., Thuerey, N., Kim, T., Gross, M., & Solenthaler, B. (2019) · Computer Graphics Forum · doi:10.1111/cgf.13619 · Cited by 372
Predicting disruptive instabilities in controlled fusion plasmas through deep learning
Kates-Harbeck, J., Svyatkovskiy, A., & Tang, W. (2019) · Nature · doi:10.1038/s41586-019-1116-4 · Cited by 367
Diabetic Retinopathy Fundus Image Classification and Lesions Localization System Using Deep Learning
Alyoubi, W. L., Abulkhair, M. F., & Shalash, W. M. (2021) · Sensors · doi:10.3390/s21113704 · Cited by 303
Data Uncertainty Learning in Face Recognition
Chang, J., Lan, Z., Cheng, C., & Wei, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00575 · Cited by 320
Neural Prototype Trees for Interpretable Fine-grained Image Recognition
Nauta, M., van Bree, R., & Seifert, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01469 · Cited by 269
Long-term cancer survival prediction using multimodal deep learning
Vale-Silva, L. A., & Rohr, K. (2021) · Scientific Reports · doi:10.1038/s41598-021-92799-4 · Cited by 268
Ayala Solares, J. R., Diletta Raimondi, F. E., Zhu, Y., Rahimian, F., Canoy, D., Tran, J., Pinho Gomes, A. C., Payberah, A. H., Zottoli, M., Nazarzadeh, M., et al. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103337 · Cited by 247
SOC Estimation of Li-ion Batteries With Learning Rate-Optimized Deep Fully Convolutional Network
Hannan, M. A., How, D. N. T., Lipu, M. S. H., Ker, P. J., Dong, Z. Y., Mansur, M., & Blaabjerg, F. (2021) · IEEE Transactions on Power Electronics · doi:10.1109/tpel.2020.3041876 · Cited by 233
Deep learning enables genetic analysis of the human thoracic aorta
Pirruccello, J. P., Chaffin, M. D., Chou, E. L., Fleming, S. J., Lin, H., Nekoui, M., Khurshid, S., Friedman, S. F., Bick, A. G., Arduini, A., et al. (2022) · Nature Genetics · doi:10.1038/s41588-021-00962-4 · Cited by 210
Hannan, M. A., How, D. N. T., Lipu, M. S. H., Mansor, M., Ker, P. J., Dong, Z. Y., Sahari, K. S. M., Tiong, S. K., Muttaqi, K. M., Mahlia, T. M. I., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-98915-8 · Cited by 196
Spectral images based environmental sound classification using CNN with meaningful data augmentation
Mushtaq, Z., Su, S.-F., & Tran, Q.-V. (2021) · Applied Acoustics · doi:10.1016/j.apacoust.2020.107581 · Cited by 188
Cross Modal Transformer: Towards Fast and Robust 3D Object Detection
Yan, J., Liu, Y., Sun, J., Jia, F., Li, S., Wang, T., & Zhang, X. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01675 · Cited by 182
Medical imaging deep learning with differential privacy
Ziller, A., Usynin, D., Braren, R., Makowski, M., Rueckert, D., & Kaissis, G. (2021) · Scientific Reports · doi:10.1038/s41598-021-93030-0 · Cited by 172
Analysis of the Human Protein Atlas Image Classification competition
Ouyang, W., Winsnes, C. F., Hjelmare, M., Cesnik, A. J., Åkesson, L., Xu, H., Sullivan, D. P., Dai, S., Lan, J., Jinmo, P., et al. (2019) · Nature Methods · doi:10.1038/s41592-019-0658-6 · Cited by 131
IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection
Yin, J., Shen, J., Chen, R., Li, W., Yang, R., Frossard, P., & Wang, W. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01412 · Cited by 133
Pham, T.-C., Luong, C.-M., Hoang, V.-D., & Doucet, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-96707-8 · Cited by 101
Implicit Gradient Regularization
Barrett, D. G. T., & Dherin, B. (2020) · arXiv · doi:10.48550/arxiv.2009.11162 · Cited by 95
Boosting Fast Adversarial Training With Learnable Adversarial Initialization
Jia, X., Zhang, Y., Wu, B., Wang, J., & Cao, X. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3184255 · Cited by 69
Chan, S., Hang, Y., Tong, C., Acquah, A., Schonfeldt, A., Gershuny, J., & Doherty, A. (2024) · Scientific Data · doi:10.1038/s41597-024-03960-3 · Cited by 61
Model metamers reveal divergent invariances between biological and artificial neural networks
Feather, J., Leclerc, G., Mądry, A., & McDermott, J. H. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01442-0 · Cited by 59
Hunting for Insider Threats Using LSTM-Based Anomaly Detection
Villarreal-Vasquez, M., Modelo-Howard, G., Dube, S., & Bhargava, B. (2023) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2021.3135639 · Cited by 52
Insider Threat Detection using Deep Autoencoder and Variational Autoencoder Neural Networks
Pantelidis, E., Bendiab, G., Shiaeles, S., & Kolokotronis, N. (2021) · 2021 IEEE International Conference on Cyber Security and Resilience (CSR) · doi:10.1109/csr51186.2021.9527925 · Cited by 55
Reliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout
Cortés-Ciriano, I., & Bender, A. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00297 · Cited by 51
Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration
Gao, C., Liu, X., Gu, W., Killeen, B., Armand, M., Taylor, R., & Unberath, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-59716-0_32 · Cited by 46
An Efficient Deep Learning Based Method for Speech Assessment of Mandarin-Speaking Aphasic Patients
Mahmoud, S. S., Kumar, A., Tang, Y., Li, Y., Gu, X., Fu, J., & Fang, Q. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3011104 · Cited by 36
A Fully Differentiable Framework for 2D/3D Registration and the Projective Spatial Transformers
Gao, C., Feng, A., Liu, X., Taylor, R. H., Armand, M., & Unberath, M. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3299588 · Cited by 36
“The ProST network architecture was trained using an SGD optimizer with a cyclic learning rate between 10e-6 and 10e-4 every 100 steps [48] and a momentum of 0.9.”
MethodInfluentialPerformance Evaluation of Machine Learning Frameworks for Aphasia Assessment
Mahmoud, S. S., Kumar, A., Li, Y., Tang, Y., & Fang, Q. (2021) · Sensors · doi:10.3390/s21082582 · Cited by 27
“A learning rate range test was used to estimate the maximum learning rate’s optimum value [35].”
MethodMahmoud, S. S., Pallaud, R. F., Kumar, A., Faisal, S., Wang, Y., & Fang, Q. (2023) · Sensors · doi:10.3390/s23020857 · Cited by 25
Enhancing skin lesion classification: a CNN approach with human baseline comparison
Ajabani, D., Shaikh, Z. A., Yousef, A., Ali, K., & Albahar, M. A. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2795 · Cited by 18
Classification of error-related potentials evoked during stroke rehabilitation training
Kumar, A., Pirogova, E., Mahmoud, S. S., & Fang, Q. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac1d32 · Cited by 17
Quintus, S., Davis, D. S., & Cochrane, E. E. (2023) · Archaeological Prospection · doi:10.1002/arp.1909 · Cited by 12
Sequence-aware multimodal page classification of Brazilian legal documents
Luz de Araujo, P. H., de Almeida, A. P. G. S., Ataides Braz, F., Correia da Silva, N., de Barros Vidal, F., & de Campos, T. E. (2023) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-022-00406-7 · Cited by 9
End-to-end cryo-EM complex structure determination with high accuracy and ultra-fast speed
Wang, J., Tan, C., Gao, Z., Zhang, G., Zhang, Y., & Li, S. Z. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01056-0 · Cited by 8
A GPT-Based Approach for Sentiment Analysis and Bakery Rating Prediction
Magdaleno, D., Montes, M., Estrada, B., & Ochoa-Zezzatti, A. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-51940-6_7 · Cited by 7
Metadata
- Authors
- Smith, L. N.
- Venue
- 2017 IEEE Winter Conference on Applications of Computer Vision (WACV)
- Publisher
- IEEE
- Published
- 2017-03
- Pages
- 464-472
- ISSN
- 2472-6737 (ISSN-L)
- OpenAlex
- W2544860310
- Cited by
- 2,670
- References
- 7
- FWCI
- 86.51
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
