diffGrad: An Optimization Method for Convolutional Neural Networks
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
Stochastic gradient descent (SGD) is one of the core techniques behind the success of deep neural networks. The gradient provides information on the direction in which a function has the steepest rate of change. The main problem with basic SGD is to change by equal-sized steps for all parameters, irrespective of the gradient behavior. Hence, an efficient way of deep network optimization is to have adaptive step sizes for each parameter. Recently, several attempts have been made to improve gradient descent methods such as AdaGrad, AdaDelta, RMSProp, and adaptive moment estimation (Adam). These methods rely on the square roots of exponential moving averages of squared past gradients. Thus, these methods do not take advantage of local change in gradients. In this article, a novel optimizer is proposed based on the difference between the present and the immediate past gradient (i.e., diffGrad). In the proposed diffGrad optimization technique, the step size is adjusted for each parameter in such a way that it should have a larger step size for faster gradient changing parameters and a lower step size for lower gradient changing parameters. The convergence analysis is done using the regret bound approach of the online learning framework. In this article, thorough analysis is made over three synthetic complex nonconvex functions. The image categorization experiments are also conducted over the CIFAR10 and CIFAR100 data sets to observe the performance of diffGrad with respect to the state-of-the-art optimizers such as SGDM, AdaGrad, AdaDelta, RMSProp, AMSGrad, and Adam. The residual unit (ResNet)-based convolutional neural network (CNN) architecture is used in the experiments. The experiments show that diffGrad outperforms other optimizers. Also, we show that diffGrad performs uniformly well for training CNN using different activation functions. The source code is made publicly available at https://github.com/shivram1987/diffGrad.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“The ResNet based CNN architecture introduced by He et al. [20] is one of the most accurate models which won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [45]”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
(2016) · Cited by 11,679
Image Super-Resolution Using Deep Convolutional Networks
Dong, C., Loy, C. C., He, K., & Tang, X. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2439281 · Cited by 10,105
Path Aggregation Network for Instance Segmentation
Liu, S., Qi, L., Qin, H., Shi, J., & Jia, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00913 · Cited by 10,011
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365 · cited 1×
“The moment can be imagined as a ball rolling down a slope, where Adam behaves like a heavy ball with friction, which thus prefers flat minima in the error surface [39].”
Background(2010) · Cited by 8,261
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955 · cited 1×
“…generally outperform the traditional hand-designed approaches to solve research problems in Computer Vision [2], [3], [4], [5], Image Processing [6], [7], Signal Processing [8], [9], Robotics [10], Natural Language Processing [11], [12], and many other diverse areas of Artificial Intelligence.”
BackgroundLarge-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 1×
“Due to the availability of GPU-based highend computational facilities and the huge amount of data, deep learning based approaches generally outperform the traditional hand-designed approaches to solve research problems in Computer Vision [2], [3], [4], [5], Image Processing [6], [7], Signal Processing [8], [9], Roboti…”
BackgroundFast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513 · cited 1×
“In this experiment, we have considered four activation functions, namely, ReLU [2], Leaky ReLU (LReLU) [47], Exponential Linear Unit (ELU) [48], and Scaled ELU (SELU) [49].”
MethodLarge Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
(2013) · Cited by 3,740
Convergence Results for Neural Networks via Electrodynamics
Panigrahy, R., Rahimi, A., Sachdeva, S., & Zhang, Q. (2018) · Leibniz international proceedings in informatics · doi:10.4230/lipics.itcs.2018.22 · Cited by 2,761
On the momentum term in gradient descent learning algorithms
Qian, N. (1999) · Neural Networks · doi:10.1016/s0893-6080(98)00116-6 · Cited by 2,312
Self-Normalizing Neural Networks
Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.02515 · Cited by 2,267
HybridSN: Exploring 3-D–2-D CNN Feature Hierarchy for Hyperspectral Image Classification
Roy, S. K., Krishna, G., Dubey, S. R., & Chaudhuri, B. B. (2020) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2019.2918719 · Cited by 1,898
Self-citationOn the Convergence of Adam and Beyond
Reddi, S. J., Kale, S., & Kumar, S. (2019) · arXiv · doi:10.48550/arxiv.1904.09237 · Cited by 1,771
(2003) · Cited by 1,733
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2572 · Cited by 827
GANs Trained by a Two Time-Scale Update Rule Converge to a Nash EquilibriumPreprint
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., & Hochreiter, S. (2017) · arXiv (Cornell University) · Cited by 553
signSGD: Compressed Optimisation for Non-Convex Problems
Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., & Anandkumar, A. (2018) · arXiv · doi:10.48550/arxiv.1802.04434 · Cited by 541
Deep Learning and Its Applications to Signal and Information Processing [Exploratory DSP
Yu, D., & Deng, L. (2011) · IEEE Signal Processing Magazine · doi:10.1109/msp.2010.939038 · Cited by 455
Neural Networks for Machine Learning
(2017) · Human and Machine Hearing · doi:10.1017/9781139051699.031 · Cited by 325
Deep Belief Networks Based Voice Activity Detection
Zhang, X.-L., & Wu, J. (2013) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2012.2229986 · Cited by 313
Two problems with backpropagation and other steepest-descent learning procedures for networks
Sutton, R. S. (1986) · eScholarship (California Digital Library) · Cited by 264
Joint Sparse Representation and Robust Feature-Level Fusion for Multi-Cue Visual Tracking
Lan, X., Ma, A. J., Yuen, P. C., & Chellappa, R. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2481325 · Cited by 217
Learning Common and Feature-Specific Patterns: A Novel Multiple-Sparse-Representation-Based Tracker
Lan, X., Zhang, S., Yuen, P. C., & Chellappa, R. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2777183 · Cited by 208
A Performance Evaluation of Convolutional Neural Networks for Face Anti Spoofing
Nagpal, C., & Dubey, S. R. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852422 · Cited by 130
Self-citationReddy, S., Karri, S. T., Dubey, S. R., & Mukherjee, S. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852419 · Cited by 127
Self-citationLearning Modality-Consistency Feature Templates: A Robust RGB-Infrared Tracking System
Lan, X., Ye, M., Shao, R., Zhong, B., Yuen, P. C., & Zhou, H. (2019) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2019.2898618 · Cited by 124
Joint Discriminative Learning of Deep Dynamic Textures for 3D Mask Face Anti-Spoofing
Shao, R., Lan, X., & Yuen, P. C. (2019) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2018.2868230 · Cited by 103
A Biologically Inspired Appearance Model for Robust Visual Tracking
Zhang, S., Lan, X., Yao, H., Zhou, H., Tao, D., & Li, X. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2586194 · Cited by 94
Nostalgic Adam: Weighting More of the Past Gradients When Designing the Adaptive Learning Rate
Huang, H., Wang, C., & Dong, B. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/355 · Cited by 74
Basha, S. H. S., Ghosh, S., Babu, K. K., Dubey, S. R., Pulabaigari, V., & Mukherjee, S. (2018) · International Conference on Control, Automation, Robotics and Vision (ICARCV) · doi:10.1109/icarcv.2018.8581147 · Cited by 67
Self-citation
Cited by
Activation functions in deep learning: a comprehensive survey and benchmark
Dubey, S. R., Singh, S. K., & Chaudhuri, B. B. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.06.111 · Cited by 1,259
Self-citationArtificial Neural Networks Based Optimization Techniques: A Review
Abdolrasol, M. G. M., Hussain, S. M. S., Ustun, T. S., Sarker, M. R., Hannan, M. A., Mohamed, R., Ali, J. A., Mekhilef, S., & Milad, A. (2021) · Electronics · doi:10.3390/electronics10212689 · Cited by 756
Multimodal Fusion Transformer for Remote Sensing Image Classification
Roy, S. K., Deria, A., Hong, D., Rasti, B., Plaza, A., & Chanussot, J. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3286826 · Cited by 519
Self-citationRNN-LSTM: From applications to modeling techniques and beyond—Systematic review
Al-Selwi, S. M., Hassan, M. F., Abdulkadir, S. J., Muneer, A., Sumiea, E. H., Alqushaibi, A., & Ragab, M. G. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102068 · Cited by 505
Hyperspectral Image Classification—Traditional to Deep Models: A Survey for Future Prospects
Ahmad, M., Shabbir, S., Roy, S. K., Hong, D., Wu, X., Yao, J., Khan, A. M., Mazzara, M., Distefano, S., & Chanussot, J. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3133021 · Cited by 438
Self-citation
Metadata
- Authors
- Dubey, S. R., Chakraborty, S., Roy, S. K., Mukherjee, S., Singh, S. K., & Chaudhuri, B. B.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2020-11
- Volume
- 31
- Issue
- 11
- Pages
- 4500-4511
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W3084521418
- Cited by
- 270
- References
- 50
- FWCI
- 11.17
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.