The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Activated Gradients for Deep Neural Networks
Open accessAbstract
Deep neural networks often suffer from poor performance or even training failure due to the ill-conditioned problem, the vanishing/exploding gradient problem, and the saddle point problem. In this article, a novel method by acting the gradient activation function (GAF) on the gradient is proposed to handle these challenges. Intuitively, the GAF enlarges the tiny gradients and restricts the large gradient. Theoretically, this article gives conditions that the GAF needs to meet and, on this basis, proves that the GAF alleviates the problems mentioned above. In addition, this article proves that the convergence rate of SGD with the GAF is faster than that without the GAF under some assumptions. Furthermore, experiments on CIFAR, ImageNet, and PASCAL visual object classes confirm the GAF's effectiveness. The experimental results also demonstrate that the proposed method is able to be adopted in various deep neural networks to improve their performance. The source code is publicly available at https://github.com/LongJin-lab/Activated-Gradients-for-Deep-Neural-Networks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
(2010) · Cited by 14,134
Squeeze-and-Excitation Networks
Hu, J., Shen, L., Albanie, S., Sun, G., & Wu, E. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2913372 · Cited by 12,157
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665 · cited 2×
“ResNet deals with vanishing and exploding gradient problems by involving identity skip connection, which makes the coefficient in front of the gradient in BP close to 1 [50].”
BackgroundCascade R-CNN: Delving Into High Quality Object Detection
Cai, Z., & Vasconcelos, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00644 · Cited by 7,370
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309
Moses: open source toolkit for statistical machine translation
Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735 · cited 1×
“ARTIFICIAL neural networks are applied to a lot of fields, such as machine translation [1], speech recognition [2], object detection [3], robotics [4]–[6], and intelligent control [7].”
MethodFast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513 · cited 2×
“[35], and Kingma and Ba [43], for experiments on ImageNet, the validation set is used for testing with single crop of size 224 × 224.”
MethodOptimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,128
Measurement Reliability and Validity
(2000) · Research Methods in Applied Settings · doi:10.4324/9781410605337-29 · Cited by 2,541
Searching for activation functions
Ramachandran, P., Zoph, B., & Le, Q. V. (2017) · arXiv · doi:10.48550/arxiv.1710.05941 · Cited by 2,335 · cited 1×
“ReLU and its variants (such as Gaussian error linear units [33], Swish [34], exponential linear units [35], and scaled exponential linear units [36]) are employed to mitigate the vanishing gradient problem in deep neural network models.”
MethodOn the momentum term in gradient descent learning algorithms
Qian, N. (1999) · Neural Networks · doi:10.1016/s0893-6080(98)00116-6 · Cited by 2,312
Self-Normalizing Neural Networks
Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.02515 · Cited by 2,267
Designing Network Design Spaces
Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., & Dollar, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01044 · Cited by 2,084
Stochastic Gradient Descent Tricks
Bottou, L. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_25 · Cited by 1,929
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2572 · Cited by 827
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
Chen, J., Zhou, D., Tang, Y., Yang, Z., Cao, Y. J., & Gu, Q. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/452 · Cited by 237
On the Effectiveness of Least Squares Generative Adversarial Networks
Mao, X., Li, Q., Xie, H., Lau, R. Y. K., Wang, Z., & Smolley, S. P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2872043 · Cited by 225
Wang, H., Liu, P. X., Bao, J., Xie, X.-J., & Li, S. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2912082 · Cited by 223
Gradient Centralization: A New Optimization Technique for Deep Neural Networks
Yong, H., Huang, J., Hua, X., & Zhang, L. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_37 · Cited by 196
Accelerated Distributed Nesterov Gradient Descent
Qu, G., & Li, N. (2020) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2019.2937496 · Cited by 187
Jin, L., Yan, J., Du, X., et al. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.2964817 · Cited by 182
Self-citationLearning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization
Wang, G., Giannakis, G. B., & Chen, J. (2019) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2019.2904921 · Cited by 170
$L1$ -Norm Batch Normalization for Efficient Training of Deep Neural Networks
Wu, S., Li, G., Deng, L., Liu, L., Wu, D., Xie, Y., & Shi, L. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876179 · Cited by 166
Motor Learning and Generalization Using Broad Learning Adaptive Neural Control
Huang, H., Zhang, T., Yang, C., & Chen, C. L. P. (2020) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2019.2950853 · Cited by 165
Optimal Robot–Environment Interaction Under Broad Fuzzy Neural Adaptive Control
Huang, H., Yang, C., & Chen, C. L. P. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2998984 · Cited by 164
Wang, X., Qin, Y., Wang, Y., Xiang, S., & Chen, H. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.07.017 · Cited by 161
Dependency-Guided LSTM-CRF for Named Entity Recognition
Jie, Z., & Lu, W. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1399 · Cited by 134
Online GMM Clustering and Mini-Batch Gradient Descent Based Optimization for Industrial IoT 4.0
Messaoud, S., Bradai, A., & Moulay, E. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2945012 · Cited by 62
Weighted Quantization-Regularization in DNNs for Weight Memory Minimization Toward HW Implementation
Wess, M., Dinakarrao, S. M. P., & Jantsch, A. (2018) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2018.2857080 · Cited by 38
Perceptual Information Loss due to Impaired Speech Production
Asaei, A., Cerňak, M., & Bourlard, H. A. (2017) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2738445 · Cited by 18
Cited by
Gradient-Based Differential Neural-Solution to Time-Dependent Nonlinear Optimization
Jin, L., Wei, L., & Li, S. (2023) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2022.3144135 · Cited by 274
Self-citation
Metadata
- Authors
- Liu, M., Chen, L., Du, X., Jin, L., & Shang, M.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2023-04
- Volume
- 34
- Issue
- 4
- Pages
- 2156-2168
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W3196353935
- Cited by
- 304
- References
- 34
- FWCI
- 12.79
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.