Handling Vanishing Gradient Problem Using Artificial Derivative
Open accessAbstract
Sigmoid function and ReLU are commonly used activation functions in neural networks (NN). However, sigmoid function is vulnerable to the vanishing gradient problem, while ReLU has a special vanishing gradient problem that is called dying ReLU problem. Though many studies provided methods to alleviate this problem, there has not been an efficient feasible solution. Hence, we proposed a method replacing the original derivative function with an artificial derivative in a pertinent way. Our method optimized gradients of activation functions without varying activation functions nor introducing extra layers. Our investigations demonstrated that the method can effectively alleviate the vanishing gradient problem for both ReLU and sigmoid function with few computational cost.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In high-dimensional feature graphs, the inverted residual structure uses ReLU; in low-dimensional feature graphs, a linear activation function is used to solve information loss caused by ReLU.
“As shown in Figure 6, in high-dimensional feature graphs, the inverted residual structure uses the ReLU activation function [35], while in low-dimensional feature graphs, in order to solve the problem of information loss caused by the ReLU function when dealing with low-dimensional information, the linear activation function [36] is usually used.”
From Digital Image Copyright Protection and Management Approach—Based on Artificial Intelligence and Blockchain Technology (2025) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
(2010) · Cited by 14,134 · see citing works
(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,745 · see citing works
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513 · see citing works
Convergence Results for Neural Networks via Electrodynamics
Panigrahy, R., Rahimi, A., Sachdeva, S., et al. (2018) · Leibniz international proceedings in informatics · doi:10.4230/lipics.itcs.2018.22 · Cited by 2,848 · see citing works
Searching for activation functions
Ramachandran, P., Zoph, B., & Le, Q. V. (2017) · arXiv · doi:10.48550/arxiv.1710.05941 · Cited by 2,335 · see citing works
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
Bengio, Y., Léonard, N., & Courville, A. (2013) · arXiv · doi:10.48550/arxiv.1308.3432 · Cited by 1,910 · see citing works
All you need is a good initPreprint
Mishkin, D., & Matas, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06422 · Cited by 374 · see citing works
Qin, Y., Wang, X., & Zou, J. (2018) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2018.2856205 · Cited by 219 · see citing works
Wang, X., Yi, Q., Wang, Y., Xiang, S., & Chen, H. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.07.017 · Cited by 161 · see citing works
Failures of Gradient-Based Deep LearningPreprint
Shalev‐Shwartz, S., Shamir, O., & Shammah, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07950 · Cited by 108 · see citing works
Refinement and Universal Approximation via Sparsely Connected ReLU Convolution Nets
Heinecke, A., Ho, J., & Hwang, W.-L. (2020) · IEEE Signal Processing Letters · doi:10.1109/lsp.2020.3005051 · Cited by 33 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- L. Lu, Y. Shin, Y. Su, and G. E. Karniadakis, ''Dying ReLU and initializa- tion: Theory and numerical examples,'' 2019, arXiv:1903.06733. [Online]. Available: http://arxiv.org/abs/1903.06733 · arXiv
- A. L. Maas, A. Y. Hannun, and A. Y. Ng, ''Rectifier nonlinearities improve neural network acoustic models,'' in Proc. ICML, 2013, vol. 30, no. 1, p. 3.
- G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, ''Self- normalizing neural networks,'' in Proc. Adv. Neural Inf. Process. Syst., 2017, pp. 971-980.
- Y. Wu and K. He, ''Group normalization,'' in Proc. Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 3-19.
Cited by
Sorted by citations
A review on the recent applications of deep learning in predictive drug toxicological studies
Sinha, K., Ghosh, N., & Sil, P. C. (2023) · Chemical Research in Toxicology · doi:10.1021/acs.chemrestox.2c00375 · Cited by 64 · see citing works
Xu, J., Zhang, J., & Wang, J. (2025) · Journal of Theoretical and Applied Electronic Commerce Research · doi:10.3390/jtaer20020076 · Cited by 9 · see citing works
“…graphs, the inverted residual structure uses the ReLU activation function [35], while in low-dimensional feature graphs, in order to solve the problem of information loss caused by the ReLU function when dealing with low-dimensional information, the linear activation function [36] is usually used.”
Self-citation
- Authors
- Hu, Z., Zhang, J., & Ge, Y.
- Affiliations
- Nanchang Hangkong University (CN)
- Marche Polytechnic University (IT)
- Venue
- IEEE Access
- Published
- 2021
- Volume
- 9
- Pages
- 22371-22377
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3125415690
- Cited by
- 109
- References
- 18
- Citations by year
- 2026: 11 · 2025: 30 · 2024: 34 · 2023: 22 · 2022: 9 · 2021: 3
- FWCI
- 5.98
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- INFM-OAR (INFN Catania) (submittedVersion, cc-by-nc-nd)
- Zenodo (CERN European Organization for Nuclear Research) (other-oa)
- Funders
- National Natural Science Foundation of China (41801288)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- gradient vanishing, activation functions, sigmoid function, ReLU, gradient-based optimization, neural network training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
