The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
ReLTanh: An activation function with vanishing gradient resistance for SAE-based DNNs and its application to rotating machinery fault diagnosis
Abstract
Abstract Tanh is a sigmoidal activation function that suffers from vanishing gradient problem, so researchers have proposed some alternative functions including rectified linear unit (ReLU), however those vanishing-proof functions bring some other problem such as bias shift problem and noise-sensitiveness as well. Mainly for overcoming vanishing gradient problem as well as avoiding to introduce other problems, we propose a new activation function named Rectified Linear Tanh (ReLTanh) by improving traditional Tanh. ReLTanh is constructed by replacing Tanh’s saturated waveforms in positive and negative inactive regions with two straight lines, and the slopes of the lines are calculated by the Tanh’s derivatives at two learnable thresholds. The middle Tanh waveform provides ReLTanh with the ability of nonlinear fitting, and the linear parts contribute to the relief of vanishing gradient problem. Besides, thresholds of ReLTanh that determines the slopes of line parts are learnable, so it can tolerate the variation of inputs and help to minimize the cost function and maximize the data fitting performance. Theoretical proofs by mathematical derivations demonstrate that ReLTanh is available to diminish vanishing gradient problem and feasible to train thresholds. For verifying the practical feasibility and effectiveness of ReLTanh, fault diagnosis experiments for planetary gearboxes and rolling bearings are conducted by stacked autoencoder-based deep neural network (SAE-based DNNs). ReLTanh alleviates successfully vanishing gradient problem and the it learns faster, more steadily and precisely than Tanh, which is consistent with the theoretical analysis. Additionally, ReLTanh surpasses other popular activation functions such as ReLU family, Hexpo and Swish, which shows that ReLTanh has certain applying potential and researching value.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513
Searching for activation functions
Ramachandran, P., Zoph, B., & Le, Q. V. (2017) · arXiv · doi:10.48550/arxiv.1710.05941 · Cited by 2,335
Improving deep neural networks for LVCSR using rectified linear units and dropout
Dahl, G. E., Sainath, T. N., & Hinton, G. E. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639346 · Cited by 1,310
Cited by
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Activation functions in deep learning: a comprehensive survey and benchmark
Dubey, S. R., Singh, S. K., & Chaudhuri, B. B. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.06.111 · Cited by 1,259
Cheng, Y., Lin, M., Wu, J., Zhu, H., & Shao, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106796 · Cited by 349
Activated Gradients for Deep Neural Networks
Liu, M., Chen, L., Du, X., Jin, L., & Shang, M. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3106044 · Cited by 304
Ahmad, S., Shakeel, I., Mehfuz, S., & Ahmad, J. (2023) · Computer Science Review · doi:10.1016/j.cosrev.2023.100568 · Cited by 197
Handling Vanishing Gradient Problem Using Artificial Derivative
Hu, Z., Zhang, J., & Ge, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3054915 · Cited by 109
Metadata
- Authors
- Wang, X., Qin, Y., Wang, Y., Xiang, S., & Chen, H.
- Venue
- Neurocomputing
- Publisher
- Elsevier BV
- Published
- 2019-10
- Volume
- 363
- Pages
- 88-98
- ISSN
- 0925-2312 (ISSN-L)
- OpenAlex
- W2963009098
- Cited by
- 161
- References
- 4
- FWCI
- 13.18
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.