Empirical Evaluation of Rectified Activations in Convolutional Network
Open accessAbstract
In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evaluate these activation function on standard image classification task. Our experiments suggest that incorporating a non-zero slope for negative part in rectified activation units could consistently improve the results. Thus our findings are negative on the common belief that sparsity is the key of good performance in ReLU. Moreover, on small scale dataset, using deterministic negative slope or learning it are both prone to overfitting. They are not as effective as using their randomized counterpart. By using RReLU, we achieved 75.68\% accuracy on CIFAR-100 test set without multiple test or ensemble.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ReLU could be utilized as a non-saturating activation function to enhance the rate of convergence by reducing the vanishing gradient problem.
“Moreover, by reducing the vanishing gradient problem, ReLU [92] could be utilized as a non-saturating activation function to enhance the rate of convergence [93].”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [92]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
Flagged first, then by citations
Research works printed but not linked: 14
- Girshick, Ross, Donahue, Jeff, Darrell, Trevor, and Malik, Jitendra. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR , pp. 580-587, 2014.
- Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua. Deep sparse rectifier networks. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. JMLR W&CP Volume , volume 15, pp. 315-323, 2011.
- He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Spatial pyramid pooling in deep convolutional networks for visual recognition. In ECCV , pp. 346-361, 2014.
- Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015. · arXiv
- Krizhevsky, Alex and Hinton, Geoffrey. Learning multiple layers of features from tiny images. Computer Science Department, University of Toronto, Tech. Rep , 1(4):7, 2009.
- Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In NIPS , pp. 1097-1105, 2012.
- Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network in network. arXiv preprint arXiv:1312.4400 , 2013. · arXiv
- Maas, Andrew L, Hannun, Awni Y, and Ng, Andrew Y. Rectifier nonlinearities improve neural network acoustic models. In ICML , volume 30, 2013.
- Nair, Vinod and Hinton, Geoffrey E. Rectified linear units improve restricted Boltzmann machines. In ICML , pp. 807-814, 2010.
- Russakovsky, Olga, Deng, Jia, Su, Hao, Krause, Jonathan, Satheesh, Sanjeev, Ma, Sean, Huang, Zhiheng, Karpathy, Andrej, Khosla, Aditya, Bernstein, Michael, Berg, Alexander C., and Fei-Fei, Li. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 2015. doi: 10.1007/s11263-015-0816-y. · DOI · Paper page
- Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research , 15(1):1929-1958, 2014.
- Sun, Yi, Wang, Xiaogang, and Tang, Xiaoou. Deeply learned face representations are sparse, selective, and robust. arXiv preprint arXiv:1412.1265 , 2014. · arXiv
- Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. arXiv preprint arXiv:1409.4842 , 2014. · arXiv
- Wang, Naiyan, Li, Siyi, Gupta, Abhinav, and Yeung, Dit-Yan. Transferring rich feature hierarchies for robust visual tracking. arXiv preprint arXiv:1501.04587 , 2015. · arXiv
Cited by
Sorted by citations
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906 · see citing works
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576 · see citing works
Self-citationDeep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503 · see citing works
Time series classification from scratch with deep neural networks: A strong baseline
Wang, Z., Yan, W., & Oates, T. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966039 · Cited by 2,299 · see citing works
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685 · see citing works
Activation functions in deep learning: a comprehensive survey and benchmark
Dubey, S. R., Singh, S. K., & Chaudhuri, B. B. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.06.111 · Cited by 1,259 · see citing works
DeepInf: Social Influence Prediction with Deep Learning
Qiu, J., Tang, J., Ma, H., Dong, Y., Wang, K., & Tang, J. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220077 · Cited by 587 · see citing works
CycleGAN-VC: Non-parallel Voice Conversion Using Cycle-Consistent Adversarial Networks
Kaneko, T., & Kameoka, H. (2018) · 2018 26th European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553236 · Cited by 358 · see citing works
Deep learning in ultrasound imaging
van Sloun, R. J. G., Cohen, R., & Eldar, Y. C. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2932116 · Cited by 316 · see citing works
Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video
Pigou, L., van den Oord, A., Dieleman, S., Van Herreweghe, M., & Dambre, J. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-016-0957-7 · Cited by 290 · see citing works
Artificial intelligence in dentistry-a review
Ding, H., Wu, J., Zhao, W., Matinlinna, J. P., Burrow, M. F., & Tsoi, J. K. H. (2023) · Frontiers in Dental Medicine · doi:10.3389/fdmed.2023.1085251 · Cited by 297 · see citing works
Mazzia, V., Khaliq, A., Salvetti, F., & Chiaberge, M. (2020) · IEEE Access · doi:10.1109/access.2020.2964608 · Cited by 212 · see citing works
Active Convolution: Learning the Shape of Convolution for Image Classification
Jeon, Y., & Kim, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.200 · Cited by 180 · see citing works
Forward Noise Adjustment Scheme for Data Augmentation
Moreno-Barea, F. J., Strazzera, F., Jerez, J. M., Urda, D., & Franco, L. (2018) · 2018 IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2018.8628917 · Cited by 158 · see citing works
Aytekin, C., Ni, X., Cricri, F., & Aksu, E. (2018) · 2018 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489068 · Cited by 107 · see citing works
Blind Hyperspectral Unmixing Using Autoencoders: A Critical Comparison
Palsson, B., Sveinsson, J. R., & Ulfarsson, M. O. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3140154 · Cited by 107 · see citing works
Deep neural networks predicting oil movement in a development unit
Temirchev, P., Simonov, M., Kostoev, R., Burnaev, E., Oseledets, I., Akhmetov, A., Margarit, A., Sitnikov, A., & Koroteev, D. (2020) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2019.106513 · Cited by 75 · see citing works
Masoudnia, S., Mersa, O., Araabi, B. N., Vahabie, A.-H., Sadeghi, M. A., & Ahmadabadi, M. N. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.03.040 · Cited by 57 · see citing works
Roggeveen, L., el Hassouni, A., Ahrendt, J., Guo, T., Fleuren, L., Thoral, P., Girbes, A. R., Hoogendoorn, M., & Elbers, P. W. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.102003 · Cited by 41 · see citing works
Detecting Voice Cloning Attacks via Timbre Watermarking
Liu, C., Zhang, J., Zhang, T., Yang, X., Zhang, W., & Yu, N. (2024) · Proceedings 2024 Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24200 · Cited by 40 · see citing works
GraphCPI: Graph Neural Representation Learning for Compound-Protein Interaction
Quan, Z., Guo, Y., Lin, X., Wang, Z.-J., & Zeng, X. (2019) · 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm47256.2019.8983267 · Cited by 35 · see citing works
Deep Transfer Learning with Graph Neural Network for Sensor-Based Human Activity Recognition
Liao, T., Zhao, J., Liu, Y., Ivanov, K., Xiong, J., & Yan, Y. (2022) · 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995660 · Cited by 36 · see citing works
Toward Real-World Voice Disorder Classification
Kuo, H.-C., Hsieh, Y.-P., Tseng, H.-H., Wang, C.-T., Fang, S.-H., & Tsao, Y. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2023.3270532 · Cited by 10 · see citing works
Deploying Deep Learning Models via IOT Deployment Tools
Krishnamurthi, R., Maheshwari, R., & Gulati, R. (2019) · 2019 Twelfth International Conference on Contemporary Computing (IC3) · doi:10.1109/ic3.2019.8844946 · Cited by 6 · see citing works
- Authors
- Xu, B., Wang, N., Chen, T., & Li, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1505.00853
- OpenAlex
- W1921523184
- Cited by
- 2,264
- References
- 1
- Citations by year
- 2026: 13 · 2025: 108 · 2024: 247 · 2023: 307 · 2022: 270 · 2021: 438 · 2020: 348 · 2019: 289 · 2018: 136 · 2017: 69 · 2016: 32 · 2015: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- leaky ReLU, convolutional neural networks, image classification, overfitting, negative slope, sparsity, CIFAR-100
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
