CoChat
Sign Up
PreprintCited by 2,267Neural Networks and ApplicationsMachine Learning and ELMStochastic Gradient Optimization Techniques

The DOI matches its DataCite record.

Self-Normalizing Neural Networks

Open access
Klambauer, Günter, Unterthiner, Thomas, Mayr, Andreas, Hochreiter, Sepp·2017·arXiv
DOI 10.48550/arxiv.1706.02515arXiv 1706.02515
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: github.com/bioinf-jku/SNNs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1706.02515

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SNN has previously been shown to work well in high-dimensional low-sample-size scenarios.

    “we used the Self-Normalizing Network (SNN) which has previously been demonstrated to work well in high-dimensional low-sample size (HDLSS) scenarios (Klambauer et al., 2017).”

    From Pan-cancer integrative histology-genomic analysis via multimodal deep learning (2022) · cites this paper as Klambauer et al., 2017

    The 121 UCI datasets section reports that SNNs significantly outperformed competing feed-forward networks overall. But the paper does not establish that they work particularly well in high-dimensional, low-sample-size settings: its small-dataset results place SNNs behind SVMs and random forests, while its strongest results against other machine-learning methods are for datasets with at least 1,000 data points. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 33

  1. Abramowitz, M. and Stegun, I. (1964). Handbook of Mathematical Functions, volume 55 of Applied Mathematics Series. National Bureau of Standards, 10th edition.
  2. Ba, J. L., Kiros, J. R., and Hinton, G. (2016). Layer normalization. arXiv preprint arXiv:1607.06450. · arXiv
  3. Bengio, Y. (2013). Deep learning of representations: Looking forward. In Proceedings of the First International Conference on Statistical Language and Speech Processing, pages 1-37, Berlin, Heidelberg.
  4. Blinn, J. (1996). Consider the lowly 2×2 matrix. IEEE Computer Graphics and Applications, pages 82-88.
  5. Bradley, R. C. (1981). Central limit theorems under weak dependence. Journal of Multivariate Analysis, 11(1):1-16.
  6. Cireşan, D. and Meier, U. (2015). Multi-column deep neural networks for offline handwritten chinese character classification. In 2015 International Joint Conference on Neural Networks (IJCNN), pages 1-6. IEEE.
  7. Clevert, D.-A., Unterthiner, T., and Hochreiter, S. (2015). Fast and accurate deep network learning by exponential linear units (ELUs). 5th International Conference on Learning Representations, arXiv:1511.07289. · arXiv
  8. Dugan, P., Clark, C., LeCun, Y., and Van Parijs, S. (2016). Phase 4: Dcl system using deep learning approaches for land-based or ship-based real-time recognition and localization of marine mammals-distributed processing and big data applications. arXiv preprint arXiv:1605.00982. · arXiv
  9. Esteva, A., Kuprel, B., Novoa, R., Ko, J., Swetter, S., Blau, H., and Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639):115-118.
  10. Fernández-Delgado, M., Cernadas, E., Barro, S., and Amorim, D. (2014). Do we need hundreds of classifiers to solve real world classification problems. Journal of Machine Learning Research, 15(1):3133-3181.
  11. Goldberg, D. (1991). What every computer scientist should know about floating-point arithmetic. ACM Comput. Surv., 223(1):5-48.
  12. Graves, A., Mohamed, A., and Hinton, G. (2013). Speech recognition with deep recurrent neural networks. In IEEE International conference on acoustics, speech and signal processing (ICASSP), pages 6645-6649.
  13. Graves, A. and Schmidhuber, J. (2009). Offline handwriting recognition with multidimensional recurrent neural networks. In Advances in neural information processing systems, pages 545-552.
  14. Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 316(22):2402-2410.
  15. Harrison, J. (1999). A machine-checked theory of floating point arithmetic. In Bertot, Y., Dowek, G., Hirschowitz, A., Paulin, C., and Théry, L., editors, Theorem Proving in Higher Order Logics: 12th International Conference, TPHOLs'99, volume 1690 of Lecture Notes in Computer Science, pages 113-130. Springer-Verlag.
  16. He, K., Zhang, X., Ren, S., and Sun, J. (2015a). Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  17. He, K., Zhang, X., Ren, S., and Sun, J. (2015b). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1026-1034.
  18. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8):1735-1780.
  19. Huval, B., Wang, T., Tandon, S., et al. (2015). An empirical evaluation of deep learning on highway driving. arXiv preprint arXiv:1504.01716. · arXiv
  20. Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of The 32nd International Conference on Machine Learning, pages 448-456.
  21. Korolev, V. and Shevtsova, I. (2012). An improvement of the Berry-Esseen inequality with applications to Poisson and mixed Poisson random sums. Scandinavian Actuarial Journal, 2012(2):81-105.
  22. Krizhevsky, A., Sutskever, I., and Hinton, G. (2012). Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems, pages 1097-1105.
  23. LeCun, Y. and Bengio, Y. (1995). Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995.
  24. LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature, 521(7553):436-444.
  25. Loosemore, S., Stallman, R. M., McGrath, R., Oram, A., and Drepper, U. (2016). The GNU C Library: Application Fundamentals. GNU Press, Free Software Foundation, 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA, 2.24 edition.
  26. Lyon, R., Stappers, B., Cooper, S., Brooke, J., and Knowles, J. (2016). Fifty years of pulsar candidate selection: From simple filters to a new principled real-time classification approach. Monthly Notices of the Royal Astronomical Society, 459(1):1104-1123.
  27. Ren, C. and MacKenzie, A. R. (2007). Closed-form approximations to the error and comple- mentary error functions and their applications in atmospheric science. Atmos. Sci. Let., pages 70-73.
  28. Sak, H., Senior, A., Rao, K., and Beaufays, F. (2015). Fast and accurate recurrent neural network acoustic models for speech recognition. arXiv preprint arXiv:1507.06947. · arXiv
  29. Salimans, T. and Kingma, D. P. (2016). Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pages 901-909.
  30. Silver, D., Huang, A., Maddison, C., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489.
  31. Srivastava, R. K., Greff, K., and Schmidhuber, J. (2015). Training very deep networks. In Advances in Neural Information Processing Systems, pages 2377-2385.
  32. Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104-3112.
  33. Wainberg, M., Alipanahi, B., and Frey, B. J. (2016). Are random forests truly the best classifiers? Journal of Machine Learning Research, 17(110):1-5.

Cited by

Sorted by citations

Metadata

Authors
Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S.
Affiliations
Johannes Kepler University of Linz (AT)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1706.02515
arXiv
1706.02515
OpenAlex
W2624413595
Cited by
2,267
References
0
Citations by year
2026: 39 · 2025: 158 · 2024: 243 · 2023: 298 · 2022: 273 · 2021: 352 · 2020: 336 · 2019: 300 · 2018: 226 · 2017: 42
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Nvidia
  • Horizon 2020 Framework Programme (671555)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and ELM
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
Keywords
self-normalizing neural networks, Banach fixed point theorem, vanishing and exploding gradients, regularization, robust learning, drug discovery, Tox21, astronomical data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review