Highway Networks
Open accessAbstract
There is plenty of theoretical and empirical evidence that depth of neural networks is a crucial ingredient for their success. However, network training becomes more difficult with increasing depth and training of very deep networks remains an open problem. In this extended abstract, we introduce a new architecture designed to ease gradient-based training of very deep networks. We refer to networks with this architecture as highway networks, since they allow unimpeded information flow across several layers on "information highways". The architecture is characterized by the use of gating units which learn to regulate the flow of information through a network. Highway networks with hundreds of layers can be trained directly using stochastic gradient descent and with a variety of activation functions, opening up the possibility of studying extremely deep and efficient architectures.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In 2015, Srivastava et al. suggested a novel CNN architecture called Highway Network to overcome the issue of difficult training in deeper networks.
“In 2015, Srivastava et al. [105] suggested a novel CNN architecture, called Highway Network, to overcome this issue.”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [105]
DenseNet was presented to solve the problem of the vanishing gradient, following the same direction as ResNet and the Highway network.
“To solve the problem of the vanishing gradient, DenseNet was presented, following the same direction as ResNet and the Highway network [105, 111, 112].”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [105]
Recent advances in deep learning have dramatically improved neural network accuracy.
“Recent advances in deep learning have dramatically improved neural network accuracy (Simonyan & Zisserman, 2015; Srivastava et al., 2015; He et al., 2016; Huang et al., 2016; 2017).”
From On calibration of modern neural networks (2017) · cites this paper as Srivastava et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Cited by 912
Research works printed but not linked: 17
- Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal. Representation learning: A review and new perspectives. Pattern Analysis and Machine Intelligence, IEEE Transactions on , 35(8):1798-1828, 2013. URL http://ieeexplore.ieee.org/ xpls/abs\_all.jsp?arnumber=6472238 .
- Ciresan, Dan, Meier, Ueli, Masci, Jonathan, and Schmidhuber, J¨ urgen. A committee of neural networks for traffic sign classification. In Neural Networks (IJCNN), The 2011 International Joint Conference on , pp. 1918-1921. IEEE, 2011a. URL http://ieeexplore.ieee. org/xpls/abs\_all.jsp?arnumber=6033458 .
- Ciresan, Dan, Meier, Ueli, and Schmidhuber, J¨ urgen. Multi-column deep neural networks for image classification. In IEEE Conference on Computer Vision and Pattern Recognition , 2012.
- Ciresan, DC, Meier, Ueli, Masci, Jonathan, Gambardella, Luca M, and Schmidhuber, J¨ urgen. Flexible, high performance convolutional neural networks for image classification. In IJCAI , 2011b. URL http://www. aaai.org/ocs/index.php/IJCAI/IJCAI11/ paper/download/3098/3425%0020http: //dl.acm.org/citation.cfm?id=2283603 .
- Gers, Felix A., Schmidhuber, J¨ urgen, and Cummins, Fred. Learning to forget: Continual prediction with LSTM. In ICANN , volume 2, pp. 850-855, 1999. URL http://ieeexplore.ieee.org/ xpls/abs\_all.jsp?arnumber=818041 .
- In International Conference on Artificial Intelligence and Statistics , pp. 249-256, 2010. URL http: //machinelearning.wustl.edu/mlpapers/ paper\_files/AISTATS2010\_GlorotB10.pdf .
- H˚ astad, Johan. Computational limitations of small-depth circuits . MIT press, 1987. URL http://dl.acm. org/citation.cfm?id=SERIES9056.27031 .
- H˚ astad, Johan and Goldmann, Mikael. On the power of small-depth threshold circuits. Computational Complexity , 1(2):113-129, 1991. URL http://link.springer.com/article/10. 1007/BF01272517 .
- He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. arXiv:1502.01852 [cs] , February 2015. URL http: //arxiv.org/abs/1502.01852 . · arXiv
- Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems , 2012. URL http://books.nips.cc/ papers/files/nips25/NIPS2012\_0534.pdf .
- Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick, Zhang, Zhengyou, and Tu, Zhuowen. Deeply-supervised nets. pp. 562-570, 2015. URL http://jmlr.org/ proceedings/papers/v38/lee15a.html .
- Montufar, Guido F, Pascanu, Razvan, Cho, Kyunghyun, and Bengio, Yoshua. On the number of linear regions of deep neural networks. In Advances in Neural Information Processing Systems . 2014. URL http://papers.nips.cc/paper/5422on-the-number-of-linear-regions-ofdeep-neural-networks.pdf .
- Romero, Adriana, Ballas, Nicolas, Kahou, Samira Ebrahimi, Chassang, Antoine, Gatta, Carlo, and Bengio, Yoshua. FitNets: Hints for thin deep nets. arXiv:1412.6550 [cs] , December 2014. URL http://arxiv.org/abs/1412.6550 . · arXiv
- Saxe, Andrew M., McClelland, James L., and Ganguli, Surya. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv:1312.6120 [cond-mat, q-bio, stat] , December 2013. URL http://arxiv.org/abs/1312. 6120 . · arXiv
- Simonyan, Karen and Zisserman, Andrew. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 [cs] , September 2014. URL http: //arxiv.org/abs/1409.1556 . · arXiv
- Srivastava, Rupesh Kumar, Masci, Jonathan, Gomez, Faustino, and Schmidhuber, J¨ urgen. Understanding locally competitive networks. In International Conference on Learning Representations , 2015. URL http: //arxiv.org/abs/1410.1165 .
- Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. arXiv:1409.4842 [cs] , September 2014. URL http://arxiv.org/abs/ 1409.4842 . · arXiv
- Authors
- Srivastava, R. K., Greff, K., & Schmidhuber, J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1505.00387
- OpenAlex
- W4394643672
- Cited by
- 912
- References
- 0
- Citations by year
- 2026: 10 · 2025: 16 · 2024: 45 · 2023: 93 · 2022: 106 · 2021: 125 · 2020: 147 · 2019: 157 · 2018: 117 · 2017: 57 · 2016: 33 · 2015: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- highway network, neural network depth, information flow, gradient-based learning, stochastic gradient descent, activation functions, network architecture
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
