CoChat
Sign Up
Cited by 2,104Advanced Neural Network ApplicationsStochastic Gradient Optimization TechniquesDomain Adaptation and Few-Shot Learning

Deep Networks with Stochastic Depth

Open access
Huang, Gao, Sun, Yu, Liu, Zhuang, Sedra, Daniel, Weinberger, Kilian Q.·2016·Lecture Notes in Computer Science
DOI 10.1007/978-3-319-46493-0_39
Open this paper in CoChat — free accountOpen PDF

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/978-3-319-46493-0_39

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The improved stochastic depth saves memory and compute in proportion approximately equal to the drop rate.

    “This saves memory and compute in proportion approximately equal to the drop rate, thanks to specific fused kernels.”

    From DINOv2: Learning Robust Visual Features without Supervision (2023) · cites this paper as Huang et al., 2016

    Section 3 (p. 6) says bypassing ResBlocks avoids forward-backward computation and reports approximately 25% training-time savings under its linear-decay setting; Section 4 (p. 10) reports a corresponding measured speedup. The cited paper does not establish proportional memory savings or attribute the savings to fused kernels, so those parts of the citing sentence are unsupported. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Stochastic depth facilitates the convergence of transformers, especially deep ones.

    “We have employed stochastic depth [29], which facilitates the convergence of transformers, especially deep ones [16, 17].”

    From Training data-efficient image transformers & distillation through attention (2020) · cites this paper as [29]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. [2][2] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, IEEE (2009) 248-255 -
  2. [4][4] Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.: Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229 (2013) - · arXiv
  3. [5][5] Simonyan, K., Zisserman, A.: Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556 (2014) - · arXiv
  4. [6][6] Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 (2014) - · arXiv
  5. [7][7] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 1-9 -
  6. [11][11] Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on 5 (2) (1994) 157-166 -
  7. [12][12] Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: International conference on artificial intelligence and statistics. (2010) 249-256 -
  8. [13][13] Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. arXiv preprint arXiv:1409.5185 (2014) - · arXiv
  9. [14][14] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015) - · arXiv
  10. [15][15] Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway networks. arXiv preprint arXiv:1505.00387 (2015) - · arXiv
  11. [16][16] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15 (1) (2014) 1929-1958 -
  12. [18][18] Erhan, D., Bengio, Y., Courville, A., Manzagol, P.A., Vincent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? The Journal of Machine Learning Research 11 (2010) 625-660 -
  13. [19][19] Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10). (2010) 807-814 -
  14. [20][20] Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., Fergus, R.: Regularization of neural networks using dropconnect. In Dasgupta, S., Mcallester, D., eds.: Proceedings of the 30th International Conference on Machine Learning (ICML13). Volume 28., JMLR Workshop and Conference Proceedings (May 2013) 1058-1066 -
  15. [21][21] Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. arXiv preprint arXiv:1302.4389 (2013) - · arXiv
  16. [22][22] Smith, L.N., Hand, E.M., Doster, T.: Gradual dropin of layers to train very deep neural networks. CVPR (2016) -
  17. [24][24] Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013) - · arXiv
  18. [25][25] Graham, B.: Fractional max-pooling. arXiv preprint arXiv:1412.6071 (2014) - · arXiv
  19. [26][26] Agostinelli, F., Hoffman, M., Sadowski, P., Baldi, P.: Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830 (2014) - · arXiv
  20. [27][27] Liang, M., Hu, X.: Recurrent convolutional neural network for object recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 3367-3375 -
  21. [28][28] Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Ali, M., Adams, R.P., et al.: Scalable bayesian optimization using deep neural networks. arXiv preprint arXiv:1502.05700 (2015) - · arXiv
  22. [29][29] Srivastava, R.K., Greff, K., Schmidhuber, J.: Training very deep networks. In: Advances in Neural Information Processing Systems. (2015) 2368-2376 -
  23. [30][30] Lee, C.Y., Gallagher, P.W., Tu, Z.: Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. arXiv preprint arXiv:1509.08985 (2015) - · arXiv
  24. [31][31] Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS workshop on deep learning and unsupervised feature learning. Volume 2011., Granada, Spain (2011) 4 -
  25. [32][32] Collobert, R., Kavukcuoglu, K., Farabet, C.: Torch7: A matlab-like environment for machine learning. In: BigLearn, NIPS Workshop. (2011) -
  26. [33][33] Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: Proceedings of the 30th international conference on machine learning (ICML-13). (2013) 1139-1147 -
  27. [35][35] He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027 (2016) · arXiv

Cited by

Sorted by citations

Authors
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q.
Affiliations
  • Cornell University (US)
  • Tsinghua University (CN)
Corresponding author
Huang, Gao
ORCID
Venue
Lecture Notes in Computer Science
Publisher
Springer International Publishing
Published
2016
Pages
646-661
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
DOI
10.1007/978-3-319-46493-0_39
OpenAlex
W2331143823
Cited by
2,104
References
2
Citations by year
2026: 111 · 2025: 175 · 2024: 197 · 2023: 273 · 2022: 239 · 2021: 308 · 2020: 233 · 2019: 237 · 2018: 179 · 2017: 116 · 2016: 34 · 2015: 1
FWCI
65.44
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Division of Emerging Frontiers in Research and Innovation
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
stochastic depth, ResNet, training time reduction, gradient vanishing, network depth

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review