The DOI matches its DataCite record; 7 of the 7 references it cites were found in a registry or an index.
Improving neural networks by preventing co-adaptation of feature detectors
Open accessAbstract
When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has been shown though [52], that in dropout-like [53] systems activation units of complete models produce a weighted normalized geometric mean of per-model outputs.”
From ModDrop: Adaptive Multi-Modal Gesture Recognition (2016) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Currently, the state-of-the-art for a fully-connected 7821200-1200-10 network with dropout regularization (50% for hidden units and 20% for the input) and tanh activations [53] is 107 errors on the MNIST test set (see Table 7).”
From ModDrop: Adaptive Multi-Modal Gesture Recognition (2016) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To avoid overfitting we use dropout and extensive data augmentation.”
From You Only Look Once: Unified, Real-Time Object Detection (2016) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A dropout layer with rate = .5 after the first connected layer prevents co-adaptation between layers [18].”
From You Only Look Once: Unified, Real-Time Object Detection (2016) · cites this paper as [18]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Dropout is a recently introduced regularization technique where randomly selected activations are set to 0 during the training, so that the model becomes less sensitive to specific weights in the network [27].”
From Convolutional neural networks: an overview and application in radiology (2018) · cites this paper as [27]
The cited source says what the claim attributes to it.
In the citing paper
“There is, however, a very efficient version of model combination that only costs about a factor of two during training.”
From ImageNet classification with deep convolutional neural networks (2017) · cites this paper as [12]
The cited source says what the claim attributes to it.
In the citing paper
“This technique reduces complex co adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons.”
From ImageNet classification with deep convolutional neural networks (2017) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Self-citationBreiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1×
“Assuming the dropout networks do not all make identical predictions, the prediction of the mean network is guaranteed to assign a higher log probability to the correct answer than the mean of the log probabilities assigned by the individual dropout networks [8].”
BackgroundSelf-citationHigh-dimensional signature compression for large-scale image classification
Sanchez, J., & Perronnin, F. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995504 · Cited by 272
Research works printed but not linked: 6
- [4]4. Y. B. Y. Lecun, L. Bottou, P. Haffner, Proceedings of the IEEE 86 , 2278 (1998).
- [6]6. A. Mohamed, G. Dahl, G. Hinton, IEEE Transactions on Audio, Speech, and Language Processing, 20 , 14 (2012).
- [7]7. G. Dahl, D. Yu, L. Deng, A. Acero, IEEE Transactions on Audio, Speech, and Language Processing, 20 , 30 (2012).
- [10]10. A. Coates, A. Y. Ng, ICML (2011), pp. 921-928.
- [14]14. R. M. Neal, Bayesian Learning for Neural Networks, Lecture Notes in Statistics No. 118 (Springer-Verlag, New York, 1996).
- [19]19. D. D. Lewis, T. G. R. Y. Yang, Journal of Machine Learning 5 , 361 (2004).
Cited by
Sorted by citations
Items 101–143 of 143
Convolutional Neural Network for Multi-Category Rapid Serial Visual Presentation BCI
Manor, R., & Geva, A. B. (2015) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2015.00146 · Cited by 144
Ma, J., Wu, F., Jiang, T., Zhu, J., & Kong, D. (2017) · Medical Physics · doi:10.1002/mp.12134 · Cited by 142
Rashed-Al-Mahfuz, M., Moni, M. A., Uddin, S., Alyami, S. A., Summers, M. A., & Eapen, V. (2021) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2021.3050925 · Cited by 138
Finding strong lenses in CFHTLS using convolutional neural networks
Jacobs, C., Glazebrook, K., Collett, T., More, A., & McCarthy, C. (2017) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stx1492 · Cited by 133
Watch this: Scalable cost-function learning for path planning in urban environments
Wulfmeier, M., Wang, D. Z., & Posner, I. (2016) · 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759328 · Cited by 116
Extracting biological age from biomedical data via deep learning: too much of a good thing?
Pyrkov, T. V., Slipensky, K., Barg, M., Kondrashin, A., Zhurov, B., Zenin, A., Pyatnitskiy, M., Menshikov, L., Markov, S., & Fedichev, P. O. (2018) · Scientific Reports · doi:10.1038/s41598-018-23534-9 · Cited by 111
Image Segmentation with Cascaded Hierarchical Models and Logistic Disjunctive Normal Networks
Seyedhosseini, M., Sajjadi, M., & Tasdizen, T. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.269 · Cited by 95
Khunte, A., Sangha, V., Oikonomou, E. K., Dhingra, L. S., Aminorroaya, A., Mortazavi, B. J., Coppi, A., Brandt, C. A., Krumholz, H. M., & Khera, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00869-w · Cited by 95
An Animal Detection Pipeline for Identification
Parham, J., Stewart, C., Crall, J., Rubenstein, D., Holmberg, J., & Berger-Wolf, T. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00123 · Cited by 103
An integrative deep learning framework for classifying molecular subtypes of breast cancer
Mohaiminul Islam, M., Huang, S., Ajwad, R., Chi, C., Wang, Y., & Hu, P. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.08.005 · Cited by 90
Online Fall Detection Using Recurrent Neural Networks on Smart Wearable Devices
Musci, M., De Martini, D., Blago, N., Facchinetti, T., & Piastra, M. (2021) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2020.3027454 · Cited by 128
Mariani, S., Rendu, Q., Urbani, M., & Sbarufatti, C. (2021) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2021.107748 · Cited by 86
Amini, M., Pedram, M. M., Moradi, A., & Ouchani, M. (2021) · Computational and Mathematical Methods in Medicine · doi:10.1155/2021/5514839 · Cited by 84
QuipuNet: Convolutional Neural Network for Single-Molecule Nanopore Sensing
Misiunas, K., Ermann, N., & Keyser, U. F. (2018) · Nano Letters · doi:10.1021/acs.nanolett.8b01709 · Cited by 82
A Survey of Deep Learning Applications to Autonomous Vehicle Control
Kuutti, S., Bowden, R., Jin, Y., Barber, P., & Fallah, S. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2962338 · Cited by 96
Lu, N., Wu, G., Zhang, Z., Zheng, Y., Ren, Y., & Choo, K. R. (2020) · Concurrency and Computation: Practice and Experience · doi:10.1002/cpe.5627 · Cited by 76
Less-Forgetful Learning for Domain Expansion in Deep Neural Networks
Jung, H., Ju, J., Jung, M., & Kim, J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11769 · Cited by 75
Mohamed, E., Sirlantzis, K., & Howells, G. (2022) · Displays · doi:10.1016/j.displa.2022.102239 · Cited by 67
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2023) · arXiv · doi:10.48550/arxiv.2312.10549 · Cited by 54
Sung, J., Jin, S. H., & Saxena, A. (2018) · Springer Proceedings in Advanced Robotics · doi:10.1007/978-3-319-60916-4_40 · Cited by 52
“tuned by minimizing the negative log-likelihood with the stochastic gradient method with mini-batches: NLL = − ∑|D| i=0 log(P (Y = y |x,W, b)) To prevent over-fitting to the training data, we used dropout [17], which randomly drops a specified percentage of the output of every layer.”
MethodIyer, J. S., Juyal, D., Le, Q., Shanis, Z., Pokkalla, H., Pouryahya, M., Pedawi, A., Stanford-Moore, S. A., Biddle-Snead, C., Carrasco-Zevallos, O., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03172-7 · Cited by 54
Contrastive Learning Models for Sentence Representations
Xu, L., Xie, H., Li, Z., Wang, F. L., Wang, W., & Li, Q. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3593590 · Cited by 50
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2024) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2024.3966 · Cited by 54
A Taxonomy and Library for Visualizing Learned Features in Convolutional Neural Networks
Grün, F., Rupprecht, C., Navab, N., & Tombari, F. (2016) · arXiv · doi:10.48550/arxiv.1606.07757 · Cited by 49
Modeling multimodal cues in a deep learning-based framework for emotion recognition in the wild
Pini, S., Ahmed, O. B., Cornia, M., Baraldi, L., Cucchiara, R., & Huet, B. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143006 · Cited by 44
On the Limit of English Conversational Speech Recognition
Tüske, Z., Saon, G., & Kingsbury, B. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-211 · Cited by 41
Learning Weight Uncertainty with Stochastic Gradient MCMC for Shape Classification
Li, C., Stevens, A., Chen, C., Pu, Y., Gan, Z., & Carin, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.611 · Cited by 39
Sensor-data augmentation for human activity recognition with time-warping and data masking
Jeong, C. Y., Shin, H. C., & Kim, M. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-10600-0 · Cited by 38
On the Challenges of Physical Implementations of RBMs
Dumoulin, V., Goodfellow, I., Courville, A., & Bengio, Y. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8924 · Cited by 46
ARPNet: Antidepressant Response Prediction Network for Major Depressive Disorder
Chang, B., Choi, Y., Jeon, M., Lee, J., Han, K.-M., Kim, A., Ham, B.-J., & Kang, J. (2019) · Genes · doi:10.3390/genes10110907 · Cited by 35
Single-trial EEG RSVP classification using convolutional neural networks
Shamwell, J., Lee, H., Kwon, H., Marathe, A. R., Lawhern, V., & Nothwang, W. (2016) · SPIE Proceedings · doi:10.1117/12.2224172 · Cited by 34
Highly Accurate Mandarin Tone Classification In The Absence of Pitch Information
Ryant, N., Slaney, M., Liberman, M., Shriberg, E., & Yuan, J. (2014) · Speech Prosody 2014 · doi:10.21437/speechprosody.2014-123 · Cited by 32
Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?
Millidge, B., Salvatori, T., Song, Y., Bogacz, R., & Lukasiewicz, T. (2022) · arXiv · doi:10.48550/arxiv.2202.09467 · Cited by 31
TraMiner: vision-based analysis of locomotion traces for cognitive assessment in smart-homes
Zolfaghari, S., Khodabandehloo, E., & Riboni, D. (2022) · Cognitive Computation · doi:10.1007/s12559-020-09816-3 · Cited by 29
Surfel convolutional neural network for support detection in additive manufacturing
Huang, J., Kwok, T.-H., Zhou, C., & Xu, W. (2019) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-019-03792-1 · Cited by 26
Convolutional neural networks for multi-topic dialog state tracking,
Shi, H., Ushio, T., Endo, M., Yamagami, K., & Horii, N. (2017) · Lecture Notes in Electrical Engineering · doi:10.1007/978-981-10-2585-3_37 · Cited by 24
ChiNet: Deep recurrent convolutional learning for multimodal spacecraft pose estimation
Rondao, D., Aouf, N., & Richardson, M. A. (2022) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2022.3193085 · Cited by 24
Neural graph personalized ranking for Top-N Recommendation
Hu, Z., Wang, J., Yan, Y., Zhao, P., Chen, J., & Huang, J. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106426 · Cited by 21
A Bayesian Deep Learning Approach to Near‐Term Climate Prediction
Luo, X., Nadiga, B. T., Park, J. H., Ren, Y., Xu, W., & Yoo, S. (2022) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2022ms003058 · Cited by 18
Automated segmentation of insect anatomy from micro‐CT images using deep learning
Toulkeridou, E., Gutierrez, C. E., Baum, D., Doya, K., & Economo, E. P. (2023) · Natural Sciences · doi:10.1002/ntls.20230010 · Cited by 13
Heuristic Intrusion Detection Based on Traffic Flow Statistical Analysis
Szczepanik, W., & Niemiec, M. (2022) · Energies · doi:10.3390/en15113951 · Cited by 9
The benefit of the doubt: Uncertainty aware sensing for edge computing platforms
Qendro, L., Chauhan, J., Ramos, A. G. C. P., & Mascolo, C. (2021) · arXiv · doi:10.48550/arxiv.2102.05956 · Cited by 6
Improved regularization of convolutional neural networks with cutout
DeVries, T., & Taylor, G. W. (2017) · arXiv · doi:10.48550/arxiv.1708.04552 · Cited by 2,776
Metadata
- Authors
- Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R.
- Affiliations
- University of Toronto (CA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2012
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1207.0580
- OpenAlex
- W1904365287
- Cited by
- 6,382
- References
- 7
- Citations by year
- 2026: 22 · 2025: 171 · 2024: 361 · 2023: 402 · 2022: 392 · 2021: 814 · 2020: 750 · 2019: 795 · 2018: 703 · 2017: 600 · 2016: 511 · 2015: 439 · 2014: 284 · 2013: 125 · 2012: 10
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- dropout, co-adaptation, feature detectors, overfitting, feedforward neural networks, limited training data, generalization, speech recognition, object recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
