Improving neural networks by preventing co-adaptation of feature detectors
Open accessAbstract
When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The weighted normalized geometric mean of per-model outputs approximates the arithmetic mean better than the geometric mean, and the quality of this approximation depends on consistency in neuron activation.
“It has been shown though [52], that in dropout-like [53] systems activation units of complete models produce a weighted normalized geometric mean of per-model outputs.”
From ModDrop: Adaptive Multi-Modal Gesture Recognition (2016) · cites this paper as [53]
The main text (p. 2) says that, for a single-hidden-layer network with a softmax output, the mean network is exactly equivalent to taking the geometric mean of the dropout models’ predicted distributions. It does not say that a weighted normalized geometric mean approximates the arithmetic mean better than the geometric mean, or that approximation quality depends on consistency in neuron activation. The discussion on p. 5 instead says that dropout with probability 0.5 gives the models equal importance. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThe state-of-the-art for a fully-connected 784-1200-1200-10 network with dropout regularization (50% for hidden units and 20% for the input) and tanh activations is 107 errors on the MNIST test set.
“Currently, the state-of-the-art for a fully-connected 7821200-1200-10 network with dropout regularization (50% for hidden units and 20% for the input) and tanh activations [53] is 107 errors on the MNIST test set (see Table 7).”
From ModDrop: Adaptive Multi-Modal Gesture Recognition (2016) · cites this paper as [53]
Appendix A.1 (p. 8) lists the 784-1200-1200-10 MNIST architecture and specifies 50% dropout for hidden units and 20% for input units. The main text (p. 2) reports about 110 test errors with those dropout rates, but our copy does not establish exactly 107 errors for that architecture or specify tanh activations. (LLM, conf=0.86)
Experimental — accuracy not yet measuredTo avoid overfitting, YOLO uses dropout and extensive data augmentation.
“To avoid overfitting we use dropout and extensive data augmentation.”
From You Only Look Once: Unified, Real-Time Object Detection (2016) · cites this paper as [18]
The paper states that dropout reduces overfitting (p. 1) and describes using dropout with data augmentation for its ImageNet model (H Models for ImageNet, p. 18). It does not establish that YOLO uses either technique or describe YOLO’s augmentation as extensive. (LLM, conf=0.88)
Experimental — accuracy not yet measuredA dropout layer with rate .5 after the first connected layer prevents co-adaptation between layers.
“A dropout layer with rate = .5 after the first connected layer prevents co-adaptation between layers [18].”
From You Only Look Once: Unified, Real-Time Object Detection (2016) · cites this paper as [18]
The main text (p. 1) says dropout omits hidden units with probability 0.5 to prevent complex co-adaptations. But the discussion of fully connected layers (p. 5) does not establish that placing one dropout layer after the first fully connected layer has that effect; it says dropout in all hidden layers works better than dropout in only one. (LLM, conf=0.87)
Experimental — accuracy not yet measuredDropout randomly sets activations to 0 during training, making the model less sensitive to specific weights.
“Dropout is a recently introduced regularization technique where randomly selected activations are set to 0 during the training, so that the model becomes less sensitive to specific weights in the network [27].”
From Convolutional neural networks: an overview and application in radiology (2018) · cites this paper as [27]
There is a very efficient version of model combination that only costs about a factor of two during training.
“There is, however, a very efficient version of model combination that only costs about a factor of two during training.”
From ImageNet classification with deep convolutional neural networks (2017) · cites this paper as [12]
Dropout reduces complex co-adaptations of neurons because a neuron cannot rely on the presence of particular other neurons.
“This technique reduces complex co adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons.”
From ImageNet classification with deep convolutional neural networks (2017) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1× · see citing works
“Assuming the dropout networks do not all make identical predictions, the prediction of the mean network is guaranteed to assign a higher log probability to the correct answer than the mean of the log probabilities assigned by the individual dropout networks [8].”
BackgroundSelf-citationHigh-dimensional signature compression for large-scale image classification
Sanchez, J., & Perronnin, F. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995504 · Cited by 272 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- [4]4. Y. B. Y. Lecun, L. Bottou, P. Haffner, Proceedings of the IEEE 86 , 2278 (1998).
- [6]6. A. Mohamed, G. Dahl, G. Hinton, IEEE Transactions on Audio, Speech, and Language Processing, 20 , 14 (2012).
- [7]7. G. Dahl, D. Yu, L. Deng, A. Acero, IEEE Transactions on Audio, Speech, and Language Processing, 20 , 30 (2012).
- [10]10. A. Coates, A. Y. Ng, ICML (2011), pp. 921-928.
- [14]14. R. M. Neal, Bayesian Learning for Neural Networks, Lecture Notes in Statistics No. 118 (Springer-Verlag, New York, 1996).
- [19]19. D. D. Lewis, T. G. R. Y. Yang, Journal of Machine Learning 5 , 361 (2004).
Cited by
Sorted by citations
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
“Strong regularization such as maxout [9] or dropout [13] is applied to obtain the best results ([9, 25, 24, 34]) on this dataset.”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Self-citationAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
“Objectives may also have other sources of noise than data subsampling, such as dropout (Hinton et al., 2012b) regularization.”
BackgroundMethodYou Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · see citing works
“5 after the first connected layer prevents co-adaptation between layers [18].”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
“techniques [13, 30, 10, 36], aggressive data augmentation [18, 14, 29, 33], and large-scale data [4, 26].”
BackgroundDeep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
Self-citationWang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · see citing works
“We adopt dropout [22] after the global pooling layer, with a dropout ratio of 0.”
MethodRecent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576 · see citing works
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · see citing works
Object Detection With Deep Learning: A Review
Zhao, Z.-Q., Zheng, P., Xu, S.-T., & Wu, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876865 · Cited by 5,545 · see citing works
Convolutional neural networks: an overview and application in radiology
Yamashita, R., Nishio, M., Do, R. K. G., & Togashi, K. (2018) · Insights into Imaging · doi:10.1007/s13244-018-0639-9 · Cited by 4,682 · see citing works
“…steepestrateofincrease, andall parametersare updated in the negative direction of the gradient with a step size determined based on a learning rate Insights Imaging (2018) 9:611 – 629 set to 0 during the training, so that the model becomes less sensitive to specific weights in the network [27].”
MethodDeep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · see citing works
“Strong regularization such as maxout [9] or dropout [13] is applied to obtain the best results ([9, 25, 24, 34]) on this dataset.”
MethodDeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · see citing works
SlowFast Networks for Video Recognition
Feichtenhofer, C., Fan, H., Malik, J., & He, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00630 · Cited by 4,007 · see citing works
Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Kamnitsas, K., Ledig, C., Newcombe, V. F. J., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., & Glocker, B. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.10.004 · Cited by 3,498 · see citing works
“Brain masks were obtained using the ROBEX tool (Iglesias et al. (2011)). All images were resampled to an isotropic 1mm(3) resolution, with dimensions 193×229×193 and affinely registered (Studholme et al. (1999)) to MNI space using the atlas by Grabner et al.”
MethodInfluentialDeep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503 · see citing works
Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images
Pereira, S., Pinto, A., Alves, V., & Silva, C. A. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2538465 · Cited by 2,735 · see citing works
Knowledge Graph Embedding: A Survey of Approaches and Applications
Wang, Q., Mao, Z., Wang, B., & Guo, L. (2017) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2754499 · Cited by 2,654 · see citing works
Deep learning applications and challenges in big data analytics
Najafabadi, M. M., Villanustre, F., Khoshgoftaar, T. M., Seliya, N., Wald, R., & Muharemagic, E. (2015) · Journal of Big Data · doi:10.1186/s40537-014-0007-7 · Cited by 2,594 · see citing works
Deep learning for visual understanding: A review
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.116 · Cited by 2,504 · see citing works
Apostolopoulos, I. D., & Mpesiana, T. A. (2020) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-020-00865-4 · Cited by 2,338 · see citing works
Neuroscience-Inspired Artificial Intelligence
Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.011 · Cited by 1,557 · see citing works
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Frankle, J., & Carbin, M. (2018) · arXiv · doi:10.48550/arxiv.1803.03635 · Cited by 1,529 · see citing works
Multiscale vision transformers
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00675 · Cited by 1,407 · see citing works
Going Deeper with Convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014) · arXiv · doi:10.48550/arxiv.1409.4842 · Cited by 1,461 · see citing works
Weight Uncertainty in Neural Networks
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1505.05424 · Cited by 1,342 · see citing works
Improving deep neural networks for LVCSR using rectified linear units and dropout
Dahl, G. E., Sainath, T. N., & Hinton, G. E. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639346 · Cited by 1,310 · see citing works
Self-citationReading text in the wild with convolutional neural networks
Jaderberg, M., Simonyan, K., Vedaldi, A., & Zisserman, A. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-015-0823-z · Cited by 1,191 · see citing works
Human activity recognition with smartphone sensors using deep learning neural networks
Ronao, C. A., & Cho, S.-B. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.04.032 · Cited by 1,215 · see citing works
Recent advances and applications of deep learning methods in materials science
Choudhary, K., DeCost, B., Chen, C., Jain, A., Tavazza, F., Cohn, R., Park, C. W., Choudhary, A., Agrawal, A., Billinge, S. J. L., et al. (2022) · npj Computational Materials · doi:10.1038/s41524-022-00734-6 · Cited by 1,172 · see citing works
“Another popular regularization is dropout 59 in which we randomly set the activations for an NN layer to zero during training.”
BackgroundKriegeskorte, N. (2015) · Annual Review of Vision Science · doi:10.1146/annurev-vision-082114-035447 · Cited by 1,093 · see citing works
Guclu, U., & van Gerven, M. A. J. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.5023-14.2015 · Cited by 961 · see citing works
fPINNs: Fractional Physics-Informed Neural Networks
Pang, G., Lu, L., & Karniadakis, G. E. (2019) · SIAM Journal on Scientific Computing · doi:10.1137/18m1229845 · Cited by 1,105 · see citing works
DeepTox: toxicity prediction using deep learning
Mayr, A., Klambauer, G., Unterthiner, T., & Hochreiter, S. (2016) · Frontiers in Environmental Science · doi:10.3389/fenvs.2015.00080 · Cited by 1,081 · see citing works
Data augmentation: A comprehensive survey of modern approaches
Mumuni, A., & Mumuni, F. (2022) · Array · doi:10.1016/j.array.2022.100258 · Cited by 891 · see citing works
A Deep Learning Approach to Universal Image Manipulation Detection Using a New Convolutional Layer
Bayar, B., & Stamm, M. C. (2016) · Proceedings of the 4th ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/2909827.2930786 · Cited by 838 · see citing works
Deep convolutional neural networks for multi-modality isointense infant brain image segmentation
Zhang, W., Li, R., Deng, H., Wang, L., Lin, W., Ji, S., & Shen, D. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.12.061 · Cited by 817 · see citing works
Recent advances in deep learning for speech research at Microsoft
Deng, L., Li, J., Huang, J.-T., Yao, K., Yu, D., Seide, F., Seltzer, M., Zweig, G., He, X., Williams, J., et al. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639345 · Cited by 801 · see citing works
Deep Learning in Medical Ultrasound Analysis: A Review
Liu, S., Wang, Y., Yang, X., Lei, B., Liu, L., Li, S. X., Ni, D., & Wang, T. (2019) · Engineering · doi:10.1016/j.eng.2018.11.020 · Cited by 798 · see citing works
Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition
Jung, H., Lee, S., Yim, J., Park, S., & Kim, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.341 · Cited by 781 · see citing works
Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
Dosovitskiy, A., Fischer, P., Springenberg, J. T., Riedmiller, M., & Brox, T. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2496141 · Cited by 762 · see citing works
Fault diagnosis for rotating machinery using multiple sensors and convolutional neural networks
Xia, M., Li, T., Xu, L., Liu, L., & de Silva, C. W. (2018) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2017.2728371 · Cited by 723 · see citing works
Network Representation Learning: A Survey
Zhang, D., Yin, J., Zhu, X., & Zhang, C. (2020) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2018.2850013 · Cited by 714 · see citing works
Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks
Dou, Q., Chen, H., Yu, L., Zhao, L., Qin, J., Wang, D., Mok, V. C., Shi, L., & Heng, P.-A. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528129 · Cited by 679 · see citing works
Deep learning for molecular design—a review of the state of the art
Elton, D. C., Boukouvalas, Z., Fuge, M. D., & Chung, P. W. (2019) · Molecular Systems Design & Engineering · doi:10.1039/c9me00039a · Cited by 683 · see citing works
Deep Features for Text Spotting
Jaderberg, M., Vedaldi, A., & Zisserman, A. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10593-2_34 · Cited by 653 · see citing works
State-of-charge estimation of lithium-ion batteries based on gated recurrent neural network
Yang, F., Li, W., Li, C., & Miao, Q. (2019) · Energy · doi:10.1016/j.energy.2019.03.059 · Cited by 543 · see citing works
Deep visual tracking: Review and experimental comparison
Li, P., Wang, D., Wang, L., & Lu, H. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.11.007 · Cited by 517 · see citing works
Improving Computer-Aided Detection UsingConvolutional Neural Networks and Random View Aggregation
Roth, H. R., Lu, L., Liu, J., Yao, J., Seff, A., Cherry, K., Kim, L., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2482920 · Cited by 608 · see citing works
“It is a variation of the previously suggested “DropOut” method [29], [30].”
MethodLeveraging uncertainty information from deep neural networks for disease detection
Leibig, C., Allken, V., Ayhan, M. S., Berens, P., & Wahl, S. (2017) · Scientific Reports · doi:10.1038/s41598-017-17876-z · Cited by 510 · see citing works
Data Augmentation Using Random Image Cropping and Patching for Deep CNNs
Takahashi, R., Matsubara, T., & Uehara, K. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2935128 · Cited by 477 · see citing works
Object Detection Networks on Convolutional Feature Maps
Ren, S., He, K., Girshick, R., Zhang, X., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2601099 · Cited by 454 · see citing works
Medical image retrieval using deep convolutional neural network
Qayyum, A., Anwar, S. M., Awais, M., & Majid, M. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.05.025 · Cited by 433 · see citing works
ModDrop: Adaptive Multi-Modal Gesture Recognition
Neverova, N., Wolf, C., Taylor, G., & Nebout, F. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2461544 · Cited by 420 · see citing works
Deep learning and its application to LHC physics
Guest, D., Cranmer, K., & Whiteson, D. (2018) · Annual Review of Nuclear and Particle Science · doi:10.1146/annurev-nucl-101917-021019 · Cited by 417 · see citing works
Training deep neural networks with low precision multiplications
Courbariaux, M., Bengio, Y., & David, J.-P. (2014) · arXiv · doi:10.48550/arxiv.1412.7024 · Cited by 407 · see citing works
Pathology image analysis using segmentation deep learning algorithms
Wang, S., Yang, D. M., Rong, R., Zhan, X., & Xiao, G. (2019) · The American Journal of Pathology · doi:10.1016/j.ajpath.2019.05.007 · Cited by 382 · see citing works
Big Data: Deep Learning for financial sentiment analysis
Sohangir, S., Wang, D., Pomeranets, A., & Khoshgoftaar, T. M. (2018) · Journal of Big Data · doi:10.1186/s40537-017-0111-6 · Cited by 390 · see citing works
Standard Plane Localization in Fetal Ultrasound via Domain Transferred Deep Neural Networks
Chen, H., Ni, D., Qin, J., Li, S., Yang, X., Wang, T., & Heng, P. A. (2015) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2015.2425041 · Cited by 379 · see citing works
Opinion Mining with Deep Recurrent Neural Networks
Irsoy, O., & Cardie, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1080 · Cited by 384 · see citing works
“posed dropout technique (Hinton et al., 2012): we randomly set entries of hidden representations to 0 with a probability called the dropout rate, which is tuned over the development set.”
MethodLong, E., Lin, H., Liu, Z., Wu, X., Wang, L., Jiang, J., An, Y., Lin, Z., Li, X., Chen, J., et al. (2017) · Nature Biomedical Engineering · doi:10.1038/s41551-016-0024 · Cited by 358 · see citing works
Liu, R., Meng, G., Yang, B., Sun, C., & Chen, X. (2017) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2016.2645238 · Cited by 345 · see citing works
Compute Trends Across Three Eras of Machine Learning
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891914 · Cited by 363 · see citing works
“For example, G. Hinton et al. (2012) presents the “shared view” of which at the time were the top 4 Speech Recognition labs.”
BackgroundThe dropout learning algorithm
Baldi, P., & Sadowski, P. (2014) · Artificial Intelligence · doi:10.1016/j.artint.2014.02.004 · Cited by 340 · see citing works
Wei, X., Yang, Z., Liu, Y., Wei, D., Jia, L., & Li, Y. (2019) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2019.01.008 · Cited by 338 · see citing works
Shi, Z., Yao, W., Li, Z., Zeng, L., Zhao, Y., Zhang, R., Tang, Y., & Wen, J. (2020) · Applied Energy · doi:10.1016/j.apenergy.2020.115733 · Cited by 327 · see citing works
Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
Ioannou, Y., Robertson, D., Cipolla, R., & Criminisi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.633 · Cited by 321 · see citing works
Exploiting Argument Information to Improve Event Detection via Supervised Attention Mechanisms
Liu, S., Chen, Y., Liu, K., & Zhao, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1164 · Cited by 250 · see citing works
Integrating Machine Learning with Human Knowledge
Deng, C., Ji, X., Rainey, C., Zhang, J., & Lu, W. (2020) · iScience · doi:10.1016/j.isci.2020.101656 · Cited by 243 · see citing works
Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
Ashmore, R., Calinescu, R., & Paterson, C. (2022) · ACM Computing Surveys · doi:10.1145/3453444 · Cited by 256 · see citing works
Lin, M., Chen, Q., & Yan, S. (2013) · arXiv · doi:10.48550/arxiv.1312.4400 · Cited by 252 · see citing works
CCL: Cross-modal Correlation Learning With Multigrained Fusion by Hierarchical Network
Peng, Y., Qi, J., Huang, X., & Yuan, Y. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2742704 · Cited by 228 · see citing works
Automated defect inspection of LED chip using deep convolutional neural network
Lin, H., Li, B., Wang, X., Shu, Y., & Niu, S. (2019) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-018-1415-x · Cited by 241 · see citing works
Predicting electronic structure properties of transition metal complexes with neural networks
Janet, J. P., & Kulik, H. J. (2017) · Chemical Science · doi:10.1039/c7sc01247k · Cited by 199 · see citing works
3D multi-view convolutional neural networks for lung nodule classification
Kang, G., Liu, K., Hou, B., & Zhang, N. (2017) · PLOS ONE · doi:10.1371/journal.pone.0188290 · Cited by 191 · see citing works
CMU DeepLens: deep learning for automatic image-based galaxy–galaxy strong lens finding
Lanusse, F., Ma, Q., Li, N., Collett, T. E., Li, C.-L., Ravanbakhsh, S., Mandelbaum, R., & Póczos, B. (2018) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stx1665 · Cited by 191 · see citing works
Lee, C. K., Hofer, I., Gabel, E., Baldi, P., & Cannesson, M. (2018) · Anesthesiology · doi:10.1097/aln.0000000000002186 · Cited by 185 · see citing works
Herriott, C., & Spear, A. D. (2020) · Computational Materials Science · doi:10.1016/j.commatsci.2020.109599 · Cited by 187 · see citing works
Ultrasound image-based thyroid nodule automatic segmentation using convolutional neural networks
Ma, J., Wu, F., Jiang, T., Zhao, Q., & Kong, D. (2017) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-017-1649-7 · Cited by 187 · see citing works
Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review
Liu, R., Nageotte, F., Zanne, P., de Mathelin, M., & Dresp-Langley, B. (2021) · Robotics · doi:10.3390/robotics10010022 · Cited by 188 · see citing works
A review and perspective on hybrid modeling methodologies
Schweidtmann, A. M., Zhang, D., & von Stosch, M. (2024) · Digital Chemical Engineering · doi:10.1016/j.dche.2023.100136 · Cited by 178 · see citing works
Improving Neural Machine Translation Models with Monolingual Data
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1009 · Cited by 296 · see citing works
Insider Threat Detection with Deep Neural Network
Yuan, F., Cao, Y., Shang, Y., Liu, Y., Tan, J., & Fang, B. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93698-7_4 · Cited by 168 · see citing works
Twitter Bot Detection Using Bidirectional Long Short-Term Memory Neural Networks and Word Embeddings
Wei, F., & Nguyen, U. T. (2019) · 2019 First IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa48467.2019.00021 · Cited by 168 · see citing works
DeepMood: Modeling Mobile Phone Typing Dynamics for Mood Detection
Cao, B., Zheng, L., Zhang, C., Yu, P. S., Piscitello, A., Zulueta, J., Ajilore, O., Ryan, K., & Leow, A. D. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098086 · Cited by 145 · see citing works
The strong gravitational lens finding challenge
Metcalf, R. B., Meneghetti, M., Avestruz, C., Bellagamba, F., Bom, C. R., Bertin, E., Cabanac, R., Courbin, F., Davies, A., Decencière, E., et al. (2019) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201832797 · Cited by 152 · see citing works
On the number of response regions of deep feed forward networks with piece-wise linear activations
Pascanu, R., Montufar, G., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6098 · Cited by 153 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2017) · arXiv · Cited by 11,689 · see citing works
Classifiers for Ischemic Stroke Lesion Segmentation: A Comparison Study
Maier, O., Schröder, C., Forkert, N. D., Martinetz, T., & Handels, H. (2015) · PLOS ONE · doi:10.1371/journal.pone.0145118 · Cited by 150 · see citing works
Convolutional Neural Network for Multi-Category Rapid Serial Visual Presentation BCI
Manor, R., & Geva, A. B. (2015) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2015.00146 · Cited by 144 · see citing works
Ma, J., Wu, F., Jiang, T., Zhu, J., & Kong, D. (2017) · Medical Physics · doi:10.1002/mp.12134 · Cited by 142 · see citing works
Rashed-Al-Mahfuz, M., Moni, M. A., Uddin, S., Alyami, S. A., Summers, M. A., & Eapen, V. (2021) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2021.3050925 · Cited by 138 · see citing works
Finding strong lenses in CFHTLS using convolutional neural networks
Jacobs, C., Glazebrook, K., Collett, T., More, A., & McCarthy, C. (2017) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stx1492 · Cited by 133 · see citing works
Watch this: Scalable cost-function learning for path planning in urban environments
Wulfmeier, M., Wang, D. Z., & Posner, I. (2016) · 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759328 · Cited by 116 · see citing works
Extracting biological age from biomedical data via deep learning: too much of a good thing?
Pyrkov, T. V., Slipensky, K., Barg, M., Kondrashin, A., Zhurov, B., Zenin, A., Pyatnitskiy, M., Menshikov, L., Markov, S., & Fedichev, P. O. (2018) · Scientific Reports · doi:10.1038/s41598-018-23534-9 · Cited by 111 · see citing works
- Authors
- Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R.
- Affiliations
- University of Toronto (CA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2012
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1207.0580
- OpenAlex
- W1904365287
- Cited by
- 6,382
- References
- 3
- Citations by year
- 2026: 22 · 2025: 171 · 2024: 361 · 2023: 402 · 2022: 392 · 2021: 814 · 2020: 750 · 2019: 795 · 2018: 703 · 2017: 600 · 2016: 511 · 2015: 439 · 2014: 284 · 2013: 125 · 2012: 10
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- dropout, co-adaptation, feature detectors, overfitting, feedforward neural networks, limited training data, generalization, speech recognition, object recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
