The dropout learning algorithm
Open accessAbstract
Dropout is a recently introduced algorithm for training neural network by randomly dropping units during training to prevent their co-adaptation. A mathematical analysis of some of the static and dynamic properties of dropout is provided using Bernoulli gating variables, general enough to accommodate dropout on units or connections, and with variable rates. The framework allows a complete analysis of the ensemble averaging properties of dropout in linear networks, which is useful to understand the non-linear case. The ensemble averaging properties of dropout in non-linear logistic networks result from three fundamental equations: (1) the approximation of the expectations of logistic functions by normalized geometric means, for which bounds and estimates are derived; (2) the algebraic equality between normalized geometric means of logistic functions with the logistic of the means, which mathematically characterizes logistic functions; and (3) the linearity of the means with respect to sums, as well as products of independent variables. The results are also extended to other classes of transfer functions, including rectified linear functions. Approximation errors tend to cancel each other and do not accumulate. Dropout can also be connected to stochastic neurons and used to predict firing rates, and to backpropagation by viewing the backward propagation as ensemble averaging in a dropout linear network. Moreover, the convergence properties of dropout can be understood in terms of stochastic gradient descent. Finally, for the regularization properties of dropout, the expectation of the dropout gradient is the gradient of the corresponding approximation ensemble, regularized by an adaptive weight decay term with a propensity for self-consistent variance minimization and sparse representations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The regularization effect of ModDrop is adaptive to the magnitude of the weights: it is strong in the mid-range of weights, plays a less significant role when weights are small, and gradually weakens with saturation.
“As a result, it is strong in the mid-range of weights, plays a less significant role when weights are small and gradually weakens with saturation.”
From ModDrop: Adaptive Multi-Modal Gesture Recognition (2016) · cites this paper as [52]
Section 11.4 describes dropout regularization as adaptive and says its sigmoid-derivative component approaches zero near saturation. It does not discuss ModDrop or state that regularization is strongest for mid-range weights and less significant for small weights. The section also says regularization is maximal at the beginning of learning, so the claimed weight-range pattern is not established by this source. (LLM, conf=0.78)
Experimental — accuracy not yet measuredThe weighted normalized geometric mean of per-model outputs approximates the arithmetic mean better than the geometric mean, and the quality of this approximation depends on consistency in neuron activation.
“It has been shown though [52], that in dropout-like [53] systems activation units of complete models produce a weighted normalized geometric mean of per-model outputs.”
From ModDrop: Adaptive Multi-Modal Gesture Recognition (2016) · cites this paper as [52]
The multiplier proportional to the derivative of the sigmoid activation makes the regularization effect adaptive to the magnitude of the weights.
“Finally, as has been shown by [52] for dropout, the multiplier proportional to the derivative of the sigmoid activation makes the regularization effect adaptive to the magnitude of the weights.”
From ModDrop: Adaptive Multi-Modal Gesture Recognition (2016) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601 · see citing works
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · see citing works
Seel, N. M. (2012) · Springer eBooks · doi:10.1007/978-1-4419-1428-6_5195 · Cited by 19,525 · see citing works
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · see citing works
Russell, E. L., Chiang, L. H., & Braatz, R. D. (2000) · Advances in industrial control · doi:10.1007/978-1-4471-0409-4_3 · Cited by 7,938 · see citing works
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · see citing works
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382 · see citing works
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691 · see citing works
Spencer, J. (1991) · Medical Entomology and Zoology · Cited by 4,370 · see citing works
Cox, D. R. (2008) · Springer eBooks · doi:10.1007/978-0-387-32833-1_5 · Cited by 2,678 · see citing works
Plackett, R. L., & Cox, D. R. (1971) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344247 · Cited by 2,273 · see citing works
Adaptive Algorithms and Stochastic Approximations
Benveniste, A., Métivier, M., & Priouret, P. (1990) · Springer eBooks · doi:10.1007/978-3-642-75894-2 · Cited by 2,045 · see citing works
Probability and computing: randomized algorithms and probabilistic analysis
Mitzenmacher, M., & Upfal, E. (2005) · CERN Document Server (European Organization for Nuclear Research) · Cited by 1,892 · see citing works
Proceedings of the 29th International Conference on Machine Learning (ICML-12)Preprint
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,687 · see citing works
Neural networks and principal component analysis: Learning from examples without local minima
Baldi, P., & Hornik, K. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90014-2 · Cited by 1,383 · see citing works
Self-citationTraining with Noise is Equivalent to Tikhonov Regularization
Bishop, C. M. (1995) · Neural Computation · doi:10.1162/neco.1995.7.1.108 · Cited by 1,325 · see citing works
A circuit for detection of interaural time differences in the brain stem of the barn owl
Carr, C., & Konishi, M. (1990) · Journal of Neuroscience · doi:10.1523/jneurosci.10-10-03227.1990 · Cited by 821 · see citing works
Advanced Lectures on Machine Learning
Bousquet, O., Rätsch, G., & Luxburg, U. (2004) · Lecture notes in computer science · doi:10.1007/b100712 · Cited by 565 · see citing works
Baldi, P., & Sadowski, P. (2013) · neural information processing systems · Cited by 532 · see citing works
Self-citationSequential updating of conditional probabilities on directed graphical structures
Spiegelhalter, D. J., & Lauritzen, S. L. (1990) · Networks · doi:10.1002/net.3230200507 · Cited by 521 · see citing works
The Effects of Adding Noise During Backpropagation Training on a Generalization Performance
An, G. (1996) · Neural Computation · doi:10.1162/neco.1996.8.3.643 · Cited by 475 · see citing works
A Convergence Theorem for Non Negative Almost Supermartingales and Some Applications
Robbins, H. E., & Siegmund, D. (1985) · Springer eBooks · doi:10.1007/978-1-4612-5110-1_10 · Cited by 428 · see citing works
Probability and Computing: Randomized Algorithms and Probabilistic Analysis
Buot, M. (2006) · Journal of the American Statistical Association · doi:10.1198/jasa.2006.s74 · Cited by 421 · see citing works
Axonal delay lines for time measurement in the owl's brainstem.
Carr, C. E., & Konishi, M. (1988) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.85.21.8311 · Cited by 323 · see citing works
Adaptive dropout for training deep neural networks
Ba, J. L., & Frey, B. J. (2013) · neural information processing systems · Cited by 290 · see citing works
Noise injection into inputs in back-propagation learning
Matsuoka, K. (1992) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/21.155944 · Cited by 265 · see citing works
Dropout Training as Adaptive RegularizationPreprint
Wager, S., Wang, S., & Liang, P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.1493 · Cited by 260 · see citing works
Learning in linear neural networks: a survey
Baldi, P., & Hornik, K. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.392248 · Cited by 252 · see citing works
Self-citationRobbins, H. E., & Siegmund, D. (1971) · Elsevier eBooks · doi:10.1016/b978-0-12-604550-5.50015-8 · Cited by 251 · see citing works
Diaconis, P. W. (1988) · Springer eBooks · doi:10.1007/978-1-4613-8768-8_20 · Cited by 244 · see citing works
Bottou, L. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-28650-9_7 · Cited by 201 · see citing works
Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training
Murray, A. F., & Edwards, P. J. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.317730 · Cited by 201 · see citing works
Bootstrapping with Noise: An Effective Regularization Technique
Raviv, Y., & Intrator, N. (1996) · Connection Science · doi:10.1080/095400996116811 · Cited by 188 · see citing works
(2013) · Cited by 131 · see citing works
A logistic approximation to the cumulative normal distribution
Bowling, S. R., Khasawneh, M. T., Kaewkuekool, S., & Cho, B. R. (2009) · Journal of Industrial Engineering and Management · doi:10.3926/jiem.2009.v2n1.p114-127 · Cited by 121 · see citing works
Harnischfeger, G., Neuweiler, G., & Schlegel, P. (1985) · Journal of Neurophysiology · doi:10.1152/jn.1985.53.1.89 · Cited by 115 · see citing works
A logistic approximation to the cumulative normal distribution
Bowling, S. R., Khasawneh, M. T., Kaewkuekool, S., & Cho, B. R. (2009) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.3926/jiem..v2n1.p114-127 · Cited by 91 · see citing works
Generalization of an inequality of Ky Fan
Levinson, N. (1964) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(64)90089-7 · Cited by 84 · see citing works
A refinement of the arithmetic mean-geometric mean inequality
Cartwright, D. I., & Field, M. J. (1978) · Proceedings of the American Mathematical Society · doi:10.1090/s0002-9939-1978-0476971-2 · Cited by 81 · see citing works
A stochastic version of the delta rule
Hanson, S. J. (1990) · Physica D Nonlinear Phenomena · doi:10.1016/0167-2789(90)90081-y · Cited by 64 · see citing works
Self-improvement of the inequality between arithmetic and geometric means
Aldaz, J. M. (2009) · Journal of Mathematical Inequalities · doi:10.7153/jmi-03-21 · Cited by 27 · see citing works
A New Refinement of the Arithmetic Mean Geometric Mean Inequality
Alzer, H. (1997) · Rocky Mountain Journal of Mathematics · doi:10.1216/rmjm/1181071887 · Cited by 24 · see citing works
Refined Arithmetic, Geometric and Harmonic Mean Inequalities
Mercer, P. R. (2003) · Rocky Mountain Journal of Mathematics · doi:10.1216/rmjm/1181075474 · Cited by 21 · see citing works
On the Ky Fan inequality and related inequalities II
Neuman, E., & Sándor, J. (2005) · Bulletin of the Australian Mathematical Society · doi:10.1017/s0004972700034894 · Cited by 19 · see citing works
Noise modulation of synaptic weights in a biological neural network
Gardner, D. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90016-6 · Cited by 11 · see citing works
Sharp bounds for the difference between the arithmetic and geometric means
Aldaz, J. M. (2012) · Archiv der Mathematik · doi:10.1007/s00013-012-0434-7 · Cited by 10 · see citing works
On the Ky Fan inequality and related inequalities I
Neuman, E., & Sándor, J. (2002) · Mathematical Inequalities & Applications · doi:10.7153/mia-05-06 · Cited by 8 · see citing works
Some inequalities for arithmetic and geometric means
Alzer, H. (1999) · Proceedings of the Royal Society of Edinburgh Section A Mathematics · doi:10.1017/s0308210500021326 · Cited by 6 · see citing works
Flagged first, then by citations
2 of the 50 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- [11]Beckenbach EF Bellman R Inequalities 1965 Springer-Verlag Berlin
- [23]Duda RO Hart PE Stork DG 2000 Wiley New York, NY Second Edition.
- [31]Mercer AM Improved upper and lower bounds for the difference an- gn. Journal of Mathematics 2001 31 2
- [39]Rockafellar RT Convex analysis 1997 28 Princeton University Press
Cited by
Sorted by citations
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503 · see citing works
Shen, C. (2018) · Water Resources Research · doi:10.1029/2018wr022643 · Cited by 1,476 · see citing works
Searching for exotic particles in high-energy physics with deep learning
Baldi, P., Sadowski, P., & Whiteson, D. (2014) · Nature Communications · doi:10.1038/ncomms5308 · Cited by 1,164 · see citing works
Self-citationDeepARG: a deep learning approach for predicting antibiotic resistance genes from metagenomic data
Arango-Argoty, G., Garner, E., Pruden, A. J., Heath, L. S., Vikesland, P. J., & Zhang, L. (2018) · Microbiome · doi:10.1186/s40168-018-0401-z · Cited by 981 · see citing works
Urban, G., Tripathi, P. V., Alkayali, T., Mittal, M., Jalali, F., Karnes, W. E., & Baldi, P. (2018) · Gastroenterology · doi:10.1053/j.gastro.2018.06.037 · Cited by 746 · see citing works
Self-citationBaldi, P., & Sadowski, P. (2013) · neural information processing systems · Cited by 532 · see citing works
Self-citationModDrop: Adaptive Multi-Modal Gesture Recognition
Neverova, N., Wolf, C., Taylor, G., & Nebout, F. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2461544 · Cited by 420 · see citing works
Magnan, C. N., & Baldi, P. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu352 · Cited by 363 · see citing works
Self-citationVisual attention methods in deep learning: An in-depth survey
Hassanin, M., Anwar, S., Radwan, I., Khan, F. S., & Mian, A. S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102417 · Cited by 345 · see citing works
Towards dropout training for convolutional neural networks
Wu, H., & Gu, X. (2015) · Neural Networks · doi:10.1016/j.neunet.2015.07.007 · Cited by 315 · see citing works
Wang, T., Huang, J., Zhang, H., & Sun, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01077 · Cited by 291 · see citing works
Zhou, Y., Chang, F., Chang, L., Kao, I.-F., & Wang, Y.-S. (2018) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2018.10.243 · Cited by 286 · see citing works
Detecting Cardiovascular Disease from Mammograms With Deep Learning
Wang, J., Ding, H. J., Bidgoli, F. A., Zhou, B., Iribarren, C., Molloi, S. Y., & Baldi, P. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2655486 · Cited by 234 · see citing works
Self-citationMusallam, A. S., Sobhy, A., & Hussein, M. K. (2022) · IEEE Access · doi:10.1109/access.2022.3140289 · Cited by 243 · see citing works
Deconfounded Video Moment Retrieval with Causal Intervention
Yang, X., Feng, F., Ji, W., Wang, M., & Chua, T. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462823 · Cited by 210 · see citing works
Bakkouri, I., & Afdel, K. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-07988-1 · Cited by 208 · see citing works
Deep learning for chemical reaction prediction
Fooshee, D. R., Mood, A. D., Gutman, E. S., Tavakoli, M., Urban, G., Liu, F., Huynh, N. O., Vranken, D. L. V., & Baldi, P. (2017) · Molecular Systems Design & Engineering · doi:10.1039/c7me00107j · Cited by 205 · see citing works
Self-citationLee, C. K., Hofer, I., Gabel, E., Baldi, P., & Cannesson, M. (2018) · Anesthesiology · doi:10.1097/aln.0000000000002186 · Cited by 185 · see citing works
Self-citationInterventional Few-Shot LearningPreprint
Yue, Z., Zhang, H., Sun, Q., & Hua, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13000 · Cited by 187 · see citing works
Jet flavor classification in high-energy physics with deep neural networks
Guest, D., Collado, J., Baldi, P., Hsu, S.-C., Urban, G., & Whiteson, D. (2016) · Physical review. D/Physical review. D. · doi:10.1103/physrevd.94.112002 · Cited by 177 · see citing works
Self-citationDiversified Regularization Enhanced Training for Effective Manipulator Calibration
Li, Z., Li, S., Bamasag, O. O., Alhothali, A., & Luo, X. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3153039 · Cited by 171 · see citing works
BG-3DM2F: Bidirectional gated 3D multi-scale feature fusion for Alzheimer’s disease diagnosis
Bakkouri, I., Afdel, K., Benois‐Pineau, J., & Catheline, G. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-12242-2 · Cited by 142 · see citing works
A multi-resolution approach for spinal metastasis detection using deep Siamese neural networks
Wang, J., Fang, Z., Lang, N., Yuan, H., Su, M., & Baldi, P. (2017) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2017.03.024 · Cited by 139 · see citing works
Self-citationAdvancing Non-Negative Latent Factorization of Tensors With Diversified Regularization Schemes
Wu, H., Luo, X., & Zhou, M. (2020) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2020.2988760 · Cited by 137 · see citing works
A review of critical challenges in MI-BCI: From conventional to deep learning methods
Khademi, Z., Ebrahimi, F., & Kordy, H. M. (2022) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2022.109736 · Cited by 137 · see citing works
A Fortran-Keras Deep Learning Bridge for Scientific Computing
Ott, J., Pritchard, M., Best, N., Linstead, E. J., Curcic, M., & Baldi, P. (2020) · Scientific Programming · doi:10.1155/2020/8888811 · Cited by 135 · see citing works
Self-citationApplication of deep convolutional neural network on feature extraction and detection of wood defects
He, T., Liu, Y., Yu, Y., Zhao, Q., & Hu, Z. (2019) · Measurement · doi:10.1016/j.measurement.2019.107357 · Cited by 131 · see citing works
Deconfounded Image Captioning: A Causal Retrospect
Yang, X., Zhang, H., & Cai, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3121705 · Cited by 130 · see citing works
Machine-learning methods for stream water temperature prediction
Feigl, M., Lebiedzinski, K., Herrnegger, M., & Schulz, K. (2021) · Hydrology and earth system sciences · doi:10.5194/hess-25-2951-2021 · Cited by 134 · see citing works
Deep learning in biomedical data science
Baldi, P. (2018) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-080917-013343 · Cited by 119 · see citing works
Tactile object recognition using deep learning and dropout
Schmitz, A., Bansho, Y., Noda, K., Iwata, H., Ogata, T., & Sugano, S. (2014) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2014.7041493 · Cited by 117 · see citing works
CLCDSA: Cross Language Code Clone Detection using Syntactical Features and API Documentation
Nafi, K. W., Kar, T. S., Roy, B., Roy, C. K., & Schneider, K. A. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00099 · Cited by 112 · see citing works
Zhang, J., Zhu, L., Yao, L., Ding, X., Chen, D., Wu, H., Lu, Z., Zhou, W., Zhang, L., An, P., et al. (2020) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2020.04.071 · Cited by 110 · see citing works
Zhao, Y., Ding, B., Zhang, Y., Yang, L., & Hao, X. (2021) · ISA Transactions · doi:10.1016/j.isatra.2021.01.058 · Cited by 110 · see citing works
Lang, N., Zhang, Y., Zhang, E., Zhang, J., Chow, D. S., Chang, P., Yu, H. J., Yuan, H., & Su, M. (2019) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2019.02.013 · Cited by 109 · see citing works
Monte Carlo Dropout for Uncertainty Estimation and Motor Imagery Classification
Hermosilla, D. M., Codorniú, R. T., Baracaldo, R. L., Zamora, R. S., Delisle-Rodríguez, D., Mayor, J. J. V., & Álvarez, J. R. N. (2021) · Sensors · doi:10.3390/s21217241 · Cited by 108 · see citing works
What time is it? Deep learning approaches for circadian rhythms
Agostinelli, F., Ceglia, N., Shahbaba, B., Sassone‐Corsi, P., & Baldi, P. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw243 · Cited by 107 · see citing works
Self-citationEnhanced Higgs Boson to τ+τ− Search with Deep Learning
Baldi, P., Sadowski, P., & Whiteson, D. (2015) · Physical Review Letters · doi:10.1103/physrevlett.114.111801 · Cited by 105 · see citing works
Self-citationShort-term wind speed forecasting framework based on stacked denoising auto-encoders with rough ANN
Jahangir, H., Golkar, M. A., Alhameli, F., Mazouz, A., Ahmadian, A., & Elkamel, A. (2020) · Sustainable Energy Technologies and Assessments · doi:10.1016/j.seta.2019.100601 · Cited by 101 · see citing works
DeePred-BBB: A Blood Brain Barrier Permeability Prediction Model With Improved Accuracy
Kumar, R., Sharma, A., Alexiou, A. T., Bilgrami, A. L., Kamal, M. A., & Ashraf, G. M. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.858126 · Cited by 98 · see citing works
Ling, T., Wu, L., Fu, Y., Xu, Q., An, P., Zhang, J., Hu, S., Chen, Y., He, X., Wang, J., et al. (2020) · Endoscopy · doi:10.1055/a-1229-0920 · Cited by 92 · see citing works
Xu, M., Zhou, W., Wu, L., Zhang, J., Wang, J., Mu, G., Huang, X., Li, Y., Yuan, J., Zeng, Z. R., et al. (2021) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2021.03.013 · Cited by 96 · see citing works
Piotrowski, A., Napiorkowski, J. J., & Piotrowska, A. E. (2019) · Earth-Science Reviews · doi:10.1016/j.earscirev.2019.103076 · Cited by 81 · see citing works
Shen, X., Tian, X., Liu, T., Xu, F., & Tao, D. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2750679 · Cited by 88 · see citing works
Wu, L., Xu, M., Jiang, X., He, X., Zhang, H., Ai, Y., Tong, Q.-Y., Lv, P., Lu, B., Guo, M., et al. (2021) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2021.09.017 · Cited by 80 · see citing works
Hofer, I. S., Lee, C., Gabel, E., Baldi, P., & Cannesson, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0248-0 · Cited by 77 · see citing works
Self-citationA theory of local learning, the learning channel, and the optimality of backpropagation
Baldi, P., & Sadowski, P. (2016) · Neural Networks · doi:10.1016/j.neunet.2016.07.006 · Cited by 80 · see citing works
Self-citationNiu, D., Wu, F., Dai, S., He, S., & Wu, B. (2020) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2020.125187 · Cited by 75 · see citing works
Learning in the machine: Random backpropagation and the deep learning channel
Baldi, P., Sadowski, P., & Lu, Z. (2018) · Artificial Intelligence · doi:10.1016/j.artint.2018.03.003 · Cited by 69 · see citing works
Self-citationDropout Rademacher complexity of deep neural networks
Gao, W., & Zhou, Z. (2016) · Science China Information Sciences · doi:10.1007/s11432-015-5470-z · Cited by 66 · see citing works
Machine learning action parameters in lattice quantum chromodynamics
Shanahan, P. E., Trewartha, D., & Detmold, W. (2018) · Physical Review D · doi:10.1103/physrevd.97.094506 · Cited by 65 · see citing works
Rajaraman, S., Sornapudi, S., Alderson, P. O., Folio, L. R., & Antani, S. (2020) · PLoS ONE · doi:10.1371/journal.pone.0242301 · Cited by 62 · see citing works
Classifying shoulder implants in X-ray images using deep learning
Urban, G., Porhemmat, S., Stark, M., Feeley, B. T., Okada, K., & Baldi, P. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.04.005 · Cited by 61 · see citing works
Self-citationLee, C. K., Samad, M., Hofer, I. S., Cannesson, M. P., & Baldi, P. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00377-1 · Cited by 61 · see citing works
Self-citationErshoff, B. D., Lee, C. K., Wray, C. L., Agopian, V. G., Urban, G., Baldi, P., & Cannesson, M. P. (2020) · Transplantation Proceedings · doi:10.1016/j.transproceed.2019.10.019 · Cited by 57 · see citing works
Self-citationPlant species recognition methods using leaf image: Overview
Zhang, S., Huang, W., Huang, Y., & Zhang, C. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.09.113 · Cited by 54 · see citing works
Sparse Simultaneous Recurrent Deep Learning for Robust Facial Expression Recognition
Alam, M., Vidyaratne, L., & Iftekharuddin, K. M. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2776248 · Cited by 52 · see citing works
Approximate Bayesian neural networks in genomic prediction
Waldmann, P. (2018) · Genetics Selection Evolution · doi:10.1186/s12711-018-0439-1 · Cited by 52 · see citing works
Fu, Z., Li, X., Wang, Z., Li, Z., Liu, X., Wu, X., Zhao, J., Ding, X., Wan, X., Zhong, F., et al. (2020) · Organic Chemistry Frontiers · doi:10.1039/d0qo00544d · Cited by 52 · see citing works
Piotrowski, A., Osuch, M., & Napiorkowski, J. J. (2021) · Journal of Hydrology · doi:10.1016/j.jhydrol.2021.126629 · Cited by 52 · see citing works
Al-Sharman, M., Murdoch, D., Cao, D., Lv, C., Zweiri, Y., Rayside, D., & Melek, W. (2021) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2020.1003474 · Cited by 51 · see citing works
A multi-head neural network with unsymmetrical constraints for remaining useful life prediction
Liu, Z., Liu, H., Jia, W., Zhang, D., & Tan, J. (2021) · Advanced Engineering Informatics · doi:10.1016/j.aei.2021.101396 · Cited by 49 · see citing works
Yao, L., Zhang, J., Liu, J., Zhu, L., Ding, X., Chen, D., Wu, H., Lu, Z., Zhou, W., Zhang, L., et al. (2021) · EBioMedicine · doi:10.1016/j.ebiom.2021.103238 · Cited by 48 · see citing works
Vision-and-Language Navigation via Causal Learning
Wang, L., He, Z., Dang, R., Shen, M., Liu, C., & Chen, Q. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01248 · Cited by 51 · see citing works
Zheng, Z., Fu, J., Lu, C., & Zhu, Y. (2021) · Measurement · doi:10.1016/j.measurement.2021.109285 · Cited by 47 · see citing works
On the Inductive Bias of DropoutPreprint
Helmbold, D. P., & Long, P. M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.4736 · Cited by 47 · see citing works
Kumar, V., Ramesh, K. V., & Rakesh, V. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05005-5 · Cited by 46 · see citing works
Masci, J., Rodolà, E., Boscaini, D., Bronstein, M. M., & Li, H. (2016) · SIGGRAPH Asia Courses · doi:10.1145/2988458.2988485 · Cited by 46 · see citing works
Global-connected network with generalized ReLU activation
Chen, Z., & Ho, P. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.07.006 · Cited by 46 · see citing works
Characterization and Identification of Lysine Succinylation Sites based on Deep Learning Method
Huang, K., Hsu, J. B., & Lee, T.-Y. (2019) · Scientific Reports · doi:10.1038/s41598-019-52552-4 · Cited by 44 · see citing works
Hssayni, E. H., Joudar, N., & Ettaouil, M. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109567 · Cited by 45 · see citing works
Pan, J., You, W., Lü, X., Wang, S., You, Z., & Sun, Y. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.06.014 · Cited by 42 · see citing works
Mid-Level Feature Representation via Sparse Autoencoder for Remotely Sensed Scene Classification
Li, E., Du, P., Samat, A., Meng, Y., & Che, M. (2016) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2016.2621011 · Cited by 41 · see citing works
Analyzing fusion of regularization techniques in the deep learning‐based intrusion detection system
Thakkar, A., & Lohiya, R. (2021) · International Journal of Intelligent Systems · doi:10.1002/int.22590 · Cited by 41 · see citing works
Convergence and objective functions of noise-injected multilayer perceptrons with hidden multipliers
Wang, X., Wang, J., Zhang, K., Lin, F., & Chang, Q. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.03.119 · Cited by 40 · see citing works
Ensemble dropout extreme learning machine via fuzzy integral for data classification
Zhai, J., Zang, L., & Zhou, Z. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.09.047 · Cited by 40 · see citing works
Qin, B., Chung, F.-L., & Wang, S. (2019) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2019.2907497 · Cited by 40 · see citing works
Multi-Step-Ahead Time Series Prediction Method with Stacking LSTM Neural Network
Wang, X., & Zhang, Y. (2020) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd49809.2020.9137492 · Cited by 40 · see citing works
Tavakoli, M., Mood, A. D., Vranken, D. L. V., & Baldi, P. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.1c01400 · Cited by 41 · see citing works
Self-citationConvolutional Neural Network for Segmentation and Measurement of Intima Media Thickness
Sudha, S., B, J. K., Rajasekaran, C., Madian, N., & Sunder, T. (2018) · Journal of Medical Systems · doi:10.1007/s10916-018-1001-y · Cited by 39 · see citing works
Lu, Z., Xu, Y., Yao, L., Zhou, W., Gong, W., Yang, G., Guo, M., Zhang, B., Huang, X., He, C., et al. (2021) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2021.11.049 · Cited by 39 · see citing works
Dropout with Expectation-linear RegularizationPreprint
Ma, X., Gao, Y., Hu, Z., Yu, Y., Deng, Y., & Hovy, E. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08017 · Cited by 38 · see citing works
Sun, Y., Pang, S., & Zhang, Y. (2024) · Geoenergy Science and Engineering · doi:10.1016/j.geoen.2024.213015 · Cited by 37 · see citing works
Yu, H., Zhang, Z., Yang, K., Zhang, L., Wang, W., Yang, S., Li, J., & Liu, X. (2023) · Journal of Energy Storage · doi:10.1016/j.est.2023.108915 · Cited by 37 · see citing works
Improved energy reconstruction in NOvA with regression convolutional neural networks
Baldi, P., Bian, J. M., Hertel, L., & Li, L. (2019) · Physical review. D/Physical review. D. · doi:10.1103/physrevd.99.012011 · Cited by 36 · see citing works
Self-citationHuang, G., Shen, Q., Zhang, G., Wang, P., & Yu, Z. (2021) · BioMed Research International · doi:10.1155/2021/9923112 · Cited by 38 · see citing works
Altitude Training: Strong Bounds for Single-Layer DropoutPreprint
Wager, S., Fithian, W., Wang, S. I., & Liang, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.3289 · Cited by 36 · see citing works
Toa, C. K., Sim, K. S., & Tan, S. C. (2021) · IEEE Access · doi:10.1109/access.2021.3072731 · Cited by 35 · see citing works
Deep Sequential Neural NetworkPreprint
Denoyer, L., & Gallinari, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0510 · Cited by 35 · see citing works
Feng, S., Yufei, Z., Yujie, W., Shanyong, W., & Ruilang, C. (2020) · Engineering Geology · doi:10.1016/j.enggeo.2020.105529 · Cited by 34 · see citing works
Leingang, O., Riedl, S., Mai, J., Reiter, G. S., Faustmann, G., Fuchs, P., Scholl, H. P. N., Sivaprasad, S., Rueckert, D., Lotery, A. J., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-46626-7 · Cited by 31 · see citing works
Xue, D., Qian, S., & Xu, C. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3398012 · Cited by 31 · see citing works
Transfer-Learning Approach for Enhanced Brain Tumor Classification in MRI Imaging
Amarnath, A., Bataineh, A. A., & Hansen, J. A. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4030095 · Cited by 31 · see citing works
Prediction of multiproject resource conflict risk via an artificial neural network
Bai, L., Wang, Z., Wang, H., Huang, N., & Shi, H. (2020) · Engineering Construction & Architectural Management · doi:10.1108/ecam-03-2020-0201 · Cited by 30 · see citing works
Galapon, A., Thummerer, A., Langendijk, J. A., Wagenaar, D., & Both, S. (2023) · Medical Physics · doi:10.1002/mp.16838 · Cited by 29 · see citing works
Learning in the machine: The symmetries of the deep learning channel
Baldi, P., Sadowski, P., & Lu, Z. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.08.008 · Cited by 29 · see citing works
Self-citationBellin, N., Calzolari, M., Callegari, E., Bonilauri, P., Grisendi, A., Dottori, M., & Rossi, V. (2021) · Infection Genetics and Evolution · doi:10.1016/j.meegid.2021.105034 · Cited by 29 · see citing works
Deep neural networks for human microRNA precursor detection
Zheng, X., Fu, X., Wang, K., & Wang, M. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-3339-7 · Cited by 28 · see citing works
- Authors
- Baldi, P., & Sadowski, P.
- Affiliations
- University of California, Irvine (US)
- Corresponding author
- Baldi, Pierre
- Venue
- Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 2014-05
- Volume
- 210
- Pages
- 78-122
- ISSN
- 0004-3702 (ISSN-L)
- PMC
- PMC3996711
- OpenAlex
- W2164370980
- Cited by
- 340
- References
- 50
- Citations by year
- 2026: 12 · 2025: 23 · 2024: 39 · 2023: 38 · 2022: 19 · 2021: 40 · 2020: 45 · 2019: 31 · 2018: 26 · 2017: 21 · 2016: 24 · 2015: 10 · 2014: 11 · 2013: 1
- FWCI
- 21.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Artificial Intelligence (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by-nc-nd)
- Funders
- NSF (IIS-0513376, NSF-IIS-1321053)
- NIH (LM010235)
- NIH NLM (T15 LM07443)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- dropout, neural network training, ensemble averaging, logistic function, rectified linear units, stochastic neurons, stochastic gradient descent, regularization, weight decay, sparse representation, variance minimization, co-adaptation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
