The DOI matches its Crossref record; 48 of the 50 references it cites were found in a registry or an index.
The dropout learning algorithm
Open accessAbstract
Dropout is a recently introduced algorithm for training neural network by randomly dropping units during training to prevent their co-adaptation. A mathematical analysis of some of the static and dynamic properties of dropout is provided using Bernoulli gating variables, general enough to accommodate dropout on units or connections, and with variable rates. The framework allows a complete analysis of the ensemble averaging properties of dropout in linear networks, which is useful to understand the non-linear case. The ensemble averaging properties of dropout in non-linear logistic networks result from three fundamental equations: (1) the approximation of the expectations of logistic functions by normalized geometric means, for which bounds and estimates are derived; (2) the algebraic equality between normalized geometric means of logistic functions with the logistic of the means, which mathematically characterizes logistic functions; and (3) the linearity of the means with respect to sums, as well as products of independent variables. The results are also extended to other classes of transfer functions, including rectified linear functions. Approximation errors tend to cancel each other and do not accumulate. Dropout can also be connected to stochastic neurons and used to predict firing rates, and to backpropagation by viewing the backward propagation as ensemble averaging in a dropout linear network. Moreover, the convergence properties of dropout can be understood in terms of stochastic gradient descent. Finally, for the regularization properties of dropout, the expectation of the dropout gradient is the gradient of the corresponding approximation ensemble, regularized by an adaptive weight decay term with a propensity for self-consistent variance minimization and sparse representations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, it is strong in the mid-range of weights, plays a less significant role when weights are small and gradually weakens with saturation.”
From ModDrop: Adaptive Multi-Modal Gesture Recognition (2016) · cites this paper as [52]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It has been shown though [52], that in dropout-like [53] systems activation units of complete models produce a weighted normalized geometric mean of per-model outputs.”
From ModDrop: Adaptive Multi-Modal Gesture Recognition (2016) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“Finally, as has been shown by [52] for dropout, the multiplier proportional to the derivative of the sigmoid activation makes the regularization effect adaptive to the magnitude of the weights.”
From ModDrop: Adaptive Multi-Modal Gesture Recognition (2016) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Seel, N. M. (2012) · Springer eBooks · doi:10.1007/978-1-4419-1428-6_5195 · Cited by 19,482
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
Russell, E. L., Chiang, L. H., & Braatz, R. D. (2000) · Advances in industrial control · doi:10.1007/978-1-4471-0409-4_3 · Cited by 7,938
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691
Spencer, J. (1991) · Medical Entomology and Zoology · Cited by 4,310
Cox, D. R. (2008) · Springer eBooks · doi:10.1007/978-0-387-32833-1_5 · Cited by 2,687
Plackett, R. L., & Cox, D. R. (1971) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344247 · Cited by 2,274
Adaptive Algorithms and Stochastic Approximations
Benveniste, A., Métivier, M., & Priouret, P. (1990) · Springer eBooks · doi:10.1007/978-3-642-75894-2 · Cited by 2,045
Probability and computing: randomized algorithms and probabilistic analysis
Mitzenmacher, M., & Upfal, E. (2005) · CERN Document Server (European Organization for Nuclear Research) · Cited by 1,892
Proceedings of the 29th International Conference on Machine Learning (ICML-12)Preprint
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,687
Neural networks and principal component analysis: Learning from examples without local minima
Baldi, P., & Hornik, K. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90014-2 · Cited by 1,383
Self-citationTraining with Noise is Equivalent to Tikhonov Regularization
Bishop, C. M. (1995) · Neural Computation · doi:10.1162/neco.1995.7.1.108 · Cited by 1,325
A circuit for detection of interaural time differences in the brain stem of the barn owl
Carr, C., & Konishi, M. (1990) · Journal of Neuroscience · doi:10.1523/jneurosci.10-10-03227.1990 · Cited by 819
Advanced Lectures on Machine Learning
Bousquet, O., Rätsch, G., & Luxburg, U. (2004) · Lecture notes in computer science · doi:10.1007/b100712 · Cited by 585
Baldi, P., & Sadowski, P. (2013) · neural information processing systems · Cited by 532
Self-citationSequential updating of conditional probabilities on directed graphical structures
Spiegelhalter, D. J., & Lauritzen, S. L. (1990) · Networks · doi:10.1002/net.3230200507 · Cited by 521
Dropout Training as Adaptive RegularizationPreprint
Wager, S., Wang, S. I., & Liang, P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.1493 · Cited by 498
The Effects of Adding Noise During Backpropagation Training on a Generalization Performance
An, G. (1996) · Neural Computation · doi:10.1162/neco.1996.8.3.643 · Cited by 475
A Convergence Theorem for Non Negative Almost Supermartingales and Some Applications
Robbins, H. E., & Siegmund, D. (1985) · Springer eBooks · doi:10.1007/978-1-4612-5110-1_10 · Cited by 428
Probability and Computing: Randomized Algorithms and Probabilistic Analysis
Buot, M. (2006) · Journal of the American Statistical Association · doi:10.1198/jasa.2006.s74 · Cited by 421
Axonal delay lines for time measurement in the owl's brainstem.
Carr, C. E., & Konishi, M. (1988) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.85.21.8311 · Cited by 323
Adaptive dropout for training deep neural networks
Ba, J. L., & Frey, B. J. (2013) · neural information processing systems · Cited by 290
Noise injection into inputs in back-propagation learning
Matsuoka, K. (1992) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/21.155944 · Cited by 265
Learning in linear neural networks: a survey
Baldi, P., & Hornik, K. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.392248 · Cited by 252
Self-citationRobbins, H. E., & Siegmund, D. (1971) · Elsevier eBooks · doi:10.1016/b978-0-12-604550-5.50015-8 · Cited by 251
Diaconis, P. W. (1988) · Springer eBooks · doi:10.1007/978-1-4613-8768-8_20 · Cited by 244
Bottou, L. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-28650-9_7 · Cited by 218
Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training
Murray, A. F., & Edwards, P. J. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.317730 · Cited by 201
Bootstrapping with Noise: An Effective Regularization Technique
Raviv, Y., & Intrator, N. (1996) · Connection Science · doi:10.1080/095400996116811 · Cited by 188
(2013) · Cited by 131
A logistic approximation to the cumulative normal distribution
Bowling, S. R., Khasawneh, M. T., Kaewkuekool, S., & Cho, B. R. (2009) · Journal of Industrial Engineering and Management · doi:10.3926/jiem.2009.v2n1.p114-127 · Cited by 121
Harnischfeger, G., Neuweiler, G., & Schlegel, P. (1985) · Journal of Neurophysiology · doi:10.1152/jn.1985.53.1.89 · Cited by 115
A logistic approximation to the cumulative normal distribution
Bowling, S. R., Khasawneh, M. T., Kaewkuekool, S., & Cho, B. R. (2009) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.3926/jiem..v2n1.p114-127 · Cited by 91
Generalization of an inequality of Ky Fan
Levinson, N. (1964) · Journal of Mathematical Analysis and Applications · doi:10.1016/0022-247x(64)90089-7 · Cited by 84
A refinement of the arithmetic mean-geometric mean inequality
Cartwright, D. I., & Field, M. J. (1978) · Proceedings of the American Mathematical Society · doi:10.1090/s0002-9939-1978-0476971-2 · Cited by 81
A stochastic version of the delta rule
Hanson, S. J. (1990) · Physica D Nonlinear Phenomena · doi:10.1016/0167-2789(90)90081-y · Cited by 65
Self-improvement of the inequality between arithmetic and geometric means
Aldaz, J. M. (2009) · Journal of Mathematical Inequalities · doi:10.7153/jmi-03-21 · Cited by 27
A New Refinement of the Arithmetic Mean Geometric Mean Inequality
Alzer, H. (1997) · Rocky Mountain Journal of Mathematics · doi:10.1216/rmjm/1181071887 · Cited by 24
Refined Arithmetic, Geometric and Harmonic Mean Inequalities
Mercer, P. R. (2003) · Rocky Mountain Journal of Mathematics · doi:10.1216/rmjm/1181075474 · Cited by 21
On the Ky Fan inequality and related inequalities II
Neuman, E., & Sándor, J. (2005) · Bulletin of the Australian Mathematical Society · doi:10.1017/s0004972700034894 · Cited by 19
Noise modulation of synaptic weights in a biological neural network
Gardner, D. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90016-6 · Cited by 11
Sharp bounds for the difference between the arithmetic and geometric means
Aldaz, J. M. (2012) · Archiv der Mathematik · doi:10.1007/s00013-012-0434-7 · Cited by 10
On the Ky Fan inequality and related inequalities I
Neuman, E., & Sándor, J. (2002) · Mathematical Inequalities & Applications · doi:10.7153/mia-05-06 · Cited by 8
Some inequalities for arithmetic and geometric means
Alzer, H. (1999) · Proceedings of the Royal Society of Edinburgh Section A Mathematics · doi:10.1017/s0308210500021326 · Cited by 6
Flagged first, then by citations
2 of the 50 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Mercer, 2001, Improved upper and lower bounds for the difference an-gn, J. Math., 31
Cited by
Sorted by citations
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503
Shen, C. (2018) · Water Resources Research · doi:10.1029/2018wr022643 · Cited by 1,522
Searching for exotic particles in high-energy physics with deep learning
Baldi, P., Sadowski, P., & Whiteson, D. (2014) · Nature Communications · doi:10.1038/ncomms5308 · Cited by 1,261
Self-citation
Metadata
- Authors
- Baldi, P., & Sadowski, P.
- Affiliations
- University of California, Irvine (US)
- Corresponding author
- Baldi, Pierre
- Venue
- Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 2014-05
- Volume
- 210
- Pages
- 78-122
- ISSN
- 0004-3702 (ISSN-L)
- PMC
- PMC3996711
- OpenAlex
- W2164370980
- Cited by
- 340
- References
- 50
- Citations by year
- 2026: 12 · 2025: 23 · 2024: 39 · 2023: 38 · 2022: 19 · 2021: 40 · 2020: 45 · 2019: 31 · 2018: 26 · 2017: 21 · 2016: 24 · 2015: 10 · 2014: 11
- FWCI
- 21.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Artificial Intelligence (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by-nc-nd)
- Funders
- NSF (IIS-0513376, NSF-IIS-1321053)
- NIH (LM010235)
- NIH NLM (T15 LM07443)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- dropout, neural network training, ensemble averaging, logistic function, rectified linear units, stochastic neurons, stochastic gradient descent, regularization, weight decay, sparse representation, variance minimization, co-adaptation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
