Regularization for Deep Learning: A Taxonomy
Open accessThe DOI matches its OpenAlex record; 44 of the 44 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
(2014) · Cited by 33,419
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · cited 2×
“A similar concept of sharing knowledge between tasks is also utilized in meta-learning, where multiple tasks from the same domain are learned sequentially, using previously gained knowledge as bias for new tasks (Baxter, 2000); and transfer learning, where knowledge from one domain is transferred into another domain (…”
BackgroundNeural networks for pattern recognition
(1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,538
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“Model compression (mimic models, distilled models) (Bucilă et al., 2006; Ba and Caruana, 2014; Hinton et al., 2015) p(θ|x,D) Target Deterministic Training”
MethodRepresentation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · cited 1×
“Dilated convolutions (Yu and Koltun, 2015) Layer operation Like convolutional networks.”
BackgroundImproving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382 · cited 6×
“The most frequently used noisy model is Dropout (Hinton et al., 2012; Srivastava et al., 2014).”
BackgroundMethodInfluentialUnderstanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838 · cited 1×
“This is in line with (Zhang et al., 2017) who find in a series of experiments that regularization (such as Dropout, data augmentation, and weight decay) is by itself neither necessary nor sufficient for good generalization.”
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6229 · Cited by 2,680
Why Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Bengio, S. (2010) · Journal of Machine Learning Research · Cited by 2,641
An Overview of Multi-Task Learning in Deep Neural Networks
Ruder, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.05098 · Cited by 2,265
Empirical Evaluation of Rectified Activations in Convolutional Network
Xu, B., Wang, N., Chen, T., & Li, M. (2015) · arXiv · doi:10.48550/arxiv.1505.00853 · Cited by 2,264 · cited 1×
“, 2016b) p(θ) Hidden features Random Training Noisy activation functions (Nair and Hinton, 2010; Xu et al., 2015; Gülçehre et al., 2016a) p(θ|x) Hidden features Random Training”
BackgroundDeep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068 · cited 1×
“Model compression (mimic models, distilled models) (Bucilă et al., 2006; Ba and Caruana, 2014; Hinton et al., 2015) p(θ|x,D) Target Deterministic Training”
Method(2010) · doi:10.5555/1756006.1859921 · Cited by 1,924
A guide to convolution arithmetic for deep learning
Dumoulin, V., & Visin, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.07285 · Cited by 1,172
Understanding Data Augmentation for Classification: When to Warp?
Wong, S. C., Gatt, A., Stamatescu, V., & McDonnell, M. D. (2016) · 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta.2016.7797091 · Cited by 1,007
Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition
Fukushima, K., & Miyake, S. (1982) · Lecture Notes in Biomathematics · doi:10.1007/978-3-642-46466-9_18 · Cited by 991
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Saxe, A., McClelland, J. L., & Ganguli, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6120 · Cited by 931
Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
Zeiler, M. D., & Fergus, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3557 · Cited by 755
Synthesizing Robust Adversarial Examples
Athalye, A., Engstrom, L., Ilyas, A., & Kwok, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.07397 · Cited by 625
Semi-Supervised Learning with Ladder Networks
Rasmus, A., Harri Valpola, B., Honkala, M., Berglund, M., & Raiko, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.02672 · Cited by 527
The Marginal Value of Adaptive Gradient Methods in Machine Learning
Wilson, A., Roelofs, R., Stern, M., Srebro, N., & Recht, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08292 · Cited by 522
The Effects of Adding Noise During Backpropagation Training on a Generalization Performance
An, G. (1996) · Neural Computation · doi:10.1162/neco.1996.8.3.643 · Cited by 475 · cited 5×
“For example, Tangent prop simulates minimal data augmentation (Simard et al., 1992); Injection of small-variance Gaussian noise (Bishop, 1995b; An, 1996) is an approximation of Jacobian penalty (Rifai et al., 2011c); and Fast dropout (Wang and Manning, 2013) is (in shallow networks) a deterministic…”
BackgroundInfluentialGeneralization and Parameter Estimation in Feedforward Nets: Some Experiments
Morgan, N., & Bourlard, H. (1989) · neural information processing systems · Cited by 380
Asymptotic statistical theory of overtraining and cross-validation
Amari, S., Murata, N., Muller, K.-R., Finke, M., & Yang, H. H. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.623200 · Cited by 351
NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles
Lu, J., Sibai, H., Fabry, E., & Forsyth, D. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.03501 · Cited by 186
Dropout as data augmentationPreprint
Bouthillier, X., Konda, K., Vincent, P., & Memisevic, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.08700 · Cited by 174
Data-dependent Initializations of Convolutional Neural Networks
Krähenbühl, P., Doersch, C., Donahue, J., & Darrell, T. J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06856 · Cited by 165
Topology Aware Fully Convolutional Networks for Histology Gland Segmentation
BenTaieb, A., & Hamarneh, G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46723-8_53 · Cited by 163 · cited 1×
“Segmentation with binary potentials (BenTaieb and Hamarneh, 2016) Penalty on anatomically implausible image segmentations.”
BackgroundOnline Batch Selection for Faster Training of Neural Networks
Loshchilov, I., & Hutter, F. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06343 · Cited by 150
Adding noise to the input of a model trained with a regularized objectivePreprint
Rifai, S., Glorot, X., Bengio, Y., & Vincent, P. (2011) · arXiv.org · doi:10.48550/arxiv.1104.3250 · Cited by 82
Gradual DropIn of Layers to Train Very Deep Neural Networks
Smith, L. N., Hand, E., & Doster, T. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.515 · Cited by 34
Deep unsupervised learning through spatial contrastingPreprint
Hoffer, E., Hubara, I., & Ailon, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.00243 · Cited by 33
A Bayesian encourages dropoutPreprint
Maeda, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.7003 · Cited by 32
Morerio, P., Cavazza, J., Volpi, R., Vidal, R., & Murino, V. (2017) · International Conference on Computer Vision · Cited by 32
Adjusting for Dropout Variance in Batch Normalization and Weight InitializationPreprint
Hendrycks, D., & Gimpel, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.02488 · Cited by 24
Invariant backpropagation: how to train a transformation-invariant neural networkPreprint
Demyanov, S., Bailey, J., Ramamohanarao, K., & Leckie, C. A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04434 · Cited by 12
Trivializing The Energy Landscape Of Deep Networks.Preprint
Chaudhari, P., & Soatto, S. (2015) · arXiv (Cornell University) · Cited by 12
Adding Gradient Noise Improves Learning for Very Deep Networks
Neelakantan, A., Vilnis, L., Le, Q. V., Sutskever, I., Kaiser, L., Kurach, K., & Martens, J. (2015) · arXiv · doi:10.48550/arxiv.1511.06807
Hoffer, E., Hubara, I., & Soudry, D. (2017) · arXiv · doi:10.48550/arxiv.1705.08741
Cited by
Albumentations: fast and flexible image augmentations
Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A., Druzhinin, M., & Kalinin, A. A. (2020) · Information · doi:10.3390/info11020125 · Cited by 2,714
Smith, L. N. (2018) · arXiv · doi:10.48550/arxiv.1803.09820 · Cited by 790
“The interested reader can see Kukačka et al. (2017) for a review of regularization methods.”
BackgroundSolving inverse problems using data-driven models
Arridge, S., Maass, P., Öktem, O., & Schönlieb, C.-B. (2019) · Acta Numerica · doi:10.1017/s0962492919000059 · Cited by 686
Data augmentation for deep-learning-based electroencephalography
Lashgari, E., Liang, D., & Maoz, U. (2020) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2020.108885 · Cited by 398
Thin MobileNet: An Enhanced MobileNet Architecture
Sinha, D., & El-Sharkawy, M. (2019) · 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON) · doi:10.1109/uemcon47517.2019.8993089 · Cited by 379
Metadata
- Authors
- Kukačka, J., Golkov, V., & Cremers, D.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2017-10-29
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1710.10686
- OpenAlex
- W2766151966
- Cited by
- 255
- References
- 44
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.