MaxDropout: Deep Neural Network Regularization Based on Maximum Output Values
Abstract
Different techniques have emerged in the deep learning scenario, such as Convolutional Neural Networks, Deep Belief Networks, and Long Short-Term Memory Networks, to cite a few. In lockstep, regularization methods, which aim to prevent overfitting by penalizing the weight connections, or turning off some units, have been widely studied either. In this paper, we present a novel approach called MaxDropout, a regularizer for deep neural network models that works in a supervised fashion by removing (shutting off) the prominent neurons (i.e., most active) in each hidden layer. The model forces fewer activated units to learn more representative information, thus providing sparsity. Regarding the experiments, we show that it is possible to improve existing neural networks and provide better results in neural networks when Dropout is replaced by MaxDropout. The proposed method was evaluated in image classification, achieving comparable results to existing regularizers, such as Cutout and RandomErasing, also improving the accuracy of neural networks that uses Dropout by replacing the existing layer by MaxDropout.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
(2014) · Cited by 33,419 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741 · see citing works
Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2662206 · Cited by 8,983 · see citing works
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244 · see citing works
Random Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7000 · Cited by 3,617 · see citing works
DeepPose: Human Pose Estimation via Deep Neural Networks
Toshev, A., & Szegedy, C. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.214 · Cited by 3,288 · see citing works
Improved regularization of convolutional neural networks with cutout
DeVries, T., & Taylor, G. W. (2017) · arXiv · doi:10.48550/arxiv.1708.04552 · Cited by 2,776 · see citing works
Variational Dropout and the Local Reparameterization TrickPreprint
Kingma, D. P., Salimans, T., & Welling, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.02557 · Cited by 708 · see citing works
Variational Dropout Sparsifies Deep Neural NetworksPreprint
Molchanov, D., Ashukha, A., & Vetrov, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.05369 · Cited by 612 · see citing works
Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise RelationsPreprint
Chen, X., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.3399 · Cited by 352 · see citing works
(2013) · Cited by 326 · see citing works
ImageNet pre-trained models with batch normalizationPreprint
Simon, M., Rodner, E., & Denzler, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.01452 · Cited by 157 · see citing works
Gated recurrent convolution neural network for OCR
Wang, J., & Hu, X. (2017) · neural information processing systems · Cited by 146 · see citing works
Gal, Y., Hron, J., & Kendall, A. (2017) · neural information processing systems · Cited by 137 · see citing works
Exploring deep learning based solutions in fine grained activity recognition in the wild
Cao, S., & Nevatia, R. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7899664 · Cited by 27 · see citing works
Adaptable deep learning structures for object labeling/tracking under dynamic visual environments
Doulamis, N. D. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-017-5349-7 · Cited by 19 · see citing works
Srivastava (2014) · doi:10.5555/2627435.2670313
Flagged first, then by citations
Cited by
Sorted by citations
Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks
Santos, C. F. G. D., & Papa, J. P. (2022) · ACM Computing Surveys · doi:10.1145/3510413 · Cited by 415 · see citing works
“3.99 ± 0.13 21.96 ± 0.24 - - + RandomErasing [77] 4.31 ± 0.07 24.03 ± 0.19 - - + MaxDropout [46] 4.66 ± 0.14 21.93 ± 0.07 - - + MaxDropout + Cutout [46] 3.76 ± 0.08 21.82 ± 0.13 - - + Mixup [71] 4.2 21.1 - - + MM [60] 2.95 ± 0.04 20.34 ± 0.52 - - + TSLA [65] - 21.45 ± 0.28 - - + TargetDrop [78]…”
BackgroundMethodInfluentialSelf-citation
- Authors
- do Santos, C. F. G., Colombo, D., Roder, M., & Papa, J. P.
- Affiliations
- Petrobras (Brazil) (BR)
- Universidade Estadual Paulista (Unesp) (BR)
- Venue
- 2020 25th International Conference on Pattern Recognition (ICPR)
- Publisher
- IEEE
- Published
- 2021-01-10
- Pages
- 2671-2676
- ISSN
- 1041-3278 (ISSN-L)
- OpenAlex
- W3045276399
- Semantic Scholar
- eb7788d3bdc0e3fbb342f99131f1452e623343d0
- Cited by
- 20
- References
- 21
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 4 · 2023: 5 · 2022: 5 · 2021: 1 · 2020: 1
- FWCI
- 1.23
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition (publishedVersion)
- Funders
- São Paulo Research Foundation - FAPESP (2013/07375-0,2014/12236-1,2017/25908-6,2019/07825-1)
- Brazilian National Council for Scientific and Technological Development - CNPq (307066/2017-7,427968/2018-6)
- Petrobras (2017/00285-6)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- neural network regularization, overfitting, image classification, deep neural network
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
