CoChat
Sign Up
Cited by 415Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

The DOI matches its Crossref record; 85 of the 86 references it cites were found in a registry or an index; none of the 115 sources its claims cite is checked for support yet.

Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks

Open access
Santos, Claudio Filipi Gonçalves Dos, Papa, João Paulo·2022·ACM Computing Surveys
DOI 10.1145/3510413
Open this paper in CoChat — free accountOpen PDF

Abstract

Several image processing tasks, such as image classification and object detection, have been significantly improved using Convolutional Neural Networks (CNN). Like ResNet and EfficientNet, many architectures have achieved outstanding results in at least one dataset by the time of their creation. A critical factor in training concerns the network’s regularization, which prevents the structure from overfitting. This work analyzes several regularization methods developed in the past few years, showing significant improvements for different CNN models. The works are classified into three main areas: the first one is called “data augmentation,” where all the techniques focus on performing changes in the input data. The second, named “internal changes,” aims to describe procedures to modify the feature maps generated by the neural network or the kernels. The last one, called “label,” concerns transforming the labels of a given input. This work presents two main differences comparing to other available surveys about regularization: (i) the first concerns the papers gathered in the manuscript, which are not older than five years, and (ii) the second distinction is about reproducibility, i.e., all works referred here have their code available in public repositories or they have been directly implemented in some framework, such as TensorFlow or Torch.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This survey reviews recent regularization methods for convolutional neural networks, grouping them into data augmentation, internal changes, and label regularization. It compiles reported results across common datasets and architectures, arguing regularization improves generalization with little or no inference cost while highlighting gaps in label-level, lightweight-CNN, and non-classification research.

Key findings

  • The surveyed works are unevenly distributed: about 44% operate on the input/data-augmentation level, about 22% on the label level, and the remainder on internal structures.
  • Cutout’s best patch sizes varied by dataset: 16×16 for CIFAR-10, 8×8 for CIFAR-100, 20×20 for SVHN, and 32×32 for STL-10.
  • RandomErasing reduced classification error by at least 0.3% across tested setups and improved object-detection mAP by 0.5 when trained only on the dataset and 0.4 when combined with PASCAL VOC 2012 training data.
  • DropBlock improved ImageNet baseline accuracy by almost 2% with ResNet-50 and around 0.3% with AmoebaNet-B; RetinaNet object detection improved by more than 1.5 AP. Bag of Tricks improved MobileNet on ImageNet by almost 1.5%.

Methods

  • This is a survey/compilation, not a new experimental study: it classifies regularization methods into “data augmentation,” “internal changes,” and “label,” and includes works no older than five years with public code or framework implementations.
  • It compares reported classification errors in Table 2 across CIFAR-10, CIFAR-100, SVHN, and ImageNet using architectures such as ResNet-18/50, Wide Residual Networks, ResNeXt, and PyramidNet.
  • It also discusses common datasets, architectures, and protocol considerations for fair evaluation, including source-code availability, combining regularizers, and avoiding confounding training changes.

Limitations

  • Stated: the paper highlights a lack of label regularizers, noting only two relevant works [53,63]; a lack of mathematical explanation for most approaches; insufficient investigation of lightweight CNNs; and few works beyond image classification, with CutBlur as the only found example.
  • Stated: several methods have narrow evaluations—SLS is evaluated only on MobileNet V2, Shake-shake only works on three-branch ResNets, ShakeDrop was not evaluated on ResNet-18, and MaxDropout was not assessed on PyramidNet—making direct comparison difficult.
  • Evident: Table 2 is incomplete and messy, and results are compiled across heterogeneous protocols and architectures; the paper itself states that defining the best regularizer is not trivial and that rankings can reverse depending on the architecture and combined regularizer.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3510413

Cites

85 of 86 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 of 115 cited sources checked · 0 supported · 0 flagged

Show all 44 claims

  • Probably not a substantial claim. 52
  • It may state two separate claims as one. 40
  • The quote only partly states this claim. 1
79 of 79

  • The paper relies on this without support.

    The surveyed works are representative of recent CNN regularization methods.

    “This study focuses on the most recent regularization techniques for CNNs.”

  • The paper relies on this without support.

    The three categories of data augmentation, internal changes, and label cover the regularization methods relevant to CNNs.

    “In this work, the regularization algorithms are divided into three main categories, each one in a given section: The first one is called”

  • The paper relies on this without support.

    The architectures and datasets described are representative of regularization research.

    “For the sake of research in regularization in image processing, two datasets are considered the most frequent, i.e., CIFAR, Imagenet, and SVHN.”

  • The authors state this as an assumption.

    A deep neural network can be considered a set of smaller neural networks.

    “A deep neural network can be considered a set of smaller neural networks.”

  • The paper relies on this without support.

    In medical and some real-world problems, class balancing is hard to obtain.

    “Usually, in medical and some real-world problems, such a balancing is hard to obtain.”

  • The paper relies on this without support.

    In health-related problems, any increase in results can lead to safer treatment or avoid misuse of medication and death.

    “In health-related problems, any increase in the results can lead to a safer treatment or even avoid misuse of medication and death.”
6 of 6

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Two datasets are named, but three are listed

    Minor

    The sentence calls these two datasets but lists CIFAR, ImageNet, and SVHN.

  • The count of label regularizers conflicts with Table 1

    Minor

    The text says only two regularizers work directly on labels, whereas Table 1 lists Label Smoothing, TSLA, SLS, and JoCoR as label regularizers.

2 of 2

Citations

Cites

  • Deep Residual Learning for Image Recognition

    He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 9×

    “In this specific case, where the results may infer in a life-threatening situation, the idea is to use a deeper CNN, like the Efficient-Net family [56] or ResNet [16].”
    BackgroundMethodResultInfluential

  • Very Deep Convolutional Networks for Large-Scale Image Recognition

    Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 2×

    “The Visual Geometry Group developed the VGG architecture [47], which demonstrates, for the first time, that stacking convolution layers with smaller kernels perform better than shallow layers with bigger kernels, even then performing over the same region.”
    BackgroundMethod

  • ImageNet classification with deep convolutional neural networks

    Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155

  • Gradient-based learning applied to document recognition

    Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×

    “Since the first CNN model [30], basic data augmentation, such as flipping and noise adding, has proven it can help the trained model generalize better.”
    Method

  • You Only Look Once: Unified, Real-Time Object Detection

    Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · cited 5×

    “For example, the YOLO architecture [40] is a Fully Convolutional Network, which means that every layer performs a 2D convolutional process.”
    BackgroundMethodInfluential

  • Going deeper with convolutions

    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · cited 3×

    “According to the authors [52], by using the smoothing factor over the encoding label, the softmax function applied over the vector produces values closer to the smoothed encoded vector, limiting the value used in the backpropagation algorithm and producing a more realistic value according to the class.”
    BackgroundMethod

  • Squeeze-and-Excitation NetworksPreprint

    Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845 · cited 2×

    “Later in 2018, Google showed the NAS could be improved when some rules are better designed, such as the computational limit, input size, and other parameters, and incorporate other architectures, such as Squeeze-and-Excitation [25], ending up in the EfficientNet family [56].”
    Background

  • Squeeze-and-Excitation Networks

    Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667 · cited 2×

    “Later in 2018, Google showed the NAS could be improved when some rules are better designed, such as the computational limit, input size, and other parameters, and incorporate other architectures, such as Squeeze-and-Excitation [25], ending up in the EfficientNet family [56].”
    Background

  • Untitled work

    (2014) · Cited by 33,419

  • Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

    Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · cited 2×

    “In a quick search, it is possible to find a diversity of works using Convolutional Neural Networks, such as image classification [15, 16, 24, 54], object detection [11, 40], and image reconstruction [71–73].”
    Background

Flagged first, then by citations

Show all 85 references

Research works printed but not linked: 1

  1. Yann LeCun and Corinna Cortes. 2010. MNIST handwritten digit database. Retrieved from http://yann.lecun.com/exdb/mnist/.

Cited by

Sorted by citations

See all 415

Related

Metadata

Authors
Santos, C. F. G. D., & Papa, J. P.
Affiliations
  • Universidade Federal de São Carlos (BR)
  • Universidade Estadual Paulista (Unesp) (BR)
ORCID
Venue
ACM Computing Surveys
Publisher
Association for Computing Machinery (ACM)
Published
2022-01-31
Volume
54
Issue
10s
Pages
1-25
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
DOI
10.1145/3510413
OpenAlex
W4206569030
Cited by
415
References
85
Citations by year
2026: 83 · 2025: 143 · 2024: 116 · 2023: 62 · 2022: 11
FWCI
26.74
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Computing Surveys (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • São Paulo Research Foundation (2014/12236-1, 2017/25908-6, and 2019/07665-4)
  • Brazilian National Council for Scientific, Technological Development (307066/2017-7, and 427968/2018-6)
  • Eldorado Research Institute
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
overfitting, regularization, convolutional neural networks, data augmentation, kernel modification, image classification, object detection, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-17, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-17 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review