CoChat
Sign Up
Cited by 415Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks

Open access
Santos, Claudio Filipi Gonçalves Dos, Papa, João Paulo·2022·ACM Computing Surveys
DOI 10.1145/3510413
Open this paper in CoChat — free accountOpen PDF

Abstract

Several image processing tasks, such as image classification and object detection, have been significantly improved using Convolutional Neural Networks (CNN). Like ResNet and EfficientNet, many architectures have achieved outstanding results in at least one dataset by the time of their creation. A critical factor in training concerns the network’s regularization, which prevents the structure from overfitting. This work analyzes several regularization methods developed in the past few years, showing significant improvements for different CNN models. The works are classified into three main areas: the first one is called “data augmentation,” where all the techniques focus on performing changes in the input data. The second, named “internal changes,” aims to describe procedures to modify the feature maps generated by the neural network or the kernels. The last one, called “label,” concerns transforming the labels of a given input. This work presents two main differences comparing to other available surveys about regularization: (i) the first concerns the papers gathered in the manuscript, which are not older than five years, and (ii) the second distinction is about reproducibility, i.e., all works referred here have their code available in public repositories or they have been directly implemented in some framework, such as TensorFlow or Torch.

AI snapshot

Written from the paper’s full text.

TL;DR

This survey reviews recent regularization methods for convolutional neural networks, grouping them into data augmentation, internal changes, and label regularization. It compiles reported results across common datasets and architectures, arguing regularization improves generalization with little or no inference cost while highlighting gaps in label-level, lightweight-CNN, and non-classification research.

Key findings

  • The surveyed works are unevenly distributed: about 44% operate on the input/data-augmentation level, about 22% on the label level, and the remainder on internal structures.
  • Cutout’s best patch sizes varied by dataset: 16×16 for CIFAR-10, 8×8 for CIFAR-100, 20×20 for SVHN, and 32×32 for STL-10.
  • RandomErasing reduced classification error by at least 0.3% across tested setups and improved object-detection mAP by 0.5 when trained only on the dataset and 0.4 when combined with PASCAL VOC 2012 training data.
  • DropBlock improved ImageNet baseline accuracy by almost 2% with ResNet-50 and around 0.3% with AmoebaNet-B; RetinaNet object detection improved by more than 1.5 AP. Bag of Tricks improved MobileNet on ImageNet by almost 1.5%.

Methods

  • This is a survey/compilation, not a new experimental study: it classifies regularization methods into “data augmentation,” “internal changes,” and “label,” and includes works no older than five years with public code or framework implementations.
  • It compares reported classification errors in Table 2 across CIFAR-10, CIFAR-100, SVHN, and ImageNet using architectures such as ResNet-18/50, Wide Residual Networks, ResNeXt, and PyramidNet.
  • It also discusses common datasets, architectures, and protocol considerations for fair evaluation, including source-code availability, combining regularizers, and avoiding confounding training changes.

Limitations

  • Stated: the paper highlights a lack of label regularizers, noting only two relevant works [53,63]; a lack of mathematical explanation for most approaches; insufficient investigation of lightweight CNNs; and few works beyond image classification, with CutBlur as the only found example.
  • Stated: several methods have narrow evaluations—SLS is evaluated only on MobileNet V2, Shake-shake only works on three-branch ResNets, ShakeDrop was not evaluated on ResNet-18, and MaxDropout was not assessed on PyramidNet—making direct comparison difficult.
  • Evident: Table 2 is incomplete and messy, and results are compiled across heterogeneous protocols and architectures; the paper itself states that defining the best regularizer is not trivial and that rankings can reverse depending on the architecture and combined regularizer.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3510413

Cites

84 of 85 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 115 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    Besides Dropout, no other study surveyed in this work is older than four years.

    “Besides Dropout [49], no other study is older than four years, making this study very much up-to-date;”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [49]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingnone given

    All works referred to in this survey have their code available in public repositories or are directly implemented in some framework such as TensorFlow or Torch.

    “all works referred here have their code available in public repositories or they have been directly implemented in some framework, such as TensorFlow or Torch.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The frequency of works for regularization compared to other problems is very low.

    “However, the frequency of works for regularization compared to other problems is very low.”
    Quote in the paperSubstantial claim
  • Findingnone given

    The surveyed regularization methods have shown significant improvements for different CNN models.

    “This work analyzes several regularization methods developed in the past few years, showing significant improvements for different CNN models.”
    Quote in the paperSubstantial claim
  • Findingown data

    The authors found only two relevant works on label-level regularization.

    “Although around 22% of the works make changes on the label as a regularization strategy, we found two relevant works on the area only [53,63].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [53]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [63]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. 52
  • May combine two claims. 40
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    All related algorithms in this study are available in some way, usually on Github.

    “All related algorithms in this study are available in some way, usually on Github.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Cutout and MaxDropout original codes show they do not execute anything during inference, but BatchNormalization executes its algorithm in deducing the test set.

    “For example, Cutout [7] and MaxDropout [44] original codes show they do not execute anything during the inference, but the BatchNormalization [25] executes its algorithm in deducing the test set.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [7]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [44]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [25]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    The majority of the methods work on input or internal structures, which shows a lack of research on label regularization methods.

    “The majority of the methods work on input or internal structures, which shows a lack of research on label regularization methods.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Some algorithms work on two different levels; for instance, CutMix and CutBlur work on both input and label levels.

    “For instance, CutMix and CutBlur work on both input and label levels.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    All techniques surveyed in this work were able to improve the results of the original models significantly.

    “All techniques here were able to improve the results of the original models significantly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    As far as the authors are concerned, they found only two recent surveys about regularization for deep neural networks.

    “As far as we are concerned, we found only two recent surveys about regularization for deep neural networks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The most common datasets used in regularization works are perfectly balanced, with every class having a similar number of samples in training and test validation.

    “Another characteristic of these datasets is that they are perfectly balanced, meaning that every possible class has a similar amount of samples in the training and test validation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    CutBlur is the only work found in the survey that addresses problems other than image classification, highlighting a lack of works in this direction.

    “The only work found so far to solve different problems than image classification is the CutBlur [66], thus highlighting the lack of works in this direction.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [66]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The works analyzed in the survey lack an evaluation of regularization methods on more complex data, such as unbalanced datasets.

    “the lack of an evaluation of methods on more complex data, such as unbalanced datasets, to provide richer information for other researchers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    All works analyzed in the survey lack proper investigation concerning lightweight CNNs.

    “One point missing in all works analyzed in this survey is the lack of proper investigation concerning lightweight CNNs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The most common datasets used in regularization works concern objects and animals that humans can easily distinguish.

    “The most common datasets used in regularization works concern objects and animals, which humans can easily distinguish.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Regularization adds little overhead during training time, with AutoAugment being perhaps the only method that increases training time, and adds no overhead at inference.

    “As a general rule, regularization adds little overhead during training time (AutoAugmentation [5] is, perhaps, the only one that increases training time to find out the better policies for data augmentation) and no overhead at all at inference.”
    • Also stated in: § 7.5 Other Problems besides Classification · plain
      “most do not require any relevant extra time, and none influences the time taken for inference.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    There are only two regularizers that work directly on labels.

    “Unfortunately, there are only two regularizers that work directly on labels.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Using a good regularizer does not influence the final model's performance; the model's computational cost for inference is the same whether or not a regularizer is used.

    “One important aspect of using a good regularizer is that it does not influence the final model's performance.It means that, independently of using or not one regularizer, the model's computational cost for inference is the same.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Premisenone given

    Even considering huge datasets such as ImageNet, the number of images available is minimal compared to the quantity of data a human brain receives through the eyes.

    “Even considering huge datasets such as ImageNet, the number of images available is minimal compared to the quantity of data a human brain receives through the eyes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    One main difference between normalization and regularization is that regularization is not performed after the training period, while normalization is kept in the model.

    “One main difference between the normalization and regularization techniques is that the second is not performed after the training period, while the first is kept in the model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    If rotation is too severe, an input 6 may be transformed into a 9, leading the model to learn wrong information.

    “If the rotation is too several, then an input”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Some transformations may distort the image into another existing class in the classification.

    “For example, some transformations may distort the image into another existing class in the classification.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    The amount of information available for training CNNs requires more data variability than is possible.

    “If the amount of information available is considered, then it becomes clear the training task requires more data variability than possible.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Regularization is a critical factor in training CNNs that prevents the network structure from overfitting.

    “A critical factor in training concerns the network's regularization, which prevents the structure from overfitting.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    When working with images, the most straightforward regularization method is random image changing, such as rotation and flipping.

    “When working with images, the most straightforward method is random image changing, such as rotation and flipping.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Regularization operates by increasing the variability of the data on different stages of a CNN.

    “It operates by increasing the variability of the data on different stages of a CNN.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    In some cases, regularization can influence performance during the training phase, using a little computational overhead or pre-train epochs.

    “However, in some cases, it can influence performance during the training phase, using a little computational overhead or pre-train epochs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The first recent survey is an extensive analysis of regularization methods and their results, but it focuses considerably on older methods such as adding noise to the input, DropConnect, and Bagging.

    “The first one [36] is an extensive analysis of regularization methods and their results.Although it is an interesting work, it focuses considerably on older methods, such as adding noise to the input, DropConnect [59], and Bagging [1].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [36]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [59]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [1]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning

    Unavailability of new data may lead to overfitting, where the model represents training data well but does not perform well on new test data.

    “This unavailability of new data may lead to a situation known as overfitting, where the model learns how to represent well the training data, but it does not perform well on new information, i.e., the test data.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    One key aspect of regularization methods, independent of the training phase, is to prevent the model from overfitting the training data.

    “One key aspect of the regularization methods, independent of the training phase it works, is to prevent the model from overfitting the training data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Depending on the chosen regularization strategy, some architectures can achieve a relevant gain on the final results.

    “Depending on the chosen regularization strategy used, some architectures can achieve a relevant gain on the final results.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    Overfitting usually happens when the model has been trained exhaustively on available training information such that it cannot generalize well to new information.

    “This situation usually happens when the model has been trained exhaustively in the available training information that it cannot generalize well in other new information.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Dropout is an important regularization method for different types of neural networks and has influenced several new approaches over the years.

    “Dropout [49] is undoubtedly an important regularization method for different types of neural networks, and it has influenced several new approaches over the years, besides being used in several different architectures.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The original Mixup work shows that models using such an approach can improve results in image classification, speech recognition, stabilization in generative adversarial networks, tabular datasets, and other problems.

    “The original work shows that models using such an approach can improve results not only in the image classification task but in speech recognition, stabilization in generative adversarial networks, tabular datasets, and other problems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    MaxDropout deactivates neurons based on their activations, unlike Dropout which randomly removes neurons.

    “While Dropout [49] randomly removes the neurons in the training phase, Maxdropout [44] deactivates the neurons based on their activations.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [49]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [44]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    RandAugment is faster than previous approaches and improves outcomes significantly.

    “Results show that the method is not only faster than previous approaches [5,21,32] but improves the outcomes significantly.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [21]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [32]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    RandomErasing was developed based on the Cutout technique.

    “RandomErasing [75] was further developed based on the Cutout technique.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [75]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Cutout randomly removes regions of the image before feeding the neural network during training.

    “During training, it randomly removes regions of the image before feeding the neural network.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisenone given

    Several works are mainly based on Dropout, while some others are new approaches.

    “Several works [33,38,44] are mainly based on Dropout, while some of them [9,64] are new approaches.”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [33]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [38]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [44]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [9]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [64]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    The authors of CutBlur showed that CutBlur helps the model generalize better on the super-resolution problem but that the same technique can be applied to reconstruct images degraded by Gaussian noise.

    “The authors showed that CutBlur helps the model generalize better on the SR problem but that the same technique can be applied to reconstruct images degraded by Gaussian noise.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [66]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Some techniques such as Dropout and Mixup present interesting insights about their inner mechanism.

    “Fortunately, some techniques such as Dropout [3] and Mixup [2] present interesting insights about their inner mechanism.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [3]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [2]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Using a higher test resolution produces a more reliable neural network and trains faster than the traditional approach, because training images are smaller than inference images.

    “This change not only produces a more reliable neural network but it trains faster than the traditional approach, for it requires less computational effort for training, since the images used for such a purpose are smaller than the ones used for inference.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [57]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    PBA demonstrated schedule policies instead of fixed policies, which improves results from previous studies.

    “[21] not only showed a novel augmentation algorithm but demonstrated schedule policies instead of fixed policies, which improves the results from the previous studies [5,32].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [21]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [32]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Fast AutoAugment reduced the search time significantly while producing similar results.

    “Fast AutoAugment [32] aimed at overcoming such a bottleneck with a new algorithm, reducing the time related to the search procedure significantly, besides producing similar results.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The original work argues that label-level regularization prevents the overconfidence problem, but it fails to explain why such a situation is avoided.

    “Even though the original work [53] argues that it prevents the overconfidence problem, it fails to explain why such a situation is avoided.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [53]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Dropout aims to remove some neurons, forcing the system to learn more features.

    “Dropout [49] was proposed as a simple but powerful regularizer that aims to remove some neurons, thus forcing the entire system to learn more features.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Structural Label Smoothing (SLS) proposes to compute the label smoothness factor by estimating the Bayes Estimation Error, which helps define label boundaries for each instance.

    “Structural Label Smoothing (SLS) [31] proposes to compute such a value by estimating the Bayes Estimation Error, which, according to authors, helps define the label's boundaries for each instance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The original work suggests that ε = 0.1 is the excellent condition for label smoothing.

    “The original work suggests that ϵ = 0.1 is the excellent condition; however, the Two-Stage Label Smoothing (TSLA) [63] suggests that, in general, the gradient descent combined with the label smoothing technique can only improve the results until a certain point of training; after that it is better to set all values to 0 and 1 for the active class.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [52]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    JoCoR could easily fit traditional supervised problems such as data classification to improve outcomes.

    “Although the method was developed for weakly supervised problems, it could easily fit traditional supervised problems, such as data classification, to improve outcomes.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [61]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    JoCoR trains two similar neural networks on the same dataset and tries to correlate two different labels.

    “JoCoR [61] trains two similar neural networks on the same dataset and tries to correlate two different labels.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [61]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    JoCoR calculates loss by adding cross-entropy losses of both networks plus contrastive loss, and uses only the most negligible losses on the batch to update parameters.

    “The method calculates the loss by adding the cross-entropy losses of both networks plus the contrastive loss between them and then uses only the most negligible losses on the batch to update the parameter of the architectures.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [61]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    MaxDropout improves ResNet18 results on CIFAR-10 and CIFAR-100 and outperforms Dropout on WideResNet-28-10.

    “The original work shows it can improve ResNet18 results on CIFAR-10 and CIFAR-100 [26] datasets, and it also outperforms Dropout on the WideResNet-28-10 model [68].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [44]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [26]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [68]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    AutoDrop patterns improve results on CIFAR-10 and ImageNet and can be transferred between datasets.

    “Results in CIFAR-10 and ImageNet show that these patterns improve results and can be transferred in between datasets.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [38]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The use of residual connections and their improvements have achieved relevant results on several tasks.

    “In the past few years, the use of residual connections, first introduced in the well-known neural architecture ResNet [16], and their further improvements [62,68] have achieved relevant results on several tasks.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [16]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [62]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [68]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    LocalDrop achieves relevant improvement on different CNN architectures targeting image classification, although it is time-consuming.

    “Although very time-consuming, the proposed method achieves relevant improvement on different CNN architectures targeting image classification.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Manifold Mixup acts like Mixup but operates in any internal layer of a CNN, not only the input layer.

    “The Manifold Mixup [58] acts like the Mixup [69], however, operating in any internal layer of a CNN, and not only in the input layer.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [58]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [69]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Shake-shake improves CIFAR-10 outcomes by at least 0.15%, achieving almost 0.6% improvement on the best result.

    “Results on CIFAR-10 show that such an approach can improve outcomes by at least 0.15%, achieving almost 0.6% improvement on the best result.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Later studies have shown that the success of residual connections is due to the creation of a structure called 'identity mapping'.

    “Later studies [17] have shown that such a success is due to the creation of a structure called”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    TargetDrop accomplishes better results than DropBlock and demonstrates more consistency in the region determining class.

    “Results show this method not only accomplishes better results than DropBlock but, by using grad-CAM [45], demonstrates more consistency in the region that determines to which class a given input belongs.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [76]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [45]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    According to the authors, using the smoothing factor over the encoding label makes the softmax function produce values closer to the smoothed encoded vector, limiting backpropagation values and producing a more realistic value according to the class.

    “According to the authors [52], by using the smoothing factor over the encoding label, the softmax function applied over the vector produces values closer to the smoothed encoded vector, limiting the value used in the backpropagation algorithm and producing a more realistic value according to the class.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [52]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Label smoothing proposes a regularization technique in the label encoding process by changing the value on each position of the one-hot representation.

    “Dating back from 2015 [52], label smoothing proposes a regularization technique in the label encoding process by changing the value on each position of the one-hot representation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [52]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The authors argue that both networks agree with predictions by using the smallest values to update parameters, and labels tend to be less noisy.

    “The authors argue that both networks agree with the predictions by using the smallest values to update parameters, and the labels tend to be less noisy.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [61]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Shake-shake only works on three-branch ResNet, making it hard to compare other methods directly.

    “Besides the improvement, this method only works on three-branches ResNet, making it hard to compare other methods directly.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    DropBlock increases baseline accuracy by almost 2% on ImageNet with ResNet-50 and around 0.3% with AmoebaNet-B.

    “Its experiments show relevant results on the ImageNet dataset, increasing the baseline accuracy by almost 2% when using ResNet-50, beating other regularizers, such as Cutout and AutoAugment, and around 0.3% when using AmoebaNet-B.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    ShakeDrop works not only on ResNeXt but also on ResNet, Wide ResNet, and PyramidNet.

    “It works not only on ResNeXt architecture but on ResNet, Wide ResNet, and PyramidNet, too.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [64]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    TSLA suggests that gradient descent combined with label smoothing can only improve results until a certain point of training; after that it is better to set all values to 0 and 1 for the active class.

    “the Two-Stage Label Smoothing (TSLA) [63] suggests that, in general, the gradient descent combined with the label smoothing technique can only improve the results until a certain point of training; after that it is better to set all values to 0 and 1 for the active class.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [63]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Shake-shake improves CIFAR-100 results by 0.4%, but removing weights' changes on backward pass improves by 0.5%.

    “Results on the CIFAR-100 were improved by 0.4%; however, in this specific case, the removal of weights' changes on the backward pass ends up in slightly better results, improving by 0.5%.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationcitation

    The unavailability of source code makes the evaluation process untrustworthy, because other transformations such as Dropout may be operating during training.

    “turning the evaluation process not trustful, since there might be other transformations working on training, such as Dropout [49].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [49]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The YOLO architecture is a fully convolutional network, meaning that every layer performs a 2D convolutional process.

    “the YOLO architecture [40] is a Fully Convolutional Network, which means that every layer performs a 2D convolutional process.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [40]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Residual Dense Network achieves outstanding results on image reconstruction from noisy and low-resolution images.

    “The Residual Dense Network has outstanding results on image reconstruction from noisy [73] and lowresolution images [72].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [73]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [72]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    In all works mentioned earlier, at least one of the architectures described in this subsection is used.

    “In all works mentioned earlier, at least one of the architectures described in this subsection is used.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    TSLA might be considered the best regularizer to be used on a label level.

    “For this reason, the TSLA might be considered the best one to be used on a label level.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Regularization is a vital tool to improve final CNN results because it helps prevent the model from overfitting on the training data.

    “Regularization is a vital tool to improve the final CNN results, since it helps prevent the model from overfitting on the training data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Smaller CNN designs usually have fewer parameters or make use of less complex calculations.

    “these smaller designs usually have fewer parameters or make use of less complex calculations.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [23]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    CNNs have state-of-the-art results in the image reconstruction domain.

    “Another domain where CNNs have state-of-the-art results is image reconstruction.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [73]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [72]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). The quote only partly states this claim.

    Interpretationnone given

    For the sake of research in regularization in image processing, CIFAR, ImageNet, and SVHN are considered the most frequent datasets.

    “For the sake of research in regularization in image processing, two datasets are considered the most frequent, i.e., CIFAR, Imagenet, and SVHN.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A zero-padded shortcut connection does not influence gradient values because no other operation is performed in the residual connection.

    “however, the study shows that a zero-padded shortcut does not influence the values, because no other operation is performed in the residual connection.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [6]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [?]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Dropout regularization between convolutional layers after ReLU activation helps perturb batch normalization operation, which prevents overfitting.

    “Dropout regularization between the convolutional layers after the ReLU activation helps perturb the batch normalization operation, which prevents overfitting.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [68]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The survey's primary inclusion condition is that a paper must have its source code available so that it can be compared directly with other methods.

    “Therefore, the primary condition to be cited in this survey is to have the source code available so it can be compared to other methods directly.”
    • Also stated in: § Additional information · plain
      “As mentioned before, we only considered papers with the source code available.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    EfficientNet provides a calculation for defining how an efficient CNN architecture should be designed, based on width, depth, and resolution.

    “EfficientNet [54] provides a clever calculation for defining how an efficient CNN architecture should be designed, based on the width, depth, and resolution.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [54]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    The likely best regularizer concerning the input layer is RandAugment, because it does not affect training time and achieves satisfactory results.

    “For example, the likely best regularizer concerning the input layer is the RandAugment, for it does not affect the time spent for training and achieves satisfactory results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Shallower and simpler neural network architectures are the ones that could benefit most from the use of regularizers.

    “however, it is particularly necessary for shallower neural networks.”
    • Also stated in: § 7.5 Other Problems besides Classification · hedged
      “the lack of using simpler architectures, which are the ones that could be more benefited from the use of regularizers”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    AutoAugment and Fast AutoAugment incorporate Cutout as one of their policies.

    “the AutoAugment [5] and the Fast AutoAugmentat [32] incorporate as one of their policies the Cutout [7].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [32]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [7]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

79 of 79

  • The paper relies on this without support.

    The surveyed works are representative of recent CNN regularization methods.

    “This study focuses on the most recent regularization techniques for CNNs.”

  • The paper relies on this without support.

    The three categories of data augmentation, internal changes, and label cover the regularization methods relevant to CNNs.

    “In this work, the regularization algorithms are divided into three main categories, each one in a given section: The first one is called”

  • The paper relies on this without support.

    The architectures and datasets described are representative of regularization research.

    “For the sake of research in regularization in image processing, two datasets are considered the most frequent, i.e., CIFAR, Imagenet, and SVHN.”

  • The authors state this as an assumption.

    A deep neural network can be considered a set of smaller neural networks.

    “A deep neural network can be considered a set of smaller neural networks.”

  • The paper relies on this without support.

    In medical and some real-world problems, class balancing is hard to obtain.

    “Usually, in medical and some real-world problems, such a balancing is hard to obtain.”

  • The paper relies on this without support.

    In health-related problems, any increase in results can lead to safer treatment or avoid misuse of medication and death.

    “In health-related problems, any increase in the results can lead to a safer treatment or even avoid misuse of medication and death.”
6 of 6

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Two datasets are named, but three are listed

    Minor

    The sentence calls these two datasets but lists CIFAR, ImageNet, and SVHN.

  • The count of label regularizers conflicts with Table 1

    Minor

    The text says only two regularizers work directly on labels, whereas Table 1 lists Label Smoothing, TSLA, SLS, and JoCoR as label regularizers.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Yann LeCun and Corinna Cortes. 2010. MNIST handwritten digit database. Retrieved from http://yann.lecun.com/exdb/mnist/.

Cited by

Sorted by citations

Related

Authors
Santos, C. F. G. D., & Papa, J. P.
Affiliations
  • Universidade Federal de São Carlos (BR)
  • Universidade Estadual Paulista (Unesp) (BR)
ORCID
Venue
ACM Computing Surveys
Publisher
Association for Computing Machinery (ACM)
Published
2022-01-31
Volume
54
Issue
10s
Pages
1-25
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
DOI
10.1145/3510413
OpenAlex
W4206569030
Cited by
415
References
85
Citations by year
2026: 83 · 2025: 143 · 2024: 116 · 2023: 62 · 2022: 11
FWCI
26.74
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Computing Surveys (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • São Paulo Research Foundation (2014/12236-1, 2017/25908-6, and 2019/07665-4)
  • Brazilian National Council for Scientific, Technological Development (307066/2017-7, and 427968/2018-6)
  • Eldorado Research Institute
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
overfitting, regularization, convolutional neural networks, data augmentation, kernel modification, image classification, object detection, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-17.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-17.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review