Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 2,101–2,200 of 2,667
Sample-based relationship for assisting diagnosis of pneumonia in medical care
Chen, H., Lu, H., Zhu, W., Zhou, Y., Yao, Y., & Wang, R. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18848-y
“Bengio et al. [25] utilized representation learning to overcome the training difficulties caused by the imbalanced data distribution.”
iNLC: Iterative Noisy Label Correction
Koo, S., Park, S., Roh, S.-D., & Chung, K. (2023) · International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) · doi:10.1109/sitis61268.2023.00024
“Zhang et al. [2] reported that when CNNs were trained with noisy data, their performance would decrease significantly because they tended to overfit to incorrectly labeled data.”
BackgroundBoosting object detection via diffusion-based data augmentation
Song, K., Li, W., & Liu, S. (2024) · International Conference on Graphics and Image Processing (ICGIP) · doi:10.1117/12.3021662
“Resampling techniques, including oversampling, undersampling, and SMOTE [19], seek to balance classes, but occasionally at the cost of overfitting or data redundancy [20].”
MethodTime-frequency domain joint precoding based on the LSTM network and repetitive HSR channelPreprint
Zhu, M., Duan, Q., & Xue, J. (2024) · doi:10.36227/techrxiv.171172921.13223722/v1
A remark on conditional entropyPreprint
Wang, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.02167
Feature learning model based on Feature space constraints and Self-Attention learning
Xu, C., Wang, Y., Zhou, Q., Wang, D., Dong, H., & Wang, J. (2023) · International Conference on Machine Learning and Computer Application (ICMLCA) · doi:10.1145/3650215.3650384
“To address the issue of ”network degradation”[10], we propose a Self-Attention Feature Learning module.”
Method(2024) · Frontiers research topics · doi:10.3389/978-2-8325-4801-1
Overparameterized maximum likelihood tests for detection of sparse vectors
Golubev, G. K. (2023) · Проблемы передачи информации · doi:10.31857/s0555292323010047
Fast Tunnel Traversal for Ground Vehicles by Bearing Estimation with Neural Networks
Cano, L., Tardioli, D., & Mosteo, A. R. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-58676-7_23
Human-Centered AI at Work: Common Ground in Theories and Methods
(2024) · Frontiers research topics · doi:10.3389/978-2-8325-4840-0
Gao, Y., Fu, J., Wang, Y., & Guo, Y. P. (2024) · Visual Computing for Industry Biomedicine and Art · doi:10.1186/s42492-024-00162-x
Personalized Sleep Monitoring Using Smartphones and Semi-supervised Learning
Mammen, P. M., Zakaria, C., & Shenoy, P. J. (2024) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-59717-6_22 · Cited by 1
Anwar, S., Yunker, A., Kettimuthu, R., Anastasio, M. A., Liu, Z., & He, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4861434
Nedeljković, Z., Milošević, D., Radulović, M., Milošević, N. T., & Rajković, N. (2024) · Medicinska istrazivanja · doi:10.5937/medi57-48847
Mastering infinite depths: Optimization and generalization in deeper neural networks
Gao, T. (2024) · doi:10.31274/td-20240617-104 · Cited by 2
Contrastive Learning Joint Regularization for Pathological Image Classification with Noisy Labels
Guo, W., Han, G., Mo, Y., Zhang, H., Fang, J., & Zhao, X. (2024) · Electronics · doi:10.3390/electronics13132456
Block Switching: Defying Fast Gradient Sign Resistance
Mwangi, J. K., Kuria, J., & Wandeto, J. (2024) · IST-Africa Conference (IST-Africa) · doi:10.23919/ist-africa63983.2024.10569920
Face Image Inpainting Using Context Encoders and Dynamically Initialized Mask
Maurya, S., & Sandhan, T. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-58535-7_1
Tree-managed network ensembles for video prediction
Fall, E., Chang, K., & Chen, L. (2024) · Machine Vision and Applications · doi:10.1007/s00138-024-01575-7
“At scale and with diverse data, increasing the network size may not yield a proportional improvement in learning capacity [3– 6].”
Augmentmia: Data Augmentation Based Enhanced Membership Inference Attack Without Shadow ModelsPreprint
Liu, S., Qian, Y., Hao, Y., & Chen, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4889219
T-Explainer: uma abordagem para a explicabilidade em Aprendizado de Máquina baseada em gradientes
Ortigossa, E. S. (2024) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2024.tde-11072024-171752 · Cited by 1
An effective method for filtering laser bathymetry echo signals
Liu, Y., Li, H., Liu, S., Mei, T., Zhao, L., Ma, Y., Sun, W., & Zhang, Y. (2024) · International Conference on Signal Processing and Communication Security (ICSPCS) · doi:10.1117/12.3038644
“The presence of a large amount of noisy data in training data can severely affect the model's predictive accuracy[13].”
The Effect of the Internet of Things, Artificial Intelligent and Tracking on Smart Transportation
Offei, M. O., & Toku, L. I. (2024) · BENTHAM SCIENCE PUBLISHERS eBooks · doi:10.2174/9789815223255124010013
Partial Label Learning with Noisy Labels
Pan, Z., Tang, L., & Pan, Z. (2024) · Annals of Data Science · doi:10.1007/s40745-024-00552-1
“Some early research works [13, 14] are dedicated to analyzing the impact of label noise on the training process of deep neural network models through experimental means.”
A mixed sample data augmentation method based on sparse adversarial attackPreprint
Xu, X., Zhang, X., Wang, Y., Liu, L., & Xu, E. (2024) · Research Square · doi:10.21203/rs.3.rs-4698850/v1
Learning from Multiple Noisy Annotations via Trustable Data Mixture
Wang, R., Chen, F., Guan, C., Xue, C., Ji, X., & Li, W. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5666-7_36
Hide-and-Seek: Data Sharing with Customizable Machine Learnability and Privacy
Xu, H., & Shu, T. (2024) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn61486.2024.10637601
“Due to a recent observation that machine learning models tend to memorize some information about the training dataset [19] [20] [21], leading to its vulnerability to the privacy attacks [22] [23] such as membership inference attack [24], the privacy-preserving machine learning that not only…”
BackgroundNoisy Label Learning Based on Weighted Neighborhood Consistency
Rong, Q., Zhang, L., Yuan, L., Ding, X., & Li, G. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-7244-5_4
Convolutional Neural Network for Predicting the Performance of a Tunnel Boring Machine
Zhu, Y. P., Li, M., Nie, Y., Wang, R., Wang, Y., & Guo, X. (2024) · Indian geotechnical journal · doi:10.1007/s40098-024-01065-7
Yang, X., Deng, S., Liu, S., Suo, Y., Wing.W.Y, N., & Zhang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70341-6_5
“Meanwhile, data augmentation is also recognized as a regularization technique that can reduce generalization error without necessarily reducing training error [7,28].”
MGAU-ResNet-multi-grained attention U-shaped residual networks for medical image segmentation
Jing, W., & Sien, L. C. (2024) · AIP conference proceedings · doi:10.1063/5.0229757
Machine Learning Models for EOS SAT-1 Satellite Image Enhancing
Popika, V., & Lelechenko, L. (2024) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss53475.2024.10640387
Progressively Robust Loss for Deep Learning with Noisy Labels
Cai, Z., Yan, S., Chen, Y., Zhang, C., Sun, Z., & Yao, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650630 · Cited by 1
Self-citationGao, L., Sun, Z., Yan, J., Wang, S., Zhang, Y., & Xiao, J. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650097
“The inclusion of spurious correlation information in the training set (External Sensitive Information) leads to poor generalization [1].”
BackgroundQuadripartite: Tackle Noisy Labels by Better Utilizing Uncertain Samples
Cui, X., Liang, X., Cao, C., Shi, G., & El-haddar, N. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651073
Sample Size for Training and Testing: Segment Anything Models and Supervised Approaches
Cuza, D., Fantozzi, C., Nanni, L., Fusaro, D., Felipe, G. Z., & Brahnam, S. (2024) · Intelligent systems reference library · doi:10.1007/978-3-031-65430-5_6
Semantic feature-based test selection for deep neural networks: A frequency domain perspective
Jiang, Z., Li, H., Tian, X., & Wang, R. (2024) · Computer Science and Information Systems · doi:10.2298/csis230907045j
“Additionally, researchers [11,18,41,60] found that DNNs lack robustness because they over-fitted to noisy high-frequency information and ignored semantic information during training.”
VF-Net: Robustness Via Understanding Distortions and Transformations
Amerehi, F., & Healy, P. J. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10647346
Zhang, Y., Jiang, J., Zhao, Y., & Liang, K. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10647657
Purposeful Regularization with Reinforcement Learning for Facial Expression Recognition In-the-Wild
Hong, S. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00464
“The regularization introduces additional constraints or penalties [20], [33] to the CNNs being trained, thereby, decreasing the generalization error on unseen data [43], [14].”
BackgroundFEGRA: Defending Membership Inference Attacks via Feature Extraction and Gradient Ascent
Lu, Y., & Zhang, X. (2024) · International Conference on Computer Information Science and Application Technology (CISAT) · doi:10.1109/cisat62382.2024.10695256
“However, in recent years, some studies [1][2] have highlighted that deep learning models are vulnerable to privacy attacks by adversaries, such as model extraction attacks, attribute inference attacks, and membership inference attacks [3], resulting in serious privacy leaks.”
BackgroundDeep Learning Based Multi-Class Classification of Invasive Plant Species
Nunavath, V., & Danilin, O. (2024) · doi:10.1109/icait61638.2024.10690602
“Data augmentation enhances the training dataset by applying domain-specific transformations such as random rotation, and shifts [20].”
MethodConcentrating Estimation Attention: Human Prior Constrained Methods for Robust Classification
Cao, Z., Yang, S., Liu, Y., Xu, X., Pei, H., Huang, Y., Yu, Y., & Zhang, R. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8499-8_39
Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture
Movahedi, S., Orvieto, A., & Moosavi-Dezfooli, S.-M. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.12025
Delving Deeper Into Clean Samples for Combating Noisy Labels
Gao, Y., Sun, Z., Yao, Y., Jiang, X., & Tang, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8692-3_13
A New Paradigm for Enhancing Ensemble Learning Through Parameter Diversification
Jiang, J., Gu, F., & Shang, C. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8487-5_20
Ammar, M. B., Brellmann, D., Mendoza, A., Antoine Manzanera, B., & Franchi, G. (2025) · neural information processing systems · doi:10.52202/085713-3066 · Cited by 1
On Adversarial Training with Incorrect Labels
Zhao, B. Z. H., Lu, J., Zhou, X., Vatsalan, D., Ikram, M., & Kâafar, M. A. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0573-6_9
Theory-Regularized Deep Learning for Demand-Curve Estimation and Prediction
Kim, C., Kim, D. S., Joo, M., & Che, H. (2024) · Artificial Intelligence for Business (AIxB) · doi:10.1109/aixb62249.2024.00008
Strainer GAN: Filtering out Impurity Samples in GAN Training
Shin, J., & Lee, S. (2024) · SIGGRAPH Asia Posters · doi:10.1145/3681756.3697961
“[Zhang et al. 2021] demonstrated that deep neural networks can learn even randomly labeled data, raising concerns about overfitting to incorrect data.”
Efficient Object Detection via Fine-Grained Regularization with Global Initialization
Chen, B., Wu, Q., Chen, S., & Kang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78122-3_5
An improved graph factorization machine based on solving unbalanced game perception
Xie, X., Jia, Y., & Ma, T. (2024) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2024.1481297
Label-Based Disentanglement Measure among Hidden Units of Deep Learning
Zhang, C., Hou, Y., & Song, D. (2024) · Neural Processing Letters · doi:10.1007/s11063-024-11708-8
“…in the model, i.e., I ( S , W ) , has been empirically used to describe the model’s complexity, demonstrating the presence of the bias-variance trade-off effect in experiments [22], which can generally not be replicated when using the number of parameters as a measure of model complexity [27].”
Self-citationChatterjee, M. J. V. (2024) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2024-16267
Leveraging Quantum Networks for Early Detection of Heart Diseases and Breast Cancer
Yelleti, D. S. D., Nakka, H. B., Velaga, H. V., Karre, A., S., S. P. V. V., & Kothalanka, A. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-8135-9.ch011
A cosine similarity-based maximal clique point cloud registration algorithm
Lin, J., Wang, X., Chang, M., Yin, Z., & Zhang, L. (2024) · Computing · doi:10.1007/s00607-024-01403-w
“Researchers have proposed replacing various steps of the classical registration process with deep learning techniques to achieve better results [14–18, 21, 22].”
Estimation of the Generalization Ability of Deep Learning Model Based on Complexity
Li, X., Ke, W., Feng, J., Liu, S., Xu, Z., & Sheng, X. (2024) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia63733.2024.10809003
Constructing Complexity Metrics for Measuring Generalization Ability of Deep Learning Models
Wen, W., Ke, W., Feng, J., Liu, S., Xu, Z., & Sheng, X. (2024) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia63733.2024.10808837
Learning Interpretable Continuous Representation for Alzheimer’s Disease Classification
Zhou, M., Wang, M., Zhang, Y., Yuan, Z., Lanfranchi, V., & Yang, P. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821731
“Empirical studies have demonstrated that deep networks, typically used as a ”black box,” are capable of fitting even random labels [17].”
G-Mix: A Generalized Mixup Learning Framework Toward Flat Minima
Li, X., & Tang, B. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3529816
“Recent studies [19], [52], [53] have observed a positive correlation between the generalization ability of over-parameterized DNNs and the sharpness of the loss landscape.”
Fast Second-order Method for Neural Networks under Small Treewidth Setting
Li, X., Long, J., Song, Z., & Zhou, T. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825379
Collaborative evaluation for performance assessment of medical imaging applications
Kassem, H., Singh, A., Aristizábal, A., Bakas, S., Sheller, M., Umeton, R., Pati, S., Moorthy, P. N., & Karargyris, A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-44-323761-4.00019-5
Learning From Mistakes: A Multilevel Optimization Framework
Zhang, L., Garg, B., Sridhara, P., Hosseini, R., & Xie, P. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3534151
“One is the uniform noise, which means the label of each training sample can uniformly change to another random class with probability p following the instruction in [72] and is the most common phenomenon in the literature.”
Interplay between Bayesian Neural Networks and Deep Learning: A SurveyPreprint
Chen, Y., Yu, S. S., Li, Z., Eshraghian, J. K., & Lim, C. P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5100162 · Cited by 1
Scale-Insensitive Neural Network Significance TestsPreprint
Fallahgoul, H. A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5112429
Haleem, A. S., & Ali, I. H. (2025) · AIP conference proceedings · doi:10.1063/5.0254437
Zhao, Y., Wang, S., Qiao, S., Zhang, Y., Wang, J., Yin, W., Liu, T., & Pang, S. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3540776
“Most studies targeting natural images largely ignore the specific sources of the images or default to images from the same source [14], [15].”
Hierarchical Learned Auctions for Communication-Sparse Satellite Environments
Lehman, H. C., & Valasek, J. (2025) · AIAA SciTech Forum · doi:10.2514/6.2025-0560
Li, Y., Yang, B., Lei, Y., Li, X., Wang, T., & Li, L. (2024) · China Automation Congress (CAC) · doi:10.1109/cac63892.2024.10865786
Meta-Data-Guided Robust Deep Neural Network Classification with Noisy Label
Lu, J., Wang, Y., Shi, A., Ma, J., & Jin, Q. (2025) · Applied Sciences · doi:10.3390/app15042080
RLGNN: Robust Graph Neural Networks with Noisy Label Learning
Qian, H., Li, D., Li, Q., & Wang, J. (2024) · IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) · doi:10.1109/ispa63168.2024.00256
“Due to the tendency of models to overfit on noisy labels during training, resulting in diminished performance [28], it is crucial to filter out noisy labels before further training.”
Regularização, aprendizagem profunda e interdisciplinaridade em problemas inversos mal-postosPreprint
Beraldo, R. G., & Suyama, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.13976
Priescu, I., Paraschiv, T., Joița, D., Răcuciu, C., Postolea, D., Dăscălescu, A. C., Apostolescu, T. C., Oncioiu, I., Pîrnău, M., Ionescu, V., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5117611
Deep Minimax Classifiers for Imbalanced Datasets With a Small Number of Minority Samples
Choi, H. M., & Seo, D. (2025) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2025.3546083
Comparative Assessment of Recent Pedestrian Detectors and Trackers
Amrani, O. T., Qadadri, H., Bendali-Braham, M., & Roussel, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-82475-3_15
Kang, C., Moore, J. A., Robertson, S., Wilms, M., Towlson, E. K., & Forkert, N. D. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107308
Integrating Deep Learning and Counterfactual Methods for Causal Inference in Genomics
Gobonamang, T. K., & Mpoeleng, D. (2024) · International Conference on Electrical and Computer Engineering Researches (ICECER) · doi:10.1109/icecer62944.2024.10920394
Tarafder, E., Khatun, S., Awais, M., Qayyum, M., Nizamani, M. M., & Wang, Y. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-24712-5.00010-5
Song, Y. G., Xie, J., Li, J., Yang, L., & Yang, X. (2024) · International Conference on Computer, Big Data and Artificial Intelligence (ICCBD+AI) · doi:10.1109/iccbd-ai65562.2024.00073
“However, CNNs are often regarded as ”black boxes” [6], lacking interpretability.”
Neural Network Pruning for Invariance Learning
Xu, D., Chen, Y., Sun, Y., & Wang, W. W. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709262
“Neural network architectures that represent data passed through invariant transformations with the same representation inherit a good inductive bias [46–48] and achieve superior performance [2, 68].”
Wu, D., Wei, G., Song, G., Li, Y., & Zhang, G. (2025) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001425580030 · Cited by 2
Utilizing Footstep Sound Event Detection by Using CNN Techniques for Assuring Property Security
Yavuz, F. Y., & Yumuşak, N. (2025) · IEEE Access · doi:10.1109/access.2025.3561076
“While creating deep learning models, another factor leading to overfitting is the complexity and depth of the neural network [24].”
Riswandi, A., Irawan, M. A., & Wekke, I. S. (2024) · Scientechno Journal of Science and Technology · doi:10.70177/scientechno.v3i3.1362
Introduction to Optimization for Deep Learning
Pauwels, E. (2025) · Springer proceedings in mathematics & statistics · doi:10.1007/978-981-97-8907-8_20
Adaptive Data Augmentation: Elliptical Masking and Dynamic Regularization with Feedback
Singh, A. K., Ambalavanan, N., Sirakov, N., & Nakhmani, A. (2025) · IEEE SoutheastCon · doi:10.1109/southeastcon56624.2025.10971474
“Data augmentation is a critical technique in deep learning, particularly for image classification tasks, as it helps prevent overfitting and improves model generalization [1].”
Solid and progressive early learning for geospatial scene classification
Zhang, H., Li, J., Meng, F. Y., Ren, H., Wang, B., & Zhao, K. (2024) · Asia Symposium on Image and Graphics (ASIG) · doi:10.1145/3718441.3718453
“Due to the common issue of over-parameterization in deep networks, they are prone to memorizing and ultimately overfitting to noise during the training process [12] [13], leading to suboptimal model performance.”
Jasim, F. T., Tayyeh, A. M., Mohamad, A. A. H., Abbas, O. A., Kadhim, N. M., Turki, A. I., Abdulwahid, E. S., Al-Badri, K. S. L., & Abdalbaqi, L. S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-0330-7.ch012
A supervised machine learning framework to predict the request fit for dynamic dial-a-ride problemsPreprint
Mader, S., Fontaine, P., & Voigt, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5236645
Hoshino, T., Kanoga, S., Takase, T., & Aoyama, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5247869
Sudha, M., & Divya, S. (2025) · Deep Science Publishing · doi:10.70593/978-93-49910-76-8
Raman, V., Sumari, P., Prabhavathy, M., & Akshaya, R. (2025) · Information systems engineering and management · doi:10.1007/978-3-031-90478-3_16
Domain Generalization with Variational Self-encoders
Ouyang, G., Guo, Y., & Lu, Y. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-2528-4_20
The recognition of wire icing risk levels based on deep learning
Wu, H., Niu, S., Yum, S. S., Lü, J., Huang, Y., Wang, T., Pan, Z., Wang, X., & Zhou, Y. (2025) · Emergency Management Science and Technology · doi:10.48130/emst-0025-0006
Tripartite: Tackling Realistic Noisy Labels with More Precise Partitions
Yu, L., Liang, X., Cao, C., Yao, L., & Liu, X. (2025) · Sensors · doi:10.3390/s25113369 · Cited by 1
Marketing Intelligence for Brand Love by Leveraging AI and Analytics
Ganesha, K. S., & Gangadhara, S. S. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-0948-4.ch015
FMW-Net: a first-order meta-weight-net approach for sample weighting
Zhou, Y., Shu, J., Liu, J., & Meng, D. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02681-2
“However, research has shown that DNNs are prone to overfitting when trained with biased data [3, 4], such as datasets with label noise [5] or class imbalance [6], then leads to poor generalization performance.”
Duco: Dual-Prototypical Contrastive Complementary Label LearningPreprint
You, Y., Wang, B., Liu, J., & Chen, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5279097
Duco: Dual-Prototypical Contrastive Complementary Label LearningPreprint
You, Y., Wang, B., Liu, J., & Chen, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5279094
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.