Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 101–200 of 2,667
Domain Generalization in Computational Pathology: Survey and Guidelines
Jahanifar, M., Raza, M., Xu, K., Vuong, T. T. L., Jewsbury, R., Shephard, A., Tajeddin, N. Z., Kwak, J. T., Raza, S. E. A., Minhas, F., et al. (2025) · ACM Computing Surveys · doi:10.1145/3724391 · Cited by 106
“Adding more data and using more diverse source domains has always been shown to improve the generalizability of trained models [251].”
Statistical Mechanics of Deep Linear Neural Networks: The Backpropagating Kernel Renormalization
Li, Q., & Sompolinsky, H. (2021) · Physical Review X · doi:10.1103/physrevx.11.031059 · Cited by 122
“Nevertheless, many fundamental questions remain unresolved including the puzzling ability of gradient-based optimization to avoid being trapped in poor local minima, and the surprising ability of complex networks to generalize well despite the fact that they are usually heavily over-parameterized – namely, the number…”
BackgroundSources of Uncertainty in Supervised Machine Learning—a Statisticians’ View
Gruber, C., Schenk, P. O., Schierholz, M., Kreuter, F., & Kauermann, G. (2026) · Statistical Science · doi:10.1214/25-sts1003 · Cited by 97
Efficient Sharpness-aware Minimization for Improved Training of Neural NetworksPreprint
Du, J., Yan, H., Feng, J., Zhou, J. T., Zhen, L., Goh, R. S. M., & Tan, V. Y. F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03141 · Cited by 87
“…studies have been conducted to better understand the generalization of DNNs and to train DNNs that generalize well across the natural distribution (Keskar et al., 2017; Neyshabur et al., 2017; Chaudhari et al., 2019; Zhang et al., 2019; Wu et al., 2020; Foret et al., 2020; Zhang et al., 2021).”
BackgroundWei, S., Luo, C., & Luo, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01919 · Cited by 136
“In detail, MAR mines the weak combination via the memorization effect [1, 16, 49] that DNNs tend to first memorize simple examples before overfitting hard examples.”
Deep learning high resolution burned area mapping by transfer learning from Landsat-8 to PlanetScope
Martins, V. S., Roy, D. P., Huang, H., Boschetti, L., Zhang, H. K., & Yan, L. (2022) · Remote Sensing of Environment · doi:10.1016/j.rse.2022.113203 · Cited by 86
GLENet: Boosting 3D Object Detectors with Generative Label Uncertainty Estimation
Zhang, Y., Zhang, Q., Zhu, Z., Hou, J., & Yuan, Y. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01869-9 · Cited by 95
“As the neural network is prone to overfit to even complete random noise (Zhang et al., 2021), it is important to prevent the network from over-fitting noisy labels.”
BackgroundRefign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditions
Brüggemann, D., Sakaridis, C., Truong, P., & Gool, L. V. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00319 · Cited by 95
“It has been shown that large neural networks easily overfit to label noise, which deteriorates their generalization performance [1, 69].”
BackgroundLearning quadrotor dynamics for precise, safe, and agile flight control
Saviolo, A., & Loianno, G. (2023) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2023.03.009 · Cited by 98
Defect graph neural networks for materials discovery in high-temperature clean-energy applications
Witman, M. D., Goyal, A., Ogitsu, T., McDaniel, A. H., & Lany, S. (2023) · Nature Computational Science · doi:10.1038/s43588-023-00495-2 · Cited by 91
Short-term air quality prediction based on EMD-transformer-BiLSTM
Dong, J., Zhang, Y., & Hu, J. (2024) · Scientific Reports · doi:10.1038/s41598-024-67626-1 · Cited by 79
IVUS-Net: An Intravascular Ultrasound Segmentation Network
Ji, Y., Tong, L., Faraji, M., & Basu, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-04375-9_31 · Cited by 88
Cognition-Driven Structural Prior for Instance-Dependent Label Transition Matrix Estimation
Zhang, R., Cao, Z., Yang, S., Si, L., Sun, H., Xu, L., & Sun, F. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3347633 · Cited by 86
“Most crucially, it raises a challenge that deep neural networks easily overfit noisy labels, resulting in terrible accuracy degradation [9], [10], [11], [12], [13].”
Xiong, W., Long, X., Bordas, S., & Jiang, C. (2024) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2024.117681 · Cited by 77
A Systematic Review of Adversarial Machine Learning Attacks, Defensive Controls and Technologies
Malik, J., Muthalagu, R., & Pawar, P. M. (2024) · IEEE Access · doi:10.1109/access.2024.3423323 · Cited by 76
Classification and Grading of Harvested Mangoes Using Convolutional Neural Network
Iqbal, H. M. R., & Hakim, A. (2022) · International Journal of Fruit Science · doi:10.1080/15538362.2021.2023069 · Cited by 81
“Feature extraction will be improved using the method presented by (Zhang et al., 2021) and relevant techniques.”
MethodClient Selection for Federated Learning With Label Noise
Yang, M., Qian, H. L., Wang, X., Zhou, Y., & Zhu, H. (2021) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2021.3131852 · Cited by 86
Pedestrian safety using the Internet of Things and sensors: Issues, challenges, and open problems
Hasan, R., & Hasan, R. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.03.036 · Cited by 85
Detecting Multivariate Time Series Anomalies with Zero Known Label
Zhou, Q., Chen, J., Liu, H., He, S., & Meng, W. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25623 · Cited by 105
“Meanwhile, it is already verified that model training procedure is prone to overfitting noisy labels (Zhang et al. 2021), so that the performance of those OCC-based methods could be severely degraded (Wang et al. 2019; Huyan et al. 2021).”
BackgroundA Primer on Bayesian Neural Networks: Review and Debates
Arbel, J., Pitas, K., Vladimirova, M., & Fortuin, V. (2026) · Statistical Science · doi:10.1214/24-sts969 · Cited by 90
Brain tumour detection using machine and deep learning: a systematic review
Rasool, N., & Bhat, J. I. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19333-2 · Cited by 99
Cruz, B. G. S., Husch, A., & Hertel, F. (2023) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2023.1216163 · Cited by 90
“conventional wisdom that attributes this small generalization error to properties of the model family or regularization techniques used during training (Zhang et al., 2021), it has been demonstrated that even with explicit regularization, state-of-the-art convolutional networks can fit random labeling of the training…”
BackgroundXygkou, A., Ang, C. S., Siriaraya, P., Kopecki, J. P., Covaci, A., Kanjo, E., & She, W. J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642538 · Cited by 90
“This specialization, while benefcial in certain contexts, can limit the AI’s ability to engage in broader, more general interactions [75], [18], [51].”
BackgroundscTab: Scaling cross-tissue single-cell annotation models
Fischer, F., Fischer, D. S., Mukhin, R., Isaev, A., Biederstedt, E., Villani, A.-C., & Theis, F. J. (2024) · Nature Communications · doi:10.1038/s41467-024-51059-5 · Cited by 59
“Due to their high model capacity, deep learning models are known to easily over fi t the training data and can even fi t random labels 36 .”
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00567 · Cited by 70
“It is clear that neural networks are capable of extensive memorization: They can memorize randomly labeled examples (Zhang et al., 2021a) and can reveal training data when subjected to adversarial attacks (Shokri et al., 2017; Carlini et al., 2019, 2021, 2023).”
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2021) · arXiv · doi:10.48550/arxiv.2111.09509 · Cited by 70
“It is clear that neural networks are capable of extensive memorization: They can memorize randomly labeled examples (Zhang et al., 2021a) and can reveal training data when subjected to adversarial attacks (Shokri et al., 2017; Carlini et al., 2019, 2021, 2023).”
On the Implicit Bias in Deep-Learning Algorithms
Vardi, G. (2023) · Communications of the ACM · doi:10.1145/3571070 · Cited by 155
“However, in practice neural networks seem to generalize well even when trained without such an explicit regularization [84], and hence the success of deep learning cannot be attributed to explicit (1)Neural networks are trained using gradient-based algorithms, where the network’s parameters are randomly initialized, a…”
BackgroundInfluentialManiriho, P., Mahmood, A. N., & Chowdhury, M. J. M. (2024) · ACM Computing Surveys · doi:10.1145/3638240 · Cited by 74
“Accordingly, the studies presented in [172, 253] have revealed that the network architectures of DL models can be prone to mem-orization problems, where the model can end up memorizing the entire dataset resulting in an overfitted model.”
BackgroundCombating Noisy Labels with Sample Selection by Mining High-Discrepancy Examples
Xia, X., Han, B., Zhan, Y., Yu, J., Gong, M., Gong, C., & Liu, T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00176 · Cited by 110
Kordani, M., Nikoo, M. R., Fooladi, M., Ahmadianfar, I., Nazari, R., & Gandomi, A. H. (2024) · Journal of Hydrologic Engineering · doi:10.1061/jhyeff.heeng-6262 · Cited by 70
A Deep Learning Framework for Detection of COVID-19 Fake News on Social Media Platforms
Tashtoush, Y. M., Alrababah, B., Darwish, O., Maabreh, M., & Alsaedi, N. (2022) · Data · doi:10.3390/data7050065 · Cited by 79
Adversarial interference and its mitigations in privacy-preserving collaborative machine learning
Usynin, D., Ziller, A., Makowski, M. R., Braren, R., Rueckert, D., Glocker, B., Kaissis, G. A., & Passerat‐Palmbach, J. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00390-3 · Cited by 72
A Task-Adaptive Deep Reinforcement Learning Framework for Dual-Arm Robot Manipulation
Cui, Y., Xu, Z., Zhong, L., Xu, P., Shen, Y., & Tang, Q. (2024) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2024.3352584 · Cited by 80
“Furthermore, owing to the generalization capabilities of deep neural networks [22], the motion policy trained using this framework can handle tasks with varying configurations.”
Mrabah, N., Bouguessa, M., & Ksantini, R. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2997772 · Cited by 82
Green AI: Enhancing Sustainability and Energy Efficiency in AI-Integrated Enterprise Systems
Dash, S. (2025) · IEEE Access · doi:10.1109/access.2025.3532838 · Cited by 67
“However, these methods often lead to notable declines in model accuracy and struggle to generalize across a variety of AI architectures [10].”
Overparameterized neural networks implement associative memory
Radhakrishnan, A., Belkin, M., & Uhler, C. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2005013117 · Cited by 108
Intelligent Fault Diagnosis With Noisy Labels via Semisupervised Learning on Industrial Time Series
Cheng, C., Liu, X., Zhou, B., & Yuan, Y. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3229130 · Cited by 73
“Previous study [5] demonstrated that noisy samples impose higher losses at early training epochs, enabling for the identification of clean and noisy samples based on the loss distributions (see GMM 1 and GMM 2 in Fig.”
BackgroundXiang, H., Shen, J., Yan, Q., Xu, M., Shi, X., & Zhu, X. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102890 · Cited by 70
Pseudo-Bag Mixup Augmentation for Multiple Instance Learning-Based Whole Slide Image Classification
Liu, P., Ji, L., Zhang, X., & Feng, Y. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3351213 · Cited by 80
Scaling and renormalization in high-dimensional regression
Atanasov, A., Zavatone-Veth, J. A., & Pehlevan, C. (2026) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ae4bba · Cited by 81
Hu, L., Shi, W., Hu, W., Chai, W., Hu, Z., Wu, J., & Li, X. (2025) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2025.115606 · Cited by 67
Zhang, J., Xie, J., & Barnes, N. M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_21 · Cited by 72
Semi-Supervised Domain Adaptation with Source Label Adaptation
Yu, Y.-C., & Lin, H.-T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02308 · Cited by 64
“With regard to the present deep neural network design [7], the aforementioned issue could worsen as deep models have the capability to fit the data set in a seemingly random manner, regardless of the quality of the labels [34].”
BackgroundNovel applications of Convolutional Neural Networks in the age of Transformers
Ersavas, T., Smith, M. A., & Mattick, J. S. (2024) · Scientific Reports · doi:10.1038/s41598-024-60709-z · Cited by 71
Scalable Neural Network Decoders for Higher Dimensional Quantum Codes
Breuckmann, N. P., & Ni, X. (2017) · Quantum · doi:10.22331/q-2018-05-24-68 · Cited by 68
Provable Tradeoffs in Adversarially Robust Classification
Dobriban, E., Hassani, H., Hong, D. K., & Robey, A. (2023) · IEEE Transactions on Information Theory · doi:10.1109/tit.2022.3205449 · Cited by 82
Robust Data Driven Analysis for Electricity Theft Attack-Resilient Power Grid
Khan, I. U., Javaid, N., Taylor, C. J., & Ma, X. (2022) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2022.3162391 · Cited by 64
“With the adoption of modern regularization techniques such as batch normalization, early stopping and weight decay to penalize large weights, the effective Rademacher complexity of the possible solutions is dramatically reduced [32]. a).”
MethodCurriculum learning for human compositional generalization
Dekker, R., Otto, F., & Summerfield, C. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2205582119 · Cited by 90
An embedded deep-clustering-based load profiling framework
Eskandarnia, E., Al-Ammal, H., & Ksantini, R. (2021) · Sustainable Cities and Society · doi:10.1016/j.scs.2021.103618 · Cited by 66
Memorisation versus Generalisation in Pre-trained Language Models
Tänzer, M., Ruder, S., & Rei, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.521 · Cited by 98
“Zhang et al. (2021) also show that our current understanding of statistical learning theory cannot explain the super-human generalisation performance of large neural models across many areas of study.”
BackgroundTowards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Models
Zhang, Z., Sheng, X.-R., Zhang, Y., Jiang, B., Han, S., Deng, H., & Zheng, B. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557479 · Cited by 84
“Zhang et al. [29] observe that conventional generalization bounds are inadequate for over-parameterized DNN. Bartlett et al. [1] present a margin-based generalization bound for neural networks that scale with their margin-normalized spectral complexity.”
BackgroundBayesian Invariant Risk Minimization
Yong, L., Dong, H., Wang, H., & Zhang, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01555 · Cited by 106
“Unfortunately, such an overfitting phenomenon is common for deep neural networks [54].”
BackgroundResultMulti-Source Uncertainty Mining for Deep Unsupervised Saliency Detection
Wang, Y., Zhang, W., Wang, L., Liu, T., & Lu, H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01143 · Cited by 74
Jiménez-Mesa, C., Arco, J. E., Martínez-Murcia, F. J., Suckling, J., Ramírez, J., & Górriz, J. M. (2023) · Pharmacological Research · doi:10.1016/j.phrs.2023.106984 · Cited by 67
“Precisely, the capacity of NN architectures to generalise is currently under discussion [183,184], as their complexity increases, they may even learn from corrupted data (permutation of data labels) [185].”
Pei, T., & Qiu, T. (2024) · Acta Geotechnica · doi:10.1007/s11440-024-02384-y · Cited by 63
“Importantly, these problems remain undetected during the development phase due to dataset limitations [104].”
Cao, Y., & Huang, X. (2022) · Remote Sensing of Environment · doi:10.1016/j.rse.2022.113371 · Cited by 69
A Comprehensive Defense Framework Against Model Extraction Attacks
Jiang, W., Li, H., Xu, G., Zhang, T., & Lu, R. (2023) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2023.3261327 · Cited by 76
Uniquely human intelligence arose from expanded information capacity
Cantlon, J. F., & Piantadosi, S. T. (2024) · Nature Reviews Psychology · doi:10.1038/s44159-024-00283-3 · Cited by 75
“A growing area of computational modelling seeks to understand so-called resource rational models, which formalize optimal or efficient use of finite mental resources such as memory or attention 265,266 .”
InfluentialDeep learning-based credit card fraud detection in federated learning
Reddy, V. V. K., Reddy, R. V. K., Munaga, M. S. K., Karnam, B., Maddila, S. K., & Kolli, C. S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124493 · Cited by 75
Accelerating battery innovation: AI-powered molecular discovery
Gao, Y., Chen, X., Yuan, Y., Chen, Y., Niu, Y.-L., Yao, N., Gao, Y.-B., Li, W. Z., & Zhang, Q. (2025) · Chemical Society Reviews · doi:10.1039/d5cs00053j · Cited by 68
An Unconstrained Layer-Peeled Perspective on Neural CollapsePreprint
Ji, W., Lu, Y., Zhang, Y., Deng, Z., & Su, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02796 · Cited by 79
“Moreover, neural networks are found to perform well in the absence of weight decay, which highlights the importance of implicit regularization (Zhang et al., 2021).”
BackgroundSELC: Self-Ensemble Label Correction Improves Learning with Noisy Labels
Lu, Y., & He, W. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/455 · Cited by 81
Learn What You Want to Unlearn: Unlearning Inversion Attacks against Machine Unlearning
Hu, H., Wang, S., Dong, T., & Xue, M. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00248 · Cited by 101
“Machine learning models can be regarded as mappings of their training data [19], [20].”
BackgroundLearning From AI-Generated Annotations for Medical Image Segmentation
Song, Y., Liu, Y., Lin, Z., Zhou, J., Li, D., Zhou, T., & Leung, M.-F. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3474037 · Cited by 79
“1, annotation errors can lead to a serious performance degradation of CNNs. Recent studies [13], [14], [15], [16] reveal that this might be attributed to the so-called ‘memorization effect’, i.e. , CNNs memorized both correct and erroneous annotations during training.”
Wei, F., Tan, L., Ma, X., Xiao, H., Patel, D., Lee, C.-G., & Yang, L. (2024) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2024.112171 · Cited by 61
Weihs, B. J., Heuschele, D. J., Tang, Z., York, L. M., Zhang, Z., & Xu, Z. (2024) · Plant Phenomics · doi:10.34133/plantphenomics.0178 · Cited by 61
“In addition, the issue of overparameterization is a new challenge brought about by DL that results from a model having a large set of learnable parameters yet has insufficient training data to overcome generalization problems expressed as overfitting [72].”
BackgroundHigh-resolution sensors and deep learning models for tree resource monitoring
Brandt, M., Chave, J., Li, S., Fensholt, R., Ciais, P., Wigneron, J., Gieseke, F. C., Saatchi, S. S., Tucker, C. J., & Igel, C. (2024) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-024-00116-8 · Cited by 73
“In practice, it is a common strategy to choose a neural network architecture that is large enough to be able to easily learn (a subset of) the training data with (almost) zero errors 68 , which in deep learning need not imply bad generalization 69 .”
Simple Attention Module Based Speaker Verification with Iterative Noisy Label Detection
Qin, X., Li, N., Weng, C., Su, D., & Li, M. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746294 · Cited by 91
Shahvandi, M. K., & Soja, B. S. (2022) · Advances in Space Research · doi:10.1016/j.asr.2022.05.042 · Cited by 62
The Importance of Understanding Deep Learning
Räz, T., & Beisbart, C. (2024) · Erkenntnis · doi:10.1007/s10670-022-00605-y · Cited by 46
“However, the reasons why these DNNs generalize well is only insufficiently understood (Zhang et al., 2017; Shwartz-Ziv & Tishby, 2017; Alain & Bengio, 2016; Zhang et al., 2021; Berner et al., 2021).”
BackgroundSoK: Model Inversion Attack Landscape: Taxonomy, Challenges, and Future Roadmap
Dibbo, S. V. (2023) · IEEE Computer Security Foundations Workshop (CSFW) · doi:10.1109/csf57540.2023.00027 · Cited by 78
“Also, an adversary can adapt techniques like an augmented synthetic dataset to query the model, which outputs prediction as memorized information from training data [31].”
MethodDeep learning in fringe projection: A review
Liu, H., Yan, N., Shao, B., Yuan, S., & Zhang, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127493 · Cited by 68
Generalization Bounds: Perspectives from Information Theory and PAC-Bayes
Hellström, F., Durisi, G., Guedj, B., & Raginsky, M. (2025) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000112 · Cited by 98
Yang, S., Xu, Z.-P., Wang, K., You, Y., Yao, H.-X., Liu, T.-L., & Xu, M. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.01904 · Cited by 85
“The memorization effect of deep neural networks [56] reveals that DNNs would first memorize training data of clean labels then those of noisy labels.”
BackgroundKaya, M. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105472 · Cited by 65
You, W., Shen, C., Guo, X., Jiang, X., Shi, J., & Zhu, Z. (2017) · Advances in Mechanical Engineering · doi:10.1177/1687814017704146 · Cited by 60
LA-Net: Landmark-Aware Learning for Reliable Facial Expression Recognition under Label Noise
Wu, Z., & Cui, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01892 · Cited by 78
“Firstly, sample selection methods assume that the neural network will fit clean samples before overfitting noisy labels [1, 44], which can lead to confusion between hard samples and noisy samples.”
BackgroundExtended : Learning with Mixed Closed-set and Open-set Noisy Labels
Xia, X., Han, B., Wang, N., Deng, J., Li, J., Mao, Y., & Liu, T. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3180545 · Cited by 82
Model-Heterogeneous Semi-Supervised Federated Learning for Medical Image Segmentation
Ma, Y., Wang, J., Yang, J., & Wang, L. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3348982 · Cited by 55
“The assigned pseudo-labels may carry the inherent risk of overconfidence [24], [25].”
BackgroundHe, X., Sun, S., Lin, W. L., Công, P. T., Abu-Rumman, A., & Halteh, K. (2023) · Economic Analysis and Policy · doi:10.1016/j.eap.2023.04.017 · Cited by 55
Wang, J., Zheng, N., Wan, H., Yao, Q., Jia, S., Zhang, X., Fu, S., Ruan, J., He, G., Chen, X., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00085-2 · Cited by 65
Multi-Label Noise Robust Collaborative Learning for Remote Sensing Image Classification
Aksoy, A., Ravanbakhsh, M., & Demir, B. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3209992 · Cited by 61
When Do We Not Need Larger Vision Models?
Shi, B., Wu, Z., Mao, M., Wang, X., & Darrell, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73242-3_25 · Cited by 98
Leveling Down in Computer Vision: Pareto Inefficiencies in Fair Deep Classifiers
Zietlow, D., Lohaus, M., Balakrishnan, G., Kleindeßner, M., Locatello, F., Schölkopf, B., & Russell, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01016 · Cited by 86
Mutual Quantization for Cross-Modal Search with Noisy Labels
Yang, E., Yao, D., Liu, T., & Deng, C. D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00740 · Cited by 72
“As deep networks have large model capacities, they can easily memorize and eventually overfit these noisy labels, which correspondingly de-generates the model generalization [51].”
BackgroundImproved deep learning method for accurate flow field reconstruction from sparse data
Zhang, J., Liu, J., & Huang, Z. (2023) · Ocean Engineering · doi:10.1016/j.oceaneng.2023.114902 · Cited by 56
Jiang, S., Zhou, Y., Xu, W., Xia, Q. X., Yi, M., & Cheng, X. (2024) · Chemical Engineering Journal · doi:10.1016/j.cej.2024.150297 · Cited by 55
HGNN−BRFE: Heterogeneous Graph Neural Network Model Based on Region Feature Extraction
Zhao, Y., Xu, S., & Duan, H. (2024) · Electronics · doi:10.3390/electronics13224447 · Cited by 56
Machine learning robustness: a primer
Braiek, H. B., & Khomh, F. (2025) · Elsevier eBooks · doi:10.1016/b978-0-44-323761-4.00012-2 · Cited by 78
Separation of scales and a thermodynamic description of feature learning in some CNNs
Seroussi, I., Naveh, G., & Ringel, Z. (2023) · Nature Communications · doi:10.1038/s41467-023-36361-y · Cited by 92
Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions
Bhattamishra, S., Patel, A., Kanade, V., & Blunsom, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.317 · Cited by 84
Cross-to-merge training with class balance strategy for learning with noisy labels
Zhang, Q., Zhu, Y., Yang, M., Jin, G., Zhu, Y., & Chen, Q. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123846 · Cited by 57
Liu, S., Chen, J., Feng, Y., Xie, Z., Pan, T., & Xie, J. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124511 · Cited by 63
Examining the challenges of blood pressure estimation via photoplethysmogram
Mehta, S., Kwatra, N., Jain, M., & McDuff, D. (2024) · Scientific Reports · doi:10.1038/s41598-024-68862-1 · Cited by 57
Real-time data visual monitoring of triboelectric nanogenerators enabled by Deep learning
Zhang, H., Liu, T., Zou, X., Zhu, Y., Chi, M., Wu, D., Jiang, K., Zhu, S., Zhai, W., Wang, S., et al. (2024) · Nano Energy · doi:10.1016/j.nanoen.2024.110186 · Cited by 52
Generalization in neural networks: A broad survey
Rohlfs, C. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128701 · Cited by 68
DRPCA-Net: Make Robust PCA Great Again for Infrared Small Target Detection
Xiong, Z.-H., Zhou, F., Wu, F.-Y., Yuan, S., Fu, M.-X., Peng, Z.-M., Yang, J., & Dai, Y.-M. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3588392 · Cited by 67
Label-Free Multivariate Time Series Anomaly Detection
Zhou, Q., He, S., Liu, H., Chen, J., & Meng, W. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3349613 · Cited by 67
“Using these training datasetswillresultinoverfittingofthemodeltonoisylabels[19].”
Human Creativity in the AIGC Era
Lou, Y. (2023) · She ji · doi:10.1016/j.sheji.2024.02.002 · Cited by 50
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.