Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 901–1,000 of 2,667
Honarvar, H., Agarwal, C., Somani, S. S., Vaid, A., Lampert, J. M., Wanyan, T., Reddy, V. Y., Nadkarni, G. N., Miotto, R., Žitnik, M., et al. (2022) · Cardiovascular Digital Health Journal · doi:10.1016/j.cvdhj.2022.07.074 · Cited by 11
“We clarify that the sub-waveform representation is solely a form of input-data transformation and to investigate its impact on learning, we intentionally kept all parameters of NN fixed.(51) Therefore, any gain is mainly owing to the sub-waveform representation.”
MethodGao, X. W., Taylor, S., Pang, W., Hui, R., Lü, X., & Braden, B. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.023 · Cited by 9
“In addition, the performance of a developed system is dependent on its ability for generalisation [19, 20].”
Improving performance of medical image alignment through super-resolution
Deng, L., Zhang, Y., Wang, J., Huang, S., & Yang, X. N. (2023) · Biomedical Engineering Letters · doi:10.1007/s13534-023-00268-w · Cited by 9
“[15] notes, the reliability of such pre-trained networks are vulnerable to patient-specific features.”
BackgroundMethodTowards Characterizing the Semantic Robustness of Face Recognition
Pérez, J. C., Alfarra, M., Thabet, A., Arbeláez, P., & Ghanem, B. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00037 · Cited by 12
“The unexpected vulnerability of DNNs against adversarial attacks highlights our narrow understanding of these models and their limitations [22,69].”
BackgroundUnraveling the complexities of pathological voice through saliency analysis
Shaikh, A. A. S., Bhargavi, M. S., & Naik, G. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107566 · Cited by 9
Supervised contrastive learning with corrected labels for noisy label learning
Ouyang, J., Lu, C., Wang, B., & Li, C. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05018-0 · Cited by 9
Three-teaching: A three-way decision framework to handle noisy labels
Chao, G., Zhang, K., Wang, X., & Chu, D. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111400 · Cited by 11
Theory II: Deep learning and optimization
Poggio, T., & Liao, Q. (2018) · Bulletin of the Polish Academy of Sciences Technical Sciences · doi:10.24425/bpas.2018.125925 · Cited by 9
Learning Invariant Feature Representation to Improve Generalization Across Chest X-Ray Datasets
Ghimire, S., Kashyap, S., Wu, J. T.-Y., Karargyris, A., & Moradi, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59861-7_65 · Cited by 11
Towards State-Aware Computation in ReRAM Neural Networks
He, Y., Wang, Y., Zhao, X., Li, H., & Li, X. (2020) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac18072.2020.9218729 · Cited by 13
On the Self-Penalization Phenomenon in Feature SelectionPreprint
Jordan, M. I., Liu, K., & Ruan, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05852 · Cited by 9
Fundamental Limits of Membership Inference Attacks on Machine Learning ModelsPreprint
Aubinais, E., Gassiat, É., & Piantanida, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13786 · Cited by 10
“This potential disclosure of sensitive information occurs as a result of various factors inherent to the training procedure, which include the extraction of patterns, associations, and subtle correlations from the data [Song et al., 2017a, Zhang et al., 2021].”
MethodFeng, X., Luo, H., Wang, C., & Gu, H. (2023) · Mathematical Geosciences · doi:10.1007/s11004-023-10094-6 · Cited by 8
“Zhang et al. (2021a) proved that a sufficiently deep network can fit any label even if that label is generated randomly.”
Quantifying impairment and disease severity using AI models trained on healthy subjects
Yu, B., Kaku, A., Liu, K., Parnandi, A., Fokas, E. E., Venkatesan, A., Pandit, N. G., Ranganath, R., Schambra, H. M., & Fernandez‐Granda, C. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01173-x · Cited by 10
“This is an instance of a general challenge inherent to the use of deep neural networks: these models are so fl exible that they can easily learn spurious structure in high-dimensional data 42,54 .”
An adaptive approach to noisy annotations in scientific information extraction
Bölücü, N., Rybinski, M., Dai, X., & Wan, S. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103857 · Cited by 11
Sun, B., Gan, W., Ma, R., Feng, P., & Chu, J. (2024) · Sensors and Actuators A Physical · doi:10.1016/j.sna.2024.115978 · Cited by 8
Knowledge distillation with adapted weight
Wu, S., Luo, X., Liu, J., & Deng, Y. (2025) · Statistics · doi:10.1080/02331888.2025.2451944 · Cited by 9
“Zhang et al. [22] demonstrated that standard Convolutional Neural Networks (CNNs) can overfit noisy labels, resulting in poor generalization.”
Zhao, X., Du, Y., & Peng, Y. (2025) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2025.102508 · Cited by 8
Stealing Training Graphs from Graph Neural Networks
Lin, M., Dai, E., Xu, J., Jia, J., Zhang, X. D., & Wang, S. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709289 · Cited by 8
“However, neural networks can memorize the training data, even when they have great generalization ability [68].”
Agent-in-the-loop to distill expert knowledge into artificial intelligence models: a survey
Gao, J., Zhang, Y., Chen, Y., Dong, Y., Chen, Y., Song, S., Tang, B., & Gu, Y. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11255-1 · Cited by 10
Xiong, X., Cai, L., Yang, Z., Cao, Z., Wu, N., & Ni, Q. (2025) · Frontiers in Oncology · doi:10.3389/fonc.2025.1571193 · Cited by 9
Chang, Y., Xu, Q., Huang, B., Zhang, X., Yu, H., Chen, H., Thévenin, D., & Guo, L. (2025) · Energy · doi:10.1016/j.energy.2025.137026 · Cited by 8
Data-driven discovery of dynamical models in biology
Prokop, B., & Gelens, L. (2026) · Nature Reviews Physics · doi:10.1038/s42254-026-00955-4 · Cited by 12
“• Limited generalization: generalization beyond the training domain is weak 147 , particularly with noisy or undersampled biological data.”
Hassoon, A., Lin, C. F., Woo, H., Irimia, R. M., Marsteller, J. A., Li, A., Banderia, A., Leo, H., Peng, X., Rastall, D. P. W., et al. (2025) · Annals of Epidemiology · doi:10.1016/j.annepidem.2025.11.004 · Cited by 12
Clean-sample guided cross-modal retrieval with adaptive weighted contrastive learning
Jiang, S., & Li, Z. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112383 · Cited by 9
Wang, J., Li, W., Wang, Z., Ayas, S., Donmez, B., He, D., & Wu, K. (2026) · Accident Analysis & Prevention · doi:10.1016/j.aap.2026.108407 · Cited by 10
AI-Driven Hybrid Architecture for Secure, Reconstruction-Resistant Multi-Cloud Storage
Ahmed, M., & Yuan, J.-S. (2026) · Future Internet · doi:10.3390/fi18020070 · Cited by 8
“These patterns are consistent with findings in hybrid-learning research demonstrating that combining real and synthetic signals mitigates overfitting and strengthens operational reliability [34].”
Li, H., Zhang, J., Gao, Q., Li, X., & Yang, Z. (2022) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2022.110826 · Cited by 8
Probing the Purview of Neural Networks via Gradient Analysis
Lee, J., Lehman, C., Prabhushankar, M., & AlRegib, G. (2023) · IEEE Access · doi:10.1109/access.2023.3263210 · Cited by 12
“This notion of capacity, defined as the types of functions that can be reached via some learning algorithm, is denoted as the effective capacity [14], [15].”
BackgroundSoK: Membership Inference is Harder Than Previously Thought
Dionysiou, A., & Athanasopoulos, E. (2023) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2023-0082 · Cited by 14
“Nonetheless, deep ANNs tend to memorize the training data because they have the capacity to do so [50, 71, 82].”
BackgroundTrustworthy artificial intelligence in healthcare
Subaşı, A., Özaltın, Ö., Mitra, A., Subasi, M. E., & Sarirete, A. (2023) · Elsevier eBooks · doi:10.1016/b978-0-443-15299-3.00015-4 · Cited by 9
SPORT: A Subgraph Perspective on Graph Classification with Label Noise
Yin, N., Shen, L., Chen, C., Hua, X., & Luo, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3687468 · Cited by 18
“Note that neural networks cannot generalize to neighborhoods of the memorized samples [78, 86], which indicates that even if training losses of noisy samples are small in some cases, they can still be seriously increased under random perturbations.”
InfluentialApplying statistical learning theory to deep learning
Gerbelot, C., Karagulyan, A., Karp, S., Ravichandran, K., Stern, M., & Srebro, N. (2024) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ad3a5f · Cited by 8
Hyperparameter Optimization in Generative Adversarial Networks (GANs) Using Gaussian AHP
Rodrigues, T. S., & Pinheiro, P. R. (2024) · IEEE Access · doi:10.1109/access.2024.3518979 · Cited by 12
Combating Label Noise with a General Surrogate Model for Sample Selection
Liang, C., Zhu, L., Shi, H., & Yang, Y. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02324-z · Cited by 8
“Learning with noisy labels suffers from the adverse effect that deep neural networks can easily memorize noisy samples (Zhang et al., 2021).”
Meta-Learning-Based Semi-Supervised Change Detection in Remote Sensing Images
Tang, Y. P., Zhang, L., Zhang, W., & Jiang, Z. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3527483 · Cited by 14
“Tobalance the accuracy and efficiency of the model, the meta-learning method employs stochastic gradient descent (SGD) [26], [27], [28], [29] in the inner loop and adaptive moment estimation weight decline (AdamW) [30] in the outer loop.”
Nasef, D., Nasef, D., Basco, K. J., Singh, A. R., Hartnett, C., Ruane, M., Tagliarino, J., Nizich, M., & Toma, M. (2025) · AI · doi:10.3390/ai6030059 · Cited by 8
“Despite their potential, ML models face critical issues such as overfitting [17], poor generalization to unseen data [18], and a lack of interpretability [19], particularly in clinical settings.”
Li, Y., Gao, Y., Yang, B., Lei, Y., Li, X., Shu, Y., & Feng, K. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129998 · Cited by 8
Chen, B., Guo, R., Zhang, Q., Zhao, Y., Wang, X., & Zhu, Z. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3549150 · Cited by 10
“Additionally, as the extrapolation prediction horizon extends, neural network models exhibit a superlinear increase in simulation errors [27].”
The Future of Medicine: AI and ML Driven Drug Discovery Advancements
Patel, D. D., Pathak, R. S., Patel, K. S., Bhatt, H., & Patel, P. K. (2025) · Current Topics in Medicinal Chemistry · doi:10.2174/0115680266346722250401191232 · Cited by 10
Wasserstein GAN for moving differential privacy protection
Liu, E., Chu, Z., & Zhang, X. (2025) · Scientific Reports · doi:10.1038/s41598-025-03178-2 · Cited by 9
Germaine, M., O'Higgins, A. C., Egan, B., & Healy, G. F. (2025) · JMIR Medical Informatics · doi:10.2196/72938 · Cited by 12
“Training ML models on unvalidated EHRs may lead to systematic errors in the model output with the potential for the model to miss, underestimate, or overestimate clinically significant relationships [8,9].”
NM-SpMM: Accelerating Matrix Multiplication Using N:M Sparsity with GPGPU
Ma, C., Wu, D., Deng, Z., Chen, J., Huang, X., Meng, J., Zhu, W., Wang, B., Zhou, A. C., Chen, P., et al. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00087 · Cited by 8
“However, artificial deep learning models are traditionally dense and over-parameterized [6]–[8], leading to significant consumption of computing and memory resources during actual real-world deployment.”
Noise-free prototype guided representation calibration under label noise
Yuan, H., Luo, T., Wu, X., & Jiang, J. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114308 · Cited by 10
UA-PDFL: A personalized approach for decentralized federated learning
Zhu, H., Fan, Y., & Xie, Z. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131579 · Cited by 8
Dyson equation for correlated linearizations and test error of random features regression
Latourelle-Vigeant, H., & Paquette, E. (2025) · Random Matrices Theory and Application · doi:10.1142/s2010326325500261 · Cited by 10
Critical properties of the SAT/UNSAT transitions in the classification problem of structured data
Pastore, M. (2021) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ac312b · Cited by 7
“Despite their elegance and generality, however, these data-independent results are usually of little interest in practice, as the bounds they provide are often quite loose in the regime relevant for applications [4, 5].”
Generalization Performance of Empirical Risk Minimization on Over-Parameterized Deep ReLU Nets
Lin, S.-B., Wang, Y., & Zhou, D. (2025) · IEEE Transactions on Information Theory · doi:10.1109/tit.2025.3531048 · Cited by 10
“The great success of deep learning brings several challenges in understanding its running mechanism, including the landscape analysis [5], convergence analysis [6], benign over-fitting verification [7], and generalization optimality demonstration [8].”
Universal mean-field upper bound for the generalization gap of deep neural networks
Ariosto, S., Pacelli, R., Ginelli, F., Gherardi, M., & Rotondo, P. (2022) · Physical review. E · doi:10.1103/physreve.105.064309 · Cited by 8
IDEAL: High-Order-Ensemble Adaptation Network for Learning with Noisy Labels
Zhang, P.-F., Huang, Z. H., Bai, G., & Xu, X.-S. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548053 · Cited by 9
“Extensive studies have shown that DNNs are prone to overfit to training data even with corrupted labels, resulting in significant performance deterioration [18, 22, 32, 40].”
BackgroundRobust Federated Learning with Local Mixed Co-teaching
Ejigu, G. F., Hong, S. H., & Hong, C. S. (2023) · International Conference on Information Networking · doi:10.1109/icoin56518.2023.10049004 · Cited by 11
Information-Theoretic Analysis of Minimax Excess Risk
Hafez-Kolahi, H., Moniri, B., & Kasaei, S. (2023) · IEEE Transactions on Information Theory · doi:10.1109/tit.2023.3249636 · Cited by 8
Migliorelli, L., Massini, L. S., Coccia, M., Villani, L., Frontoni, E., & Squartini, S. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107840 · Cited by 8
LLMs are not ready for editorial work
Lindsay, G. W. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01730-6 · Cited by 8
Prototypical Mixing and Retrieval-based Refinement for Label Noise-resistant Image Retrieval
Yang, X., Wang, H., Sun, J., Zhang, S., Chen, C., Hua, X., & Luo, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01032 · Cited by 10
“Although deep neural networks tend to learn generalized patterns at the beginning stage, the overfitting or memorization of noisy data would compromise the optimization process gradually [57].”
BackgroundA simple probabilistic neural network for machine understanding
Xie, R., & Matteo Marsili, A. (2024) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ad0a8c · Cited by 8
“The accuracy in deep neural networks can increase with complexity (i.e. with the number of parameters), without overfitting [3, 4].”
BackgroundJantre, S. R., Urban, N. M., Qian, X., & Yoon, B.-J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448265 · Cited by 15
Alqahtany, S. S., & Syed, T. A. (2024) · Applied Sciences · doi:10.3390/app14114421 · Cited by 8
Learning with Noisy Labels through Learnable Weighting and Centroid Similarity
Wani, F. A., Bucarelli, M. S., & Silvestri, F. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650366 · Cited by 10
Understanding the Initial Condensation of Convolutional Neural Networks
Zhou, Z., Zhou, H., Li, Y., & Xu, Z.-Q. J. (2025) · CSIAM Transactions on Applied Mathematics · doi:10.4208/csiam-am.so-2024-0011 · Cited by 8
“As large neural networks continue to demonstrate impressive performance in numerous practical tasks, a key challenge has come to understand the reasons behind the strong generalization capabilities often exhibited by over-parameterized networks (Breiman, 1995; Zhang et al., 2021).”
Intervention & Interaction Federated Abnormality Detection with Noisy Clients
Liu, X., Li, W., & Yuan, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16452-1_30 · Cited by 16
Minimum perturbation theory of deep perceptual learning
Shan, H., & Sompolinsky, H. (2022) · Physical review. E · doi:10.1103/physreve.106.064406 · Cited by 8
Li, H., Yu, W., & Huang, H. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.06.031 · Cited by 8
Lecué, G., & Shang, Z. (2024) · Probability Theory and Related Fields · doi:10.1007/s00440-024-01336-7 · Cited by 14
“One side of the ’deep learning experiment’ that has received a lot of attention from the statistical community during the last five years is about an unexpected behavior of some deep learning models that perfectly fit (potentially noisy) data but can still generalize well [5, 45, 44].”
BackgroundUnderstanding the Representational Power of Neural Retrieval Models Using NLP Tasks
Cohen, D., O'Connor, B., & Croft, W. B. (2018) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3234944.3234959 · Cited by 11
Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear RegressionPreprint
Wu, J., Zou, D., Braverman, V., Gu, Q., & Kakade, S. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06198 · Cited by 8
“It is widely observed in practice that modern neural networks trained by stochastic gradient descent (SGD) often generalize well (Zhang et al., 2021).”
Characterizing uncertainty in predictions of genomic sequence-to-activity modelsPreprint
Bajwa, A., Rastogi, R., Kathail, P., Shuai, R. W., & Ioannidis, N. M. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.12.21.572730 · Cited by 13
“These models may have similar losses on the training set and even test set, but have different predictions on out-of-distribution inputs, reflecting the underlying uncertainty [15].”
On the Optimal Memorization Capacity of TransformersPreprint
Kajitsuka, T., & Sato, I. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.17677 · Cited by 8
Training neural networks on data sources with unknown reliability
Capstick, A., Palermo, F., Cui, T., & Barnaghi, P. M. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103239 · Cited by 7
“The authors exploit the fact that neural networks learn clean data patterns and filter noisy instances in early training [3, 4].”
InfluentialOpen set label noise learning with robust sample selection and margin-guided module
Zhao, Y., Xia, Q., Sun, Y., Wen, Z., Ma, L., & Ying, S. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113670 · Cited by 12
Maheswaran, B., Chawla, K., Gupta, A., & Thevamaran, R. (2025) · Extreme Mechanics Letters · doi:10.1016/j.eml.2025.102346 · Cited by 6
“…BR is also a generally preferred regularization technique— particularly with smaller datasets—over traditional regularization techniques such as L1 and L2 regularization methods, that rely on cross-validated hyperparameters and dropout methods that often underfit with small datasets [41, 42].”
Banyong, C., Hantanong, N., Nanthawong, S., Se, C., Wisutwattanasak, P., Champahom, T., Ratanavaraha, V., & Jomnonkwao, S. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9060155 · Cited by 7
Customer Lifetime Value Modeling via Two Stage Selected Trees Ensembles
Khan, Z., Ali, A., Aldahmani, S., Nordmark, H., & Lausen, B. (2025) · IEEE Access · doi:10.1109/access.2025.3584709 · Cited by 7
Iqbal, A., Kong, W., Iqbal, Y., Khan, U. S., Khan, S. F., & Hussain, I. (2025) · The Visual Computer · doi:10.1007/s00371-025-04113-2 · Cited by 7
“Momentum-driven perturbations further stabilize ∇ θ L ad v by averaging gradient directions, reducing variance [45].”
Zhang, Q., Wang, W., Su, H., Yang, G., Xue, J., Hou, H., Geng, X., Cao, Q., & Xu, Z. H. (2025) · Remote Sensing · doi:10.3390/rs17162882 · Cited by 7
Zhong, M., Wei, L., & Mo, H. (2025) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1671755 · Cited by 7
“Traditional deep learning models also show clear shortcomings in generalization and classi fi cation in diverse scenarios Zhang et al. (2021).”
Luo, J., Kong, Y., & Li, G. (2025) · Journal of Business and Economic Statistics · doi:10.1080/07350015.2025.2583457 · Cited by 9
Numerical Renormalization of Glassy Dynamics
Lang, J., Subir Sachdev, A., & Diehl, S. (2025) · Physical Review Letters · doi:10.1103/z64g-nqs6 · Cited by 10
Lin, C., Tseng, T.-L., & Tsai, T.-H. (2026) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3650620 · Cited by 7
“Deep neural models are expressive and, without appropriate regularization or data coverage, can memorize noise and overfit; empirical evidence shows modern networks can even fit random labels, underscoring the need for careful regularization and validation in industrial settings [33], [34].”
Infrared Image Quality Estimation With Node-to-Graph Regression
Gu, K., Liu, H., Gao, Y.-T., Wang, C., Wong, L.-K., Lin, W., Zhai, G., Zhang, W., & Thalmann, D. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3651047 · Cited by 11
Misinformation Detection using Large Language Models with Explainability
Patel, J., Bhatt, C., Trivedi, H., & Nguyen, T. T. (2025) · International Conference on Algorithms, Computing and Artificial Intelligence (ACAI) · doi:10.1109/acai68217.2025.11406235 · Cited by 12
Quantifying Memorization Advantage in Code LLMs
Djiré, A. E., Kaboré, A. K., Samhi, J., Barr, E. T., Klein, J., & Bissyandé, T. F. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764554 · Cited by 8
“Prior work has demonstrated that neural network generalization capabilities can be evaluated through sensitivity to data perturbations [14, 61].”
Sheng, M., Sun, Z., Zhou, T., Shu, X., Pan, J., & Yao, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00092 · Cited by 17
A Generalization Gap Estimation for Overparameterized Models via the Langevin Functional Variance
Okuno, A., & Yano, K. (2023) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2023.2197488 · Cited by 7
“However, real-case studies using deep neural networks (Azulay and Weiss 2019; Zhang et al. 2021) suggest that there is a gap between the generalization performance and the training performance, a generalization gap .”
BackgroundAn improved stochastic gradient descent algorithm based on Rényi differential privacy
Cheng, X., Yao, Y., Zhang, L., Liu, A., & Li, Z. (2022) · International Journal of Intelligent Systems · doi:10.1002/int.22944 · Cited by 7
“4,5,6 Fredrikson et al. 7 implemented a model inversion attack to recover images from facial recognition systems.”
BackgroundXi, X., & Zhu, S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10347-6 · Cited by 8
“The second is that the performances on training data might not be generalized to test data or real-world data (Zhang et al. 2021).”
BackgroundAn Alternative Hard-Parameter Sharing Paradigm for Multi-Domain Learning
Zhang, L., Yang, Q., Liu, X., & Guan, H. (2023) · IEEE Access · doi:10.1109/access.2023.3240410 · Cited by 7
Noisy Label Detection and Counterfactual Correction
Qi, W., & Chelmis, C. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3271963 · Cited by 9
“Furthermore, overparameterized machine learning models, such as deep neural networks, can overfit noisy data instances by memorizing them during training [8], [9].”
BackgroundA Moment-Matching Approach to Testable Learning and a New Characterization of Rademacher Complexity
Gollakota, A., Klivans, A. R., & Kothari, P. K. (2023) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/3564246.3585206 · Cited by 34
Learning from Training Dynamics: Identifying Mislabeled Data beyond Manually Designed Features
Jia, Q., Li, X., Yu, L., Bian, J., Zhao, P., Li, S., Xiong, H., & Dou, D. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.25972 · Cited by 15
Overcoming Noisy Labels in Federated Learning Through Local Self-Guiding
Bai, D., Wang, S., Wang, W., Wang, H., Zhao, C., Yuan, P., & Chen, Z. (2023) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid57682.2023.00042 · Cited by 10
“Noisy labels will lead to degradation of model performance as deep neural network (DNN) has a huge amount of parameters to fit all training data [7].”
BackgroundMIMO Detection Under Hardware Impairments: Learning With Noisy Labels
Kwon, J., Jeon, S., Jeon, Y., & Poor, H. V. (2023) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2023.3329521 · Cited by 8
“These effects cause the DNN to be biased towards the noisy training data, resulting in overfitting that significantly degrades the generalization performance of the DNN [32].”
BackgroundHan, G., Guo, W., Zhang, H., Jin, J., Gan, X., & Zhao, X. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108489 · Cited by 9
Wang, Z., Liu, S., Guo, B., Yu, Z., & Zhang, D. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3427636 · Cited by 7
Xing, B., Ying, X., & Wang, R. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128425 · Cited by 7
Task Specific Pretraining with Noisy Labels for Remote Sensing Image Segmentation
Liu, C., Albrecht, C. M., Wang, Y., & Zhu, X. X. (2024) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss53475.2024.10641089 · Cited by 9
“Recent studies have shown that deep learning models are robust against label noise to some extent [4, 5].”
BackgroundIn Search of a Data Transformation that Accelerates Neural Field Training
Seo, J., Lee, S., Kim, K. I., & Lee, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00462 · Cited by 9
Detection of artificial spots in fundus images using modified U-Net based semantic segmentation
Parashar, A. K., & Kumar, B. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2024.109719 · Cited by 7
A Lipschitz Spaces View of Infinitely Wide Shallow Neural Networks
Bartolucci, F., Carioni, M., Iglesias, J. A., Korolev, Y. M., Naldi, E., & Vigogna, S. (2026) · SIAM Journal on Mathematical Analysis · doi:10.1137/24m1718615 · Cited by 9
RBLA: Rank-Based-LoRA-Aggregation for Fine-Tuning Heterogeneous Models in FLaaS
Chen, S., Tavallaie, O., Nazemi, N., & Zomaya, A. Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77072-2_4 · Cited by 7
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.