Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 1,301–1,400 of 2,667
Predicting Porosity in Freeze Casting with Explainable Machine Learning
de Oliveira, R. G. B., Yudi, J., da Silva, E. P., Bestard, G. A., Ribeiro, L. A., & Silva, A. M. A. (2025) · ACS Omega · doi:10.1021/acsomega.5c04133 · Cited by 5
Jeffrey, J., RajKumar, A., Pandey, S., Bathala, L., & Yalavarthy, P. K. (2025) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2025.102630 · Cited by 4
Integrating Both Driving and Response Environmental Variables to Enhance Soil Salinity Inversion
Zhou, Q., Zhang, Y., Liu, Z., Wang, D., Chen, H., & Liu, P. (2025) · Agronomy · doi:10.3390/agronomy15081995 · Cited by 4
“Research has shown that machine learning models typically require large training datasets—often comprising hundreds to thousands of samples—to achieve robust generalizability and mitigate overfitting [72].”
Buettner, R., Bertram, S., & Fischer-Brandies, L. (2025) · IEEE Access · doi:10.1109/access.2025.3604689 · Cited by 5
“This excessive complexity can negatively impact the model’s generalization and robustness capabilities, as the model fits the entire training dataset instead of learning the underlying abstract pattern [18].”
Dynamic network compression via probabilistic channel pruning
Lee, K., & Lee, H.-W. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108080 · Cited by 6
Heterogeneous Resource Allocation With Multi-Task Learning for Wireless Networks
Mitsiou, N. A., Bouzinis, P. S., Sarigiannidis, P. G., & Karagiannidis, G. K. (2025) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2025.3630058 · Cited by 4
Wang, S., Ryu, Y., Dechant, B., Zhang, H., Feng, H., Lee, J., & Choi, C. (2025) · Remote Sensing of Environment · doi:10.1016/j.rse.2025.115198 · Cited by 5
Generalization bounds in hybrid quantum-classical machine learning models
Wu, T., Bentellis, A., Sakhnenko, A., & Lorenz, J. M. (2026) · Physical Review A · doi:10.1103/6ckt-hlh8 · Cited by 7
“Several works have provided rigorous bounds on the generalization error for classical machine learning models [11–13] as well as for fully quantum learning models [1–3].”
Ye, M.-W., Huangfu, Y., Gao, S.-J., Ren, W., Liu, W.-F., & Yu, Z.-K. (2026) · IEEE journal of biomedical and health informatics · doi:10.1109/jbhi.2026.3663219 · Cited by 3
“However, replacing a potentially incorrect label with a pseudo label can be hasty in the early stages of training, as it risks reinforcing the effects of label noise rather than mitigating them [12].”
How Does Heterogeneous Label Noise Impact Generalization in Neural Nets?
Khanal, B., & Kanan, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-90436-4_18 · Cited by 6
Javadi, G., Samadi, S., Bayat, S., Sojoudi, S., Hurtado, A., Eshumani, W., Chang, S. D., Black, P. C., Mousavi, P., & Abolmaesumi, P. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-022-02707-y · Cited by 5
“Recent studies have shown that label noise can signifi-cantly impact the performance of deep learning models, and conventional regularization techniques such as data augmentation, weight decay, dropout, and batch normalization are not as effective [17,18].”
BackgroundInfluentialA Model-Agnostic Approach for Learning with Noisy Labels of Arbitrary Distributions
Hao, S., Li, P., Wu, R., & Chu, X. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00096 · Cited by 7
Agreement or Disagreement in Noise-tolerant Mutual Learning?
Liu, J., Jiang, D., Yang, Y., & Li, R. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956595 · Cited by 4
“DNNs have been proved to have a certain degree of generalization and have been leveraged to promote classification robustness [14].”
BackgroundLiu, Z., Liu, B., Zhao, Z., Chu, Q., & Yu, N. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097020 · Cited by 4
“Because of the memorization effect of CNNs [6], the noisy labels would be remembered if they are not suppressed in the early stage, which directly leads to the accumulation of negative effects during network optimization and ultimately hinders the final performance.”
BackgroundXie, G., Li, Y., Wang, Y., Li, Z., & Qu, H. (2023) · Remote Sensing · doi:10.3390/rs15122986 · Cited by 4
CCLM: Class-Conditional Label Noise Modelling
Tatjer, A., Nagarajan, B., Marques, R., & Radeva, P. I. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36616-1_1 · Cited by 4
Dong, Y., Gong, T., Chen, H., & Li, C. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/405 · Cited by 4
Investigating UD Treebanks via Dataset Difficulty Measures
Kulmizev, A., & Nivre, J. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.76 · Cited by 9
“However, a model endowed with enough parameters may learn to fit the data without necessarily compressing it — see, e.g., Zhang et al. (2021), who show that training loss can still be minimized on data that contains no inherent structure (shuffled labels).”
BackgroundFedCoop: Cooperative Federated Learning for Noisy Labels
Tam, K., Li, L., Zhao, Y. L., & Xu, C. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230529 · Cited by 10
Improving generalization in large langue model by learning prefix subspaces
Falissard, L., Guigue, V., & Soulier, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.768 · Cited by 4
Żyliński, M., Occhipinti, E., & Mandic, D. P. (2023) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc40787.2023.10340929 · Cited by 4
“Zhang et al. [19] demonstrated that complex deep neural networks easily fit random labelled datasets, but when evaluated on new observations, they produce scores which are no better than random chance.”
BackgroundLi, Q., Duan, C., & Chen, S. (2023) · Globecom · doi:10.1109/globecom54140.2023.10437272 · Cited by 6
“It has been demonstrated that the existence of noisy labels can cause severe performance degradation of the deep neural network (DNN) [7].”
BackgroundROG_PL: Robust Open-Set Graph Learning via Region-Based Prototype Learning
Zhang, Q., Li, X., Lu, J., Qiu, L., Pan, S., Chen, X., & Chen, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i8.28788 · Cited by 10
A Normalization Strategy for Weakly Supervised 3D Hand Pose Estimation
Guo, Z., Li, J., & Tan, J. (2024) · Applied Sciences · doi:10.3390/app14093578 · Cited by 4
HRCL: Hierarchical Relation Contrastive Learning for Low-Resource Relation Extraction
Guo, Q., Guo, Y., & Zhao, J. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3386611 · Cited by 6
“However, inaccurate classification leads to error accumulation and further affects subsequent training, which is known as the gradual drift problem [10].”
Shen, J., Wang, Y., & Zhang, D. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.047903 · Cited by 4
A spatio-temporal predictive learning model for efficient sea surface temperature forecasting
Wang, S., & Ren, H. (2024) · Climate Dynamics · doi:10.1007/s00382-024-07348-2 · Cited by 4
Why and how to construct an epistemic justification of machine learning?
Špelda, P., & Střítecký, V. (2024) · Synthese · doi:10.1007/s11229-024-04702-z · Cited by 5
“However, the values of VC dimension for overparameterized (state-of-the-art) deep artificial neural networks are large while R ( h ) remains stable or decreases (cf. Valle-Pérez & Louis, 2020, pp. 13–15; Zhang et al., 2017, 2021).”
Evaluating the explanatory power of the Conscious Turing Machine
Kirkeby-Hinrup, A., Stenseke, J., & Overgaard, M. S. (2024) · Consciousness and Cognition · doi:10.1016/j.concog.2024.103736 · Cited by 4
Diesel Engine Monitoring and Diagnostics Based on Artificial Neural Networks
Zhou, Z., Bao, T., Ding, J., Chen, Y., Wang, F., & Zhang, B. (2024) · International Conference on Communications, Circuits and Systems · doi:10.1109/icccas62034.2024.10652796 · Cited by 4
“Examples of such data-driven techniques include ANNs [5], Long Short-Term Memory (LSTM) [6], and Genetic Algorithms (GAs) [7].”
MethodOnline Continuous Generalized Category Discovery
Park, K.-H., Lee, H., Song, K., & Park, G.-M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73004-7_4 · Cited by 12
Qora: Neural-Enhanced Interference-Aware Resource Provisioning for Serverless Computing
Ma, R., Zhan, Y., Wu, C., Hong, Z., Ali, Y., & Xia, Y. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3526197 · Cited by 4
“Neural networks can capture complex patterns and trends [49], [50], particularly those variations created by interference and new serverless functions.”
EARLY WARNING PREDICTION WITH AUTOMATIC LABELLING IN EPILEPSY PATIENTS
ZHANG, P., Gao, T., GUO, J., Duan, J., & Nikolenko, S. (2025) · The ANZIAM Journal · doi:10.1017/s1446181124000178 · Cited by 4
“Similarly, training exclusively on noisy datasets—or a mix of both clean and noisy datasets—may also yield unsatisfactory outcomes, as large-scale models can inadvertently fit and memorize the noise [35].”
HistAttentionNAS: A CNN built via NAS for Penile Cancer Diagnosis using Histopathological Images
José Ribamar Durand, R., Bráz, G., da Silva, I. F. S., & da Costa, R. M. G. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.02.177 · Cited by 5
Efficient Diagnosis of Retinal Diseases Using Convolutional Neural Networks
Kaya, M. (2025) · Gazi University Journal of Science Part A Engineering and Innovation · doi:10.54287/gujsa.1592915 · Cited by 4
“When overfitting occurs, CNN models tend to retain the training data in memory rather than generalizing from it, so their training accuracy is very high, but their performance drops significantly when encountering unfamiliar test data (Zhang et al., 2021).”
Ensemble Linear Interpolators: The Role of Ensembling
Wu, M., & Sun, Q. (2025) · SIAM Journal on Mathematics of Data Science · doi:10.1137/24m1642548 · Cited by 8
Advanced deep architecture pruning using single-filter performance
Tzach, Y., Meir, Y., Gross, R. D., Tevet, O., Koresh, E., & Kanter, I. (2025) · Physical review. E · doi:10.1103/49t8-mh9k · Cited by 4
The Computational Evolution of Cognitive Architectures
Kotseruba, I., & Tsotsos, J. K. (2025) · doi:10.1093/oso/9780192844835.001.0001 · Cited by 7
Visualizing Deep Neural Networks with Interaction of Super-pixels
Tian, S., & Cai, Y. (2017) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3132847.3133108 · Cited by 4
Saving Energy of RRAM-Based Neural Accelerator Through State-Aware Computing
He, Y., Wang, Y., Li, H., & Li, X. (2021) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2021.3103147 · Cited by 6
Reminding the incremental language model via data-free self-distillation
Wang, H., Fu, R., Li, C., Zhang, X., Zhou, J., Bai, X., Yan, Y., & Zhao, Q. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03678-y · Cited by 5
“However, according to [38], the standard cross-entropy loss causes the model to label noise.”
BackgroundChannel Redundancy and Overlap in Convolutional Neural Networks with Channel-Wise NNK Graphs
Bonet, D., Ortega, A., Ruiz‐Hidalgo, J., & Shekkizhar, S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746186 · Cited by 6
Genetic Fuzzy Hand Gesture Classifier
Palmer, H., & Cohen, K. D. O. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-82099-2_30 · Cited by 4
Potenzial von Radiomics und künstlicher Intelligenz in der Myelombildgebung
Wennmann, M., & Murray, J. M. (2021) · Der Radiologe · doi:10.1007/s00117-021-00940-1 · Cited by 4
“Entscheidend sind u. a. die Ausgangsbedingungen, insbesondere die Qualität, Quantität und Repräsentativität der verwendeten Trainingsdaten und die Rechenleistung der IT-Infrastruktur sowie die Güte der Durchführung eines Projekts hinsichtlich Auswahl, Training, Validierung und Testung des maschinellen Lernmodells.”
BackgroundInfluentialLiu, Z., Li, R., Miao, D., Ren, L., & Zhao, Y. (2022) · Security and Communication Networks · doi:10.1155/2022/1615476 · Cited by 5
“Because the model will learn more unconventional data points and member information [30] at the later epochs, which improves the accuracy of member inference attacks.”
MethodSelf-Assessment and Robust Anomaly Detection with Bayesian Deep Learning
Carannante, G., Dera, D., Aminul, O., Bouaynaya, N. C., & Rasool, G. (2022) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion49751.2022.9841358 · Cited by 6
Real-World Computer Vision for Real-World Applications: Challenges and Directions
Tabkhi, H. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-16072-1_53 · Cited by 4
Gerg, I. D., & Monga, V. (2022) · MTS/IEEE OCEANS Conference (OCEANS) · doi:10.1109/oceans47191.2022.9977362 · Cited by 4
“Venn diagram comparing human and CNN perception properties as articulated by [2], [5], [10], [12]–[14].”
BackgroundMeta-Probability Weighting for Improving Reliability of DNNs to Label Noise
Wang, Z., Jin, S., Li, L., Dong, Y., & Hu, Q. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3237033 · Cited by 5
Peaks Fusion assisted Early-stopping Strategy for Overhead Imagery Segmentation with Noisy Labels
Liu, C., Albrecht, C. M., Wang, Y., & Zhu, X. X. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020164 · Cited by 4
“In spite of the convenience and speed, these rule-based labels inevitably bear some noises, threatening to bias the model training [9].”
BackgroundTransformer-Based Neural Augmentation of Robot Simulation Representations
Serifi, A., Knoop, E., Schumacher, C., Kumar, N., Groß, M. H., & Bächer, M. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3271812 · Cited by 8
“Over the training time, however, we observe an overfitting effect that leads to increasing position and velocity errors, which is a known issue with neural networks [29].”
BackgroundLearning Fair Representations through Uniformly Distributed Sensitive Attributes
Kenfack, P. J., Rivera, A. R., Khan, A., & Mazzara, M. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00014 · Cited by 10
“[20] show that deep neural networks can easily fit data with partially or completely noisy labels.”
BackgroundConfidence-aware calibration and scoring functions for curriculum learning
Ao, S., Rueger, S., & Siddharthan, A. (2023) · International Conference on Machine Vision · doi:10.1117/12.2679353 · Cited by 4
“Recent deep learning models are easily over-fitting with noisy labels [22], and training with label smoothing can significantly improve the model performance under various levels of noise [23].”
BackgroundStochastic Gradient Descent and Anomaly of Variance-Flatness Relation in Artificial Neural Networks
Xia, X., Chen, Y.-C., Shi, C., & Ao, P. (2023) · Chinese Physics Letters · doi:10.1088/0256-307x/40/8/080202 · Cited by 4
halmuratov, G., & Veselý, A. (2023) · WSEAS TRANSACTIONS ON COMPUTER RESEARCH · doi:10.37394/232018.2023.11.12 · Cited by 4
Concept Evolution in Deep Learning Training: A Unified Interpretation Framework and Discoveries
Park, H., Lee, S., Hoover, B., Wright, A. P., Shaikh, O., Duggal, R., Das, N., Li, K., Hoffman, J., & Chau, D. H. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3614819 · Cited by 6
“Crucially, understanding the progression of concepts detected by individual neurons, which we refer to as the neuron’s concept evolution, and its association with model deficiencies like poor generalizability [18, 23, 58] or convergence failures [2, 42] remains lacking.”
BackgroundCurriculum Guided Domain Adaptation in the Dark
Jahan, C. S., & Savakis, A. E. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3330829 · Cited by 4
Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations
Hao, Y., Wang, R., Cao, Z., Wang, Z., Cui, Y., & Sadigh, D. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341728 · Cited by 6
“Deep neural networks are known to have the capacity to memorize noise or pick up spurious correlations [34, 35].”
BackgroundA robust optimization method for label noisy datasets based on adaptive threshold: Adaptive-k
Dedeoglu, E., Kesgin, H. T., & Amasyalı, M. F. (2023) · Frontiers of Computer Science · doi:10.1007/s11704-023-2430-4 · Cited by 4
“In addition, deep learning models can easily fit the datasets with random labels [1].”
BackgroundAdvances in image denoising techniques: a comprehensive review
he, Y. (2024) · doi:10.1117/12.3017758 · Cited by 4
A Review of Research on Generalization Error Analysis of Deep Learning Models
Zhang, Z., & Zhang, X. (2023) · International Conference on Image Processing, Computer Vision and Machine Learning (ICICML) · doi:10.1109/icicml60161.2023.10424837 · Cited by 4
“Agreement was expressed with Zhang et al [12] that DNNs can adapt perfectly to noise and therefore their generalization ability cannot be explained by traditional statistical learning theory.”
BackgroundMethodInfluentialKiehas, F., Reiter, M., Torres, J. P., Jerabek, M., & Major, Z. (2024) · Polymer · doi:10.1016/j.polymer.2024.126797 · Cited by 4
Dong, B., Xu, Y., Chi, S., Shi, Z., & Du, Z. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00283 · Cited by 6
“In fact, Zhang et al. [69] conducted experiments demonstrating that deep neural networks can easily fit training datasets with various levels of noise, resulting in poor generalization of the models.”
BackgroundMisra, V., Menon, S. S., Saha, S., Mathur, A., Yu, H., & Raychoudhury, V. (2024) · SN Computer Science · doi:10.1007/s42979-024-03181-w · Cited by 4
Self-Paced Knowledge Distillation for Real-Time Image Guided Depth Completion
Wang, S., Hu, M. H., Li, B., & Gong, X. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3154243 · Cited by 5
An amortized approach to non-linear mixed-effects modeling based on neural posterior estimationPreprint
Arruda, J., Schälte, Y., Peiter, C., Teplytska, O., Jaehde, U., & Hasenauer, J. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.08.22.554273 · Cited by 5
Jafarigol, E., & Trafalis, T. B. (2023) · Research Square · doi:10.21203/rs.3.rs-3492014/v1 · Cited by 6
Near-Optimal Solutions of Constrained Learning Problems
Elenter, J., Chamon, L. F. O., & Ribeiro, A. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2403.11844 · Cited by 6
“That is the case, e.g., for deep neural networks (Zhang et al., 2021; Brutzkus & Globerson, 2017; Soltanolkotabi et al., 2018; Ge et al., 2017).”
BackgroundOptimized deep learning-based multimodal method for irregular medical timestamped data
Rabhi, S. (2022) · theses.fr (ABES) · doi:10.70675/154ab19dz373dz4f8az9c16zaba9b75e66ed · Cited by 5
Fast Convergence in Learning Two-Layer Neural Networks with Separable Data
Taheri, H., & Thrampoulidis, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26186 · Cited by 3
Small Data, Big Challenges: Pitfalls and Strategies for Machine Learning in Fatigue Detection
Jeworutzki, A., Schwarzer, J., von Luck, K., Stelldinger, P., Draheim, S., & Wang, Q. (2023) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3594806.3594825 · Cited by 3
“As described by Zhang et al. [69, 70], the sample size, input dimension, and output classes need to be considered to avoid over-fitting.”
MethodReconstructing brain functional networks through identifiability and deep learning
Zanin, M., Aktürk, T., Yıldırım, E., Yerlikaya, D., Yener, G., & Güntekin, B. (2023) · Network Neuroscience · doi:10.1162/netn_a_00353 · Cited by 3
“Specifically, long enough time series have to be used, or alternatively a large number of trials per patients; in any case, enough data have to be available to avoid model overfitting (C. Zhang, Bengio, Hardt, Recht, & Vinyals, 2021).”
BackgroundEnhancing Classification Accuracy on Limited Data via Unconditional GAN
Hong, C., Cha, B., Kim, B., & Oh, T.-H. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00113 · Cited by 5
Learning from the Past Training Trajectories: Regularization by Validation
Zhang, E., Wahib, M., Zhong, R., & Munetomo, M. (2024) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2024.p0067 · Cited by 3
Zhang, M., Wang, J., Wang, R., Yang, F., & Meng, J. (2024) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2024.3362388 · Cited by 3
“Data quality and parameter adjustments influence the improvement in inversion accuracy among various AI meth-ods [18], [19].”
BackgroundHoshino, T., Kanoga, S., Tsubaki, M., & Aoyama, A. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106261 · Cited by 3
Rethinking confidence scores for source-free unsupervised domain adaptation
Tian, Q., & Sun, C. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09867-9 · Cited by 3
“They trained a weak network from scratch and continuously integrate the parameters of the weak network into the strong network originating from the previous discovery that neural networks tend to prioritize memorizing clean labels before memorizing noisy labels [48].”
Towards Optimal Inverse Temperature in the Gibbs Algorithm
Bu, Y. (2024) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit57864.2024.10619441 · Cited by 3
Learning Representation for Multitask Learning Through Self-supervised Auxiliary Learning
Shin, S., Do, H., & Son, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72989-8_14 · Cited by 9
D4A: An efficient and effective defense across agnostic adversarial attacks
Li, X., Gan, Z., Bai, Y., Su, L., Li, D., & Wang, J. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106938 · Cited by 3
CODE-CL: Conceptor-Based Gradient Projection for Deep Continual Learning
Apolinario, M. P., Choudhary, S., & Roy, K. (2024) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.00080 · Cited by 8
Wei, Z., Zhang, X., & Tan, Z. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107205 · Cited by 4
Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networks
Chen, P., Jiang, R., & Wang, P. (2026) · Mathematics of Operations Research · doi:10.1287/moor.2025.1205 · Cited by 3
Auxiliary action unit model for facial expression adversarial training
Sun, Y., Zhang, F., & Yang, M. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111493 · Cited by 3
Neri, M., & Battisti, F. (2025) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2025.3553305 · Cited by 3
“As stated in [45], a reduced number of trainable parameters can reduce the likelihood of over-fitting, which is particularly beneficial for small datasets.”
Wang, S., Yang, F., Wang, S., & Sun, R. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1590824 · Cited by 4
“Zhang et al. (8) argued that deep models, such as convolutional neural networks, can accurately memorize arbitrary labels of training data.”
SV-SAE: Layer-Wise Pruning for Autoencoder Based on Link Contributions
Rheey, J., & Park, H. (2025) · IEEE Access · doi:10.1109/access.2025.3565296 · Cited by 3
“As the number of real-world problems solved using deep learning-based approaches increases, models have become larger due to the use of dense neural networks to improve performance [1].”
Zhang, X., Zhang, Y., Luo, X., Liu, J., Wu, S., & Geng, J. (2025) · Geophysics · doi:10.1190/geo2024-0601.1 · Cited by 3
Artificial Neural Network Text Generation as a New Type of Masking the Authorship
Ogorelkov, I., & Mahaeva, E. (2025) · Virtual Communication and Social Networks · doi:10.21603/2782-4799-2025-4-2-163-171 · Cited by 3
Cheng, L., Qiu, X., Tan, X., Wang, H., & Xiong, Y.-J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111619 · Cited by 3
CLTR: Continual learning time-varying regularization for robust classification of noisy label images
Li, Y., Guo, Z., & Wang, L. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112137 · Cited by 3
DPL++: Advancing the Network Performance via Image and Label Perturbations
Song, Z., Gong, X., Hu, G., Dou, S., Zhao, Q., & Zhao, C. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3594149 · Cited by 3
“Our analysis revealed three significant findings: a) Label perturbations enhance the alignment between labels and Accordingtopreviousresearch[75],[76],[77],thesecharacter-isticsreflectthattheDPL++paradigmendowsthemodelwith theabilitytoquicklycapturekeyfeaturesfromwell-aligned, forward-lookingtrainingsignals(Fig.10)andsti…”
Longitudinal Ensemble Integration for Sequential Classification with Multimodal Data
Susman, A., Krishnamurthy, R., Olaimat, M. A., Bozdag, S., Varghese, B. A., Sheikh‐Bahaei, N., & Pandey, G. (2025) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh67620.2025.00014 · Cited by 4
“Since deep learning methods like LSTMs are well-known to be hard to interpret, as demonstrated by Zhang et al. (2021), we adopted an alternate approach based on the interpretation of static EI models (Li et al., 2022).”
Nie, C. X., Jeong, H., Park, S., Jiang, H., Hyun, K.-A., Kim, J. S., Yang, J. Y., Park, S. J., Shin, J., & Jung, H. (2025) · Sensors and Actuators B Chemical · doi:10.1016/j.snb.2025.138557 · Cited by 3
Multiple descents in deep learning as a sequence of order-chaos transitions in LSTM networks
Wei, W., Xu, F., Chong, N. J. L., Lai, C. H., & Feng, L. (2026) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2026.131754 · Cited by 3
Sheinkman, A., & Wade, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06109-6_8 · Cited by 3
Wu, Y., Xu, X., Liu, Y., Shi, S., Ma, A., & Zhong, Y. (2025) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2025.104913 · Cited by 5
CoviSwin: A Deep Vision Transformer for Automatic Segmentation of COVID-19 CT Scans
Alsenan, A., Youssef, B. B., & Alhichri, H. S. (2025) · Bioengineering · doi:10.3390/bioengineering12111227 · Cited by 3
“Zhang et al. highlighted in their work that deep neural networks possess a strong capacity to fit data and generalize effectively, even without strict reliance on standard validation or regularization mechanisms [35].”
Compound flooding: A review on assessment framework
Akter, A., & Ataullah, S. M. (2025) · Journal of Coastal Conservation · doi:10.1007/s11852-025-01153-7 · Cited by 3
“MLPs typically have more parameters than GLMs and are usually applied to larger datasets, posing challenges for memory storage and computing the loss and its deriva - tives (Belkin et al. 2019 ; Hochreiter and Schmidhuber 1997; Zhang et al. 2021 ).”
Mudin, I., Ali, M., & Hidayat, A. (2025) · Intiqad Jurnal Agama dan Pendidikan Islam · doi:10.30596/24275 · Cited by 3
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.