Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 1,701–1,800 of 2,667
A Conceptual Foundation for Generalisation in Natural and Artificial Learning AgentsPreprint
Gabriel, G., Morehead, J. R., & Mushtaq, F. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ydtqu · Cited by 2
Generalization Bounds for Dependent Data using Online-to-Batch ConversionPreprint
Chatterjee, S., Mukherjee, M., & Sethi, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.13666 · Cited by 2
Bianco, F., Rigato, S., Filippetti, M. L., & Ognibene, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.18022 · Cited by 2
Taming the Curse of Dimensionality: Quantitative Economics with Deep LearningPreprint
Fernández‐Villaverde, J., Nuño, G., & Perla, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5043048 · Cited by 4
Kani, S. N., Soyak, B. C., Gokce, M., Duyar, Z., Alicikus, H., Yapıcıer, Ö., & Öner, M. Ü. (2025) · medRxiv · doi:10.1101/2025.01.16.25320663 · Cited by 2
“Their complexity also increases the risk of overfitting, especially with smaller datasets (Zhang et al. [2021]).”
Can Bayesian Neural Networks Make Confident Predictions?Preprint
Fisher, K., & Marzouk, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.11773 · Cited by 3
Simon, T., Hidalgo, A. D., Velina, Y., & Majumdar, S. (2025) · ChemRxiv · doi:10.26434/chemrxiv-2025-1pnxw · Cited by 2
Morality in AI. A plea to embed morality in LLM architectures and frameworksPreprint
Bombaerts, G. J. T., Delisse, B., & Kaymak, U. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.20689 · Cited by 2
“…and moral attention mechanisms both face uncertainty and at the same time provide the “structure” to reorganize themselves by generating qualitative transformations: emergent neural representations (Zhang et al., 2021) and enhanced moral perception through loving attention (Murdoch, 2013b).”
Is data-efficient learning feasible with quantum models?
Sakhnenko, A., Mendl, C. B., & Lorenz, J. M. (2026) · Quantum Machine Intelligence · doi:10.24406/publica-6648 · Cited by 2
Dufumier, B. (2022) · theses.fr (ABES) · doi:10.70675/d7d71a04z0514z43e3z840cz71ee3bb8a35b · Cited by 4
An Empirical Analysis of Boosting Deep Networks
Rambhatla, S. S., Jones, M., & Chellappa, R. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892204 · Cited by 1
Gaps in Generalization Theory of Neural Networks
Pandey, E., & Kumar, S. (2023) · Advances in transdisciplinary engineering · doi:10.3233/atde221272 · Cited by 1
Calibration-based Steganalysis for Neural Network Steganography
Zhao, N., Chen, K., Qin, C., Yin, Y., Zhang, W., & Yu, N. (2023) · ACM Workshop on Information Hiding and Multimedia Security (IH&MMSec) · doi:10.1145/3577163.3595100 · Cited by 1
“Meanwhile, the literature[18] found that there is also much redundancy of NNs, i.e., only a small proportion of parameters play significant roles in the corresponding task, while the changes in the remaining parameters will not affect the performance.”
BackgroundToken-level Fitting Issues of Seq2seq Models
Bao, G., Teng, Z., & Zhang, Y. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.23 · Cited by 1
DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer
Kumar, S., Soujanya, A., Dandapat, S., Sitaram, S., & Choudhury, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.29 · Cited by 1
A Classification Performance Evaluation Measure Considering Data Separability
Xue, L., Zhang, X., Jiang, W., Huo, K., & Shen, Q. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44207-0_1 · Cited by 1
Haugstvedt, E. J., Calero, A. M., Lundby, E. T. B., Rasheed, A., & Gravdahl, J. T. (2023) · IEEE Access · doi:10.1109/access.2023.3336292 · Cited by 1
AI on Edge: A Mass Accessible Tool for Decision Support Systems
Dayal, S. (2024) · IntechOpen eBooks · doi:10.5772/intechopen.1003945 · Cited by 1
Can Overfitted Deep Neural Networks in Adversarial Training Generalize? An Approximation Viewpoint
Shi, Z., Liu, F., Cao, Y., & Suykens, J. A. K. (2026) · SIAM Journal on Mathematics of Data Science · doi:10.1137/24m1634023 · Cited by 1
Data reweighting net for web fine-grained image classification
Liu, Y., Wu, Z., Lo, S., Chen, Z., Ke, G., & Yue, C. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18598-x · Cited by 1
“The memory effect [28, 29] has demonstrated that incorrect labels inhibit the improvement of model performance.”
AUTOSGM: A Unified Lowpass Regularization Framework for Accelerated Learning
Somefun, O. A., Lee, S., & Mathews, V. J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448203 · Cited by 1
“Although the question of optimal learning algorithms was treated in [11], relatively recent works have continued to posit that stochastic gradient learning theory and practice needs more explanation and unification, especially for large-scale non-convex optimization [12, 13].”
BackgroundGeneralization Error in the Deep Ritz Method with Smooth Activation Functions
Siipola, J. (2024) · Communications in Computational Physics · doi:10.4208/cicp.oa-2023-0253 · Cited by 1
Scaling Laws of Data-Driven Machine Learning Models: A Survey and Taxonomy
Wang, C., Wang, Y., Li, Z., Jia, Q., & Liu, W. (2026) · ACM Computing Surveys · doi:10.36227/techrxiv.171340392.27668396/v1 · Cited by 2
Larger language models better align with neural representations of natural language
Hong, Z., Wang, H., Zada, Z., Gazula, H., Turner, D. D., Aubrey, B., Niekerken, L., Doyle, W., Devore, S., Dugan, P. C., et al. (2024) · eLife · doi:10.7554/elife.101204 · Cited by 4
“Recent work has argued that the ‘size’ of these models—the number of learnable parameters—is critical, as some linguistic competencies only emerge in larger models with more parameters ( Bommasani et al., 2021 ; Kaplan et al., 2020 ; Manning et al., 2020 ; Sutton, 2019 ; Zhang et al., 2021 ).”
Zender, J., Murawa, A., Pinzer, B., & Layh, M. (2024) · Optical Instrument Science, Technology, and Applications · doi:10.1117/12.3016889 · Cited by 1
“15,16 A primary factor for this lack of variance could be that the global pin geometry is not adequately varied.”
Rejoinder to Discussion of "Data Science at the Singularity"
Donoho, D. L. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.e81732b3 · Cited by 1
CoMix: Confronting with Noisy Label Learning with Co-training Strategies on Textual Mislabeling
Zhao, S. G., Zhao, Z., Xu, Y., & Sun, X. N. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3678175 · Cited by 1
“Deep neural networks easily overfit to noisy label samples [39].”
G-T correcting: an improved training of image segmentation under noisy labels
Gao, Y., Fu, J., Guo, Y. P., & Wang, Y. (2024) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-024-03170-4 · Cited by 1
“Existing study [2] indicates that DNNs can easily over-fit to noisy labels due to their strong learning capabilities, resulting in performance decay.”
Rostand, J., Hsu, C.-C. J., & Lu, C. (2024) · Journal of the Chinese Institute of Engineers · doi:10.1080/02533839.2024.2383592 · Cited by 1
“However, when trained using traditional loss functions, models for which the number of parameters exceeds the amount of training data by a wide margin tend to be prone to overfitting (Zhang et al. 2021).”
Adaptive Control for Autonomous Underwater Vehicles without Kinetic Model using Deep Neural Network
Jiang, S., Zhang, T., Fang, J., Liu, X., & Shen, D. (2024) · IEEE Data Driven Control and Learning Systems Conference (DDCLS) · doi:10.1109/ddcls61622.2024.10606810 · Cited by 1
“Hence, some adaptive control methods are coupled with single-layer neural networks to provide accurate estimates of gradient information without model information [14, 15].”
MethodAdaptively Denoising Graph Neural Networks for Knowledge Distillation
Guo, Y., Yang, C. H., Shi, C., Tu, K., Wu, Z., Zhang, Z., & Zhou, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70371-3_15 · Cited by 1
Rasyid, M. R., Sulistiyo, M. D., Sthevanie, F., Wulandari, G. S., & Haryadi, D. (2024) · International Conference on Data Science and Its Applications (ICoDSA) · doi:10.1109/icodsa62899.2024.10651791 · Cited by 1
“Zhang et al. (2021) proposed EfficientPose, a lightweight pose estimation network that achieves competitive performance while being computationally efficient.”
BackgroundEarly-Late Dropout for DivideMix: Learning with Noisy Labels in Deep Neural Networks
Wu, H., Jia, B., & Sheng, G. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650652 · Cited by 1
“Based on previous studies[23], [21], [17], we evaluate our method on two types of label noise and different noise rates, η ∈ { 20% , 50% , 80% } for symmetric noise and η ∈ { 40% } for asymmetric labeling noise.”
BackgroundMethodPassive Identification of WiFi Devices At-Scale: A Data-Driven Approach
Bogahawatta, N., Karunanayaka, Y. S., Seneviratne, S., Thilakarathna, K., Masood, R., Kanhere, S. S., & Seneviratne, A. P. (2024) · Conference on Local Computer Networks · doi:10.1109/lcn60385.2024.10639764 · Cited by 2
An Unbiased Risk Estimator for Partial Label Learning with Augmented Classes
Hu, J., Shu, S., Li, B., Xiang, T., & He, Z. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3700137 · Cited by 1
“Several representative tasks have been investigated, such as semi-supervised learning [8, 64], noisy-label learning [19, 56], partial-label learning [49], multi-label learning [18, 37, 40, 55, 58], etc.”
BackgroundAcoustic Event-Based Prestressing Concrete Wire Breakage Detection
Farhadi, S., Corrado, M., & Ventura, G. (2024) · Procedia Structural Integrity · doi:10.1016/j.prostr.2024.09.305 · Cited by 1
Evaluating Neural Network Parameter Obliteration in Catastrophic Forgetting
Sehra, U., Dutta, R. R., & Pandit, M. K. (2024) · IEEE Region 10 Symposium (TENSYMP) · doi:10.1109/tensymp61132.2024.10752250 · Cited by 1
Gaur, S., Kandwal, A., & Pandey, B. (2024) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v37.i2.pp1284-1293 · Cited by 1
“Traditional ML models struggle with complex data, while CNNs in deep learning (DL) excel due to advanced pre-processing, convolutional functions, and adaptability with deeper networks [7], [8].”
Reducing overfitting in vehicle recognition by decorrelated sparse representation regularisation
Wei, W., Fu, X., Ma, S., Zhu, Y., & Lu, N. (2024) · IET Computer Vision · doi:10.1049/cvi2.12320 · Cited by 2
What is in a food store name? Leveraging large language models to enhance food environment data
Etheredge, A. J., Hosmer, S., Crossa, A., Suss, R., & Torrey, M. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1476950 · Cited by 1
GroupLearning: Label Noise Mitigation Through Multi-Expert Collaboration for Medical Image Analysis
Cao, Q., Liu, D., Fan, J., & Cai, W. (2024) · International Conference on Image and Vision Computing New Zealand (IVCNZ) · doi:10.1109/ivcnz64857.2024.10794199 · Cited by 1
Robustness of topological persistence in knowledge distillation for wearable sensor data
Jeon, E. S., Choi, H., Shukla, A., Wang, Y., Buman, M. P., Lee, H., & Turaga, P. K. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-024-00512-y · Cited by 2
“To investigate tolerance of models under noises, we set different scenarios of evaluation with corruptions for testing as well as training data [18–21], which are major factors that affect the performance of time series data analysis.”
Co-active: an efficient selective relabeling model for resource constrained edge AI
Hou, C., Jiang, K., Li, T., Zhou, M., & Jiang, J. (2025) · Wireless Networks · doi:10.1007/s11276-025-03903-9 · Cited by 3
“Consequently, the models are less reliable and dependable, making them unsuitable for deployment in applications with high-stakes scenarios [5].”
Cascade Ownership Verification Framework Based on Invisible Watermark for Model Copyright Protection
Wang, R., Zhu, Y., & Xia, D. (2025) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.8394 · Cited by 2
“Even when trained with incorrectly labeled data, deep models can fit these anomalous relationships quite well [12].”
Segment Any Bone in CT with Partial Supervision
Liang, T., Li, X., Peng, Y., & Xu, M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890755 · Cited by 1
Can Quality Survive Scale? Toward an Equal-Quality Instance-Dependent Label Noise Model
Song, H., Xiang, Q., & Shu, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888857 · Cited by 1
Exponential Convergence of Stochastic Mirror Descent in Over-parameterized Linear Models
Varma, K. N., & Hassibi, B. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888059 · Cited by 1
Anti-infective Molecules (Antimicrobial Property) from Mangrove Microbiome
Ramesh, S., Annamalai, K. K., Kaari, M., Saravanan, T. G., Singaraj, R., Baskaran, A., Dharmalingam, S., Panneerselvam, K. P. M., Saravanan, S. M. K., Gopikrishnan, V., et al. (2025) · Springer eBooks · doi:10.1007/978-981-96-2602-1_18 · Cited by 1
A practical generalization metric for deep networks benchmarking
Huang, M., Yu, H., & Zhang, J. (2025) · Scientific Reports · doi:10.1038/s41598-025-93005-5 · Cited by 5
“There is a growing consensus that traditional approaches in machine learning theory, grounded in worst-case analyses, are inadequate to fully elucidate the generalization of deep learning models (Zhang et al. (2021b)).”
Wang, Y., Pang, H., Qin, Y. H., Wei, S., & Zhao, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107464 · Cited by 1
Generalization memorization machine for regression
Sijie, Z., Bai, L., & Wang, Z. (2025) · International Conference on Computer Application and Information Security (ICCAIS) · doi:10.1117/12.3061042 · Cited by 1
“Inspired by recent research on the generalization efficiency and memory properties of deep neural networks 13-15 , Vapnik and Izmailov 2 introduced a novel generalization-memorization kernel into SVM, enhancing its memory capability (termed SVM m ).”
Temporal-Feedback Self-Training for Semi-Supervised Object Detection in Remote Sensing Images
Zhu, X., Zhang, X., Zhang, T., Tang, X., Chen, P., Zhou, H., & Jiao, L. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3559224 · Cited by 1
“Prior studies [18], [19] have demonstrated that deep learning first learns simple patterns before fitting noise data.”
Yu, F., Chen, J., Kong, Y., Wang, J., & Chen, X. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-3317-3_36 · Cited by 1
Raju, G. V. S., & Sharma, N. K. (2025) · South Asian Journal of Management · doi:10.62206/sajm.32.1.2025.147-168 · Cited by 1
Extending the scope of case-based reasoning through multiple case-based generalizations
Ghalem, Z., ZEGOUR, D. E., Bouabana‐Tebibel, T., & Rubin, S. H. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02472-9 · Cited by 1
“This constraint implies they may encounter difficulties when attempting to utilize acquired information in scenarios that necessitate comprehension of implicit, universally acknowledged human knowledge [51].”
Artificial neural network modeling for predicting PM10, PM2.5, NOX, and SO2 in coal mining areas
Mishra, A., Prasad, N., Bhattacharya, T., & Lal, B. (2025) · Earth Science Informatics · doi:10.1007/s12145-025-01922-w · Cited by 1
“A notable discrepancy between training performance and validation performance in SO 2 suggests that the model may have memorized the training data instead of acquiring the ability to generalize from it (Arpit et al. 2017; Zhang et al. 2021).”
Double Deep Q-Learning Network for Package Pickup and Delivery Route Prediction
Gubena, M. M., Zhang, Y., Ma, Z., Ma, X., & Yang, Y. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3577902 · Cited by 1
Gondoza, G. N. F., Ncwadi, R., Fernandes, J. M., & Nyika, F. D. (2025) · BRICS Journal of Economics · doi:10.3897/brics-econ.6.e141556 · Cited by 1
“This confirms the results of Zhang et al. (2021), showing that an artificial neural network model with more input neurons is better than that with fewer input neurons.”
DNN-based Parameterization for B-Spline Curve Approximation
Tang, W., & Yang, Z. (2025) · Computer-Aided Design · doi:10.1016/j.cad.2025.103897 · Cited by 1
Snyder, S. J., Lee, H., Chu, E., Heo, Y. J., Miller, R. H., & Shim, J. K. (2025) · Journal of Applied Biomechanics · doi:10.1123/jab.2024-0181 · Cited by 1
Inventory Optimization in Retail Supply Chains Using Deep Reinforcement Learning
Park, M.-J., Turner, O., & Becker, T. (2025) · Advances in Management and Intelligent Technologies · doi:10.62177/amit.v1i3.470 · Cited by 1
Pallones, M. M., & Recto, K. H. A. (2025) · Proceedings on Engineering Sciences · doi:10.24874/pes07.02a.017 · Cited by 1
Improving classifier decision boundaries and interpretability using nearest neighbors
Schneider, J., & Casanova, A. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00369-8 · Cited by 1
“While large neural networks are known to effectively “memorize” noisy data [55], noisy labels still affect a model’s ability to generalize effectively.”
Liao, Z., Hu, S., Zhang, Y., & Xia, Y. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103702 · Cited by 2
Unsupervised Segmentation and Classification of Waveform-Distortion Data Using Non-Active Current
Mariscotti, A., Salles, R. S., & Rönnberg, S. K. (2025) · Energies · doi:10.3390/en18133536 · Cited by 1
Can Informativity Effects Be Predictability Effects in Disguise?
Kapatsinski, V. M. (2025) · Entropy · doi:10.3390/e27070739 · Cited by 2
“Modern deep neural networks also appear to be rich memory systems that implement a kind of hierarchical inference when generating text, in that best performance is achieved when the network is more than big enough to memorize the entire training corpus [51,52].”
Zhang, W., Lin, M., Chan, K. L., Chen, H., & Chiu, B. C. Y. (2025) · Medical Physics · doi:10.1002/mp.17952 · Cited by 1
Adu, J. T., & Kumarasamy, M. (2025) · Proceedings of the International Conference of Recent Trends in Environmental Science and Engineering · doi:10.11159/rtese25.147 · Cited by 1
NoTAC: A Noise-Tolerance Automatic Cleaning Framework for Bone Marrow Karyotyping Data
Huang, R., Chen, S., Li, Y., Zhang, C., Zhang, Y., Yang, C., Li, N., Hu, J., Xu, A., Su, J., et al. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3594369 · Cited by 2
“Zhang et al. [56] observed that although DNNs can fit a training set with noisy (even random) labels, the generalization error significantly increases on a test set with the same level of noise.”
Self-citationImproving the Instance-Dependent Transition Matrix Estimation by Exploiting Self-Supervised Learning
Lin, Y., Yao, Y., Wang, Z., Shen, X., Yu, J., Han, B., & Liu, T. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3595613 · Cited by 1
Shi, J., Liu, C., Ishi, C. T., & Ishiguro, H. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3585950 · Cited by 1
Adaptive Vision-Language Prompt Learners for Learning with Noisy Labels
Hu, C., Nagarajan, B., Marques, R., & Radeva, P. I. (2025) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2025.104550 · Cited by 1
From Birth to Loss of Representations in Artificial Neural Networks
Stecher, P. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-94748-3_20 · Cited by 2
Inan, T. T., Shultana, S., & Shehu, A. (2025) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3712255.3726629 · Cited by 1
“Studies suggest SGD’s stochasticity helps models escape poor local minima and saddle points [23], [6], enabling more effective exploration of the loss landscape.”
Tuning-Free Online Robust Principal Component Analysis Through Implicit Regularization
Jayalal, L., Muthukrishnan, G., & Kalyani, S. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3599784 · Cited by 1
“In this paper, we demonstrate that the dependency on the aforementioned regularization parameters can be alleviated through the use of implicit regularization (IR) methods [8]–[14].”
BackgroundThe Climate Challenge: Sustaining Soil Health and Crop Resilience
Ghafoor, I., Naz, T., Nawaz, S., Iqbal, M. M., Ambreen, U., Hameed, S., Perveen, S., Rashid, A., Batool, M., Riaz, S., et al. (2025) · Frontier studies in soil science · doi:10.1007/978-3-031-91114-9_14 · Cited by 2
Green Finance: A Bibliometric Review
Richter, M. F., Machado, A. D. B., dos Santos, J. R., & Pesqueira, A. M. (2025) · World sustainability series · doi:10.1007/978-3-031-92390-6_19 · Cited by 1
GENGait: Exploring a Skeleton-Based Gait Recognition Framework With Strong Generalizability
Liu, W., Wei, S., Luo, H.-F., Wang, C., & Wang, R. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3593347 · Cited by 1
Storytelling in Kindergarten with Tangible Programming: A Literature Review
Palioura, M., Rapti, S., Sapounidis, T., & Georgaki, G. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-98762-5_26 · Cited by 1
Pactol, R. E., & Romana, C. L. C. S. (2025) · International Conference on Education and Training Technologies (ICETT) · doi:10.1109/icett66247.2025.11137068 · Cited by 1
“[14] There is however, a case to be made that both k-nearest neighbors and clustering might work in this context.”
Matt, H., & Stöger, D. (2025) · International Conference on Sampling Theory and Applications · doi:10.1109/sampta64769.2025.11133567 · Cited by 2
“However, it is not fully understood why these networks generalize so well [1].”
An accelerated approach for roadside LiDAR point cloud vehicle recognition
Xu, Y. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100022 · Cited by 1
Deep Learning-Based Inversion Method for Ocean Swell Waveheights From HF Radar
Shan, Z., Wang, W., Liu, T., Wu, X., He, S., Li, P., Song, J., & Wang, S. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3606628 · Cited by 1
Combating Noisy Labels in Knowledge Distillation for Efficient Edge Device Deployment
Sheng, M., Yan, S., Sun, Z., Chen, T., Liu, H., & Yao, Y. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3610163 · Cited by 1
Delving Into the Training Dynamics for Image Classification
Li, M., Zhou, X., & Wu, O. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3618395 · Cited by 1
“For instance, an over-parameterized model will overfit, while, surprisingly, over-parameterized DNNs can also generalize to new data e ff ectively [1].”
The effect of label noise on the information content of neural representations
Umar, A. H., Tezoh, F. K. N., Barbier, J., Acevedo, S., & Laio, A. (2026) · Frontiers in Physics · doi:10.3389/fphy.2026.1717253 · Cited by 2
Dynamic Optimization Noisy Cross-Modal Hashing
Yao, Z., Fu, H., Yang, Y., & Gu, G. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755684 · Cited by 1
Consistency Change Detection Framework for Unsupervised Remote Sensing Change Detection
Liu, Y., & Lu, Y. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209384 · Cited by 1
“Due to this patch-level training strategy, the generator network may memorize the specific input-output relationships of the remote sensing patch pairs, rather than learning more generalizable representations [18]–[20].”
Generalization analysis of quantum neural networks using dynamical Lie algebras
Ohno, H. (2025) · Quantum Information Processing · doi:10.1007/s11128-025-04990-5 · Cited by 1
“In [20] and [8], which address classical ML and QML, respectively, the limitations and issues of conventional uniform generalization bounds (e.g., those based on VC dimension and Rademacher complexity) were discussed.”
Dai, M., Luo, W., & Li, T. (2025) · International Conference on Computers, Information Processing and Advanced Education (CIPAE) · doi:10.1109/cipae66821.2025.00090 · Cited by 1
A Validation Strategy for Deep Learning Models: Evaluating and Enhancing Robustness
Nuhu, A.-R., Kebria, P. M., Hemmati, V., Lartey, B., Mahmoud, M. N., Homaifar, A., & Tunstel, E. W. (2026) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2025.3650722 · Cited by 1
Zhang, Y., Xu, X., Fu, Z., Wang, Y., Zhao, Y., & Zhang, F. (2025) · Atmosphere · doi:10.3390/atmos16111300 · Cited by 2
“Machine learning methods have been shown to demonstrate superior performance in comparison with statistical models in terms of regularization [34] and generalization [35].”
Liu, C., Perantoni, G., Bruzzone, L., & Zhu, X. X. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3660680 · Cited by 1
“Monitoring performance metrics on a clean validation set is ideal, as model performance typically improves, plateaus, then declines (indicating overfitting) [40], [41].”
How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning
Weltevrede, M., Zanger, M. A., Spaan, M. T. J., & Böhmer, W. (2025) · neural information processing systems · doi:10.52202/085713-4537 · Cited by 1
Smart and Digital Wellbeing Initiatives in Healthcare: The Way Forward
Varma, A., Saraiya, A. S., Tuli, K., Suresh, S., & Srivastava, S. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-7917-3.ch009 · Cited by 1
He, Y., & Yang, X. (2025) · Geoscientific model development · doi:10.5194/gmd-18-9257-2025 · Cited by 1
Recent Advances in Microfluidic Detection of Circulating Tumor Cells Based on Deep Learning
Liu, Y., Liu, S., & Cheng, C. (2025) · IEEE International Conference on Manipulation, Manufacturing and Measurement on the Nanoscale (3M-NANO) · doi:10.1109/3m-nano65639.2025.11261025 · Cited by 1
Omathil, G. G., & Sreeja, C. S. (2025) · IEEE India Council International Subsections Conference (INDISCON) · doi:10.1109/indiscon66021.2025.11252043 · Cited by 1
Lavín-Pallero, E., & Ruiz-García, M. (2025) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ae1b86 · Cited by 3
Jin, M., Xiao, L., & Wei, Z. (2025) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2025.3641031 · Cited by 1
“Thirdly, information extraction differs across layers [37], [38]: shallow layers capture low-level features such as edges and textures, while deeper layers extract more abstract patterns.”
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.