Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 1,401–1,500 of 2,667
DANG: Data Augmentation Based on NIR-II Guided Diffusion Model for Fluorescence Molecular Tomography
Huang, Q., Li, C., Xiao, A., Tian, J., & Hu, Z. (2025) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2025.3643313 · Cited by 3
“Adiverseandmassivedatasetleads toneuralnetworkswithgoodgeneralizationability[27].”
Guo, Q., Guo, Y., & Zhao, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129733 · Cited by 3
In Which Ways Is Machine Learning Opaque?
Beisbart, C. (2025) · Synthese Library/Synthese library · doi:10.1007/978-3-032-03083-2_1 · Cited by 2
Robust Weakly Supervised Bearing Fault Diagnosis via Dual Learning and Curriculum Strategies
Ding, Y., Miao, Q., Zhao, X., Yan, X., & Lee, C.-G. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3645912 · Cited by 3
“These noises severely constrain the implementation of existing DTL-based fault diagnosis methods, as directly aligning a highly noisy source domain with the target domain often leads to negative transfer [16], [17].”
Partial derivative regularized knowledge transformation for deep neural networks generalization
Morabbi, S., Soltanizadeh, H., Mozaffari, S., & Fadaeieslam, M. J. (2025) · Pattern Analysis and Applications · doi:10.1007/s10044-025-01559-x · Cited by 4
“The primary goal of the explicit methods is to stop overfit - ting in neural networks by reducing the capacity of a model [30].”
KRAM: Knowledge-driven robust training against label noise for medication recommendation
Jin, Y., & Zhang, Z. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131330 · Cited by 3
Yuan, X., Hu, C., Liu, H., Yang, F., Gu, J., & Chai, Z. (2026) · Aerospace Science and Technology · doi:10.1016/j.ast.2026.111890 · Cited by 3
Li, Y., Wu, J., Zhang, X., Gu, X., Xu, J., Cai, H., & Peng, X. (2026) · Agricultural Water Management · doi:10.1016/j.agwat.2026.110223 · Cited by 3
Neural networks generalize on low complexity data
Chatterjee, S., & Sudijono, T. (2026) · The Annals of Statistics · doi:10.1214/25-aos2570 · Cited by 6
Sheikh, M. A. A., Dey, A., & Kole, A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01080-y · Cited by 5
“This not only enhanced model robustness and reduced overfitting but also balanced class distributions [106, 107].”
Zhang, X., Huang, H., Zhang, D., Zhuang, S., Han, S., Liu, H., & Lai, P. (2026) · Advanced Photonics · doi:10.1117/1.ap.8.3.036015 · Cited by 3
Ban, Y., Ren, Y., Dong, W., Sun, D., & Tiwari, B. K. (2026) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2026.105834 · Cited by 3
Explicit matrix gradient expression for residual network
Sun, Y., Zheng, K., Yin, J., Wu, C., & Niu, X. (2023) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2023.2238738 · Cited by 3
“In particular, they are applied in critical security applications for instance surveillance systems (Tourani et al., 2020), autonomous vehicles (Pu et al., 2021; Wang, Yang, et al., 2020; C. Zhang et al., 2021), and big data (Duan et al., 2017; Wang et al., 2020).”
Cannaday, A. B., Davis, C. H., & Bajkowski, T. M. (2023) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss52108.2023.10281409 · Cited by 3
Roughness Index for Loss Landscapes of Neural Network Models of Partial Differential Equations*
Wu, K., Jian, X., Du, R., Chen, J., & Zhou, X. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386909 · Cited by 3
“The loss functions J (θ) are non-convex now, but in practice one can generally find the global minima due to the perfect fitting capability of the neural network [8], [30].”
BackgroundEvoAug-TF: extending evolution-inspired data augmentations for genomic deep learning to TensorFlow
Yu, Y., Muthukumar, S., & Koo, P. K. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae092 · Cited by 3
Short Review on Supervised Learning Under Adversarial Label Poisoning and Clean-Label Data Poisoning
Tavllali, P., & Behzadan, V. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00067 · Cited by 3
“Numerious studies had already been done with the focus of functions that neural nets can learn [42]–[46] but the authors of [4] focus on expressivity of neural nets over a finite sample size.”
BackgroundVisual Analysis of Prediction Uncertainty in Neural Networks for Deep Image Synthesis
Dutta, S., Nizar, F., Amaan, A., & Acharya, A. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3406959 · Cited by 3
Power Control for NN-Based Wireless Distributed Inference With Improved Model Calibration
Hou, Q., Lee, M., Yu, G., & Cai, Y. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3417305 · Cited by 3
Yousefirizi, F., Liyanage, A., Klyuzhin, I. S., & Rahmim, A. (2024) · Journal of medical imaging and radiation sciences · doi:10.1016/j.jmir.2024.101745 · Cited by 3
Silva, G., & Rodríguez, P. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10302-2 · Cited by 4
“For CIFAR-10 and Tiny ImageNet, we performed data augmentation that consists of random horizontal flipping, random cropping and z-score normalization, see code 4 for numerical details.”
Focusing on All Refined Attention Regions for Noisy Label Facial Expression Recognition
Wang, W., & Li, S. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10688109 · Cited by 3
“Specifically, sample selection methods rest on the small loss assumption [1], which might confuse difficult samples and noisy samples as they both yield large loss values during training.”
BackgroundCORAL: Collaborative Automatic Labeling System Based on Large Language Models
Zhu, Z., Wang, Y., Yang, S., Lin, L., Wu, R., Xiu, T., Zhao, J., & Wang, H. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3685800.3685885 · Cited by 3
The phase diagram of kernel interpolation in large dimensions
Zhang, H., Lu, W., & Lin, Q. (2024) · Biometrika · doi:10.1093/biomet/asae057 · Cited by 3
Guan, G., Zhi, T., Cai, H., Cao, Y., & Xie, H. (2024) · IEEE International Conference on Computer and Communication Engineering Technology (CCET) · doi:10.1109/ccet62233.2024.10838122 · Cited by 3
Enhanced Model Robustness by Integrated Local and Global Processing
Wang, L., Ghimire, A., & Santosh, K. (2024) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi62246.2024.00038 · Cited by 3
The Effectiveness of Local Updates for Decentralized Learning Under Data Heterogeneity
Wu, T., Li, Z., & Sun, Y. (2025) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2025.3533208 · Cited by 12
“This leads to the over-parameterization regime with the models learned perfectly interpolating the training data [13], [14].”
MethodASDIA: An Adversarial Sample to Preserve Privacy Program in Federated Learning
Tian, S., Tan, Y., Wang, H., Liu, H., & Li, Z. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3545599 · Cited by 4
“This occurs because deep learning models have a tendency to acquire and retain features from the data that extend beyond the primary learning task, as mentioned in [17].”
Parametric PDE Control with Deep Reinforcement Learning and L0 Sparse Polynomial Policies
Botteghi, N., & Fasel, U. (2024) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc56724.2024.10886215 · Cited by 8
Shao, Y., Wang, S., & Zhao, W. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.112910 · Cited by 3
Graph Neural Networks with Coarse- and Fine-Grained Division for mitigating label noise and sparsity
Li, S., Zhang, B., Song, J., Ruan, G., Wang, C., & Xie, J. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107338 · Cited by 4
Preserving predictive information under biologically plausible compression
Durian, S. C. L., Bojanek, K., Marre, O., & Palmer, S. E. (2026) · PNAS Nexus · doi:10.1093/pnasnexus/pgag224 · Cited by 3
“The stimuli were 20 s movies of trees, grasses, water, fish, and optic flow, which were repeated 83, 80, 84, 91, and 85times, respectively.”
Wang, R., Zhu, Y., & Xia, D. (2024) · International Conference on Information Technology in Medicine and Education · doi:10.1109/itme63426.2024.00112 · Cited by 3
“Even when trained with mislabeled data, deep models can fit these anomalous relationships well [10].”
Concrete section segmentation with advanced deep learning models and refined labeling approaches
Mesfin, W. M., Kim, G., & Kim, H.-K. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127697 · Cited by 4
Mohottala, S., Gawesha, A., Kasthurirathna, D., Samarasinghe, P. D., & Abhayaratne, C. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104410 · Cited by 3
Cross patient seizure detection via unsupervised domain adaptation based on uncertainty estimation
Wang, S., Lv, H., Feng, H., Peng, H., Feng, W., Nie, C., & Zhao, Y. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06755-0 · Cited by 3
“Drawing on Zhang et al.’s findings [20] that neural networks may overfit datasets containing noisy labels, leading to degraded generalization performance.”
G-CutMix: A CutMix-based graph data augmentation method for bot detection in social networks
Li, Y., Shi, S., Guo, X., Zhou, C., & Hu, Q. (2025) · PLoS ONE · doi:10.1371/journal.pone.0331978 · Cited by 3
Unicorn: Unified Neural Image Compression with One Number Reconstruction
Zheng, Q., Wang, H., Liu, Z. Q., Liu, J., Hao, Z., Chen, B., Li, M., Rui, W., Liu, P., Lu, Y., et al. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755599 · Cited by 7
“Giventhattheoverfittingproblemoffake/random labels can be easily handled by neural networks [46], we initialize a set of fake/random index uniformly sampled in { 0 , ...,𝑀 − 1 } to construct a bijection function with the image set, denoted as = 0 .”
Interpretable and efficient data-driven discovery and control of distributed systems
Wolf, F., Botteghi, N., Fasel, U., & Manzoni, A. (2025) · Data-Centric Engineering · doi:10.1017/dce.2025.10027 · Cited by 4
“We obtain the following dynamics equations: • 8-dimensional Latent space:”
Probabilistic Models, Compressible Interactions, and Neural Coding
Ramirez, L., Bialek, W., Palmer, S. E., & Schwab, D. J. (2026) · PRX Life · doi:10.1103/pj4s-g4jc · Cited by 5
“In truth, we do not understand the success of these models [43,44].”
Self-Regularity of Non-Negative Output Weights for Overparameterized Two-Layer Neural Networks
Gamarnik, D., Kızıldağ, E. C., & Zadik, I. (2022) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3156702 · Cited by 3
Does SGD Implicitly Optimize for Smoothness?
Volhejn, V., & Lampert, C. H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-71278-5_18 · Cited by 3
Data-informed deep optimization
Zhang, L., Xu, Z. J., & Zhang, Y. (2022) · PLoS ONE · doi:10.1371/journal.pone.0270191 · Cited by 3
Gaussian Perturbations in ReLU Networks and the Arrangement of Activation Regions
Daróczy, B. (2022) · Mathematics · doi:10.3390/math10071123 · Cited by 4
“Based on [29], where the authors of [17] argue that measuring the difference in complexity based on random and true labels may be misleading, we choose to capture generalisation with the empirical difference in loss as in [1,23].”
BackgroundRidgeless Regression with Random Features
Li, J., Liu, Y., & Zhang, Y. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/445 · Cited by 3
“However, recent studies empirically observed that neural networks still can interpolate the training data and generalize well when λ = 0 [ Zhang et al. , 2021; Belkin et al. , 2018 ] .”
BackgroundOn how to avoid exacerbating spurious correlations when models are overparameterized
Behnia, T., Wang, K., & Thrampoulidis, C. (2022) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit50566.2022.9834839 · Cited by 5
Mahdavi-Hormat, A., Menhaj, M. B., & Shakarami, A. (2022) · Advances in Computational Intelligence · doi:10.1007/s43674-022-00046-8 · Cited by 3
Gütter, J., Niebling, J., & Zhu, X. X. (2022) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss46834.2022.9884570 · Cited by 3
Learning from the Past: Regularization by Validation
Zhang, E., Wahib, M., & Munetomo, M. (2022) · Joint International Conference on Soft Computing and Intelligent Systems and International Symposium on Advanced Intelligent Systems (SCIS&ISIS) · doi:10.1109/scisisis55246.2022.10002143 · Cited by 3
Privacy Leakage in GAN Enabled Load Profile Synthesis
Huang, J., & Wu, C. (2022) · IEEE Sustainable Power and Energy Conference (iSPEC) · doi:10.1109/ispec54162.2022.10033029 · Cited by 3
“random labels, which means there is no feature to learn at all, a neural network is still capable of achieving zero loss, indicating that the network has memorized all of them [12].”
BackgroundA Noise Rate Estimation Method for Image Classification with Label Noise
Chen, Z., Song, A., Wang, Y., Huang, X., & Kong, Y. (2023) · Journal of Physics Conference Series · doi:10.1088/1742-6596/2433/1/012039 · Cited by 3
“With limited budgets, the machine learning models would be misguided by the noisy datasets and memorize wrong relations [3, 4].”
BackgroundTowards Understanding Variation-Constrained Deep Neural Networks
Li, G., & Ding, J. (2023) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2023.3251747 · Cited by 3
Capacity Abuse Attack of Deep Learning Models Without Need of Label Encodings
Luo, W., Zhang, L., Wu, Y., Liu, C., Han, P., & Zhuang, R. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3266419 · Cited by 4
“Its principle is to use the powerful fitting ability of the deep neural network model [27].”
BackgroundData-Driven Interaction-Aware Behavior Prediction for Autonomous Vehicles
Westny, T. (2023) · Linköping studies in science and technology. Thesis · doi:10.3384/9789180751797 · Cited by 3
Physical Backdoor Attack using Multi-Trigger to Same Class
Singh, G. P., Kumar, J., & Mohapatra, R. K. (2023) · International Conference on Artificial Intelligence and Signal Processing (AISP) · doi:10.1109/aisp57993.2023.10134794 · Cited by 3
Local Intrinsic Dimensional Entropy
Ghosh, R., & Motani, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25935 · Cited by 3
Memorization Weights for Instance Reweighting in Adversarial Training
Zhang, J., Hong, Y., & Zhao, Q. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26329 · Cited by 3
“As well known, overparameterized DNNs have tremendous capacity to make them easy to perfectly fit the training dataset (Zhang et al. 2021; Neyshabur et al. 2017; Belkin et al. 2019; Bubeck and Sellke 2021; Zhang et al. 2017).”
BackgroundITFD: an instance-level triplet few-shot detection network under weighted pair-resampling
Chen, X., Peng, C., Qiu, C., Luo, L., Huang, D., & Liu, Z. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04600-w · Cited by 3
“Although the effectiveness of deep learning methods has been proved in many areas, human beings are capable of quickly learning from very few-shot of samples without thousands of repeat [4].”
BackgroundEnsemble Methods for Multi-Organ Segmentation in CT series
Crespi, L., Roncaglioni, P., Dei, D., Franzese, C., Lambri, N., Loiacono, D., Mancosu, P., & Scorsetti, M. (2023) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms58004.2023.00270 · Cited by 3
Doubly Robust AUC Optimization against Noisy and Adversarial Samples
Zhang, C., Shi, W., Luo, L., & Gu, B. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599316 · Cited by 4
“Overfitting these noisy samples will eventually result in poor generalization on the test dataset [68].”
BackgroundSelf-citationLanguage Models Understand Us, Poorly
Moore, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.16 · Cited by 6
Wei, J., Zhang, Q., & Ning, W. (2023) · Digital Signal Processing · doi:10.1016/j.dsp.2023.104199 · Cited by 3
Data Engineering Challenges in AI automation
Reddy, D. (2023) · International Conference on Computing, Electronics & Communications Engineering (iCCECE) · doi:10.1109/iccece59400.2023.10238496 · Cited by 3
Chu, J., Cao, W., Gan, C., & Yang, Y. (2023) · Chinese Control Conference (CCC) · doi:10.23919/ccc58697.2023.10239801 · Cited by 3
“From 1), the width of the model improves the hyper-segmentation accuracy by obtaining enough feature information, and similarly enough feature information (greater gradient correlation) can ensure the generalization ability of the model[11].”
BackgroundNeri, M., Pallotta, L., & Carli, M. (2023) · International symposium on image and signal processing and analysis/ISPA ... · doi:10.1109/ispa58351.2023.10279538 · Cited by 3
“Moreover, regarding the computational complexity of deep neural networks, as explained in [26], having a smaller number of learnable parameters can help mitigate the risk of overfitting, especially when dealing with small datasets.”
BackgroundKaloev, M., & Krastev, G. (2023) · International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT) · doi:10.1109/ismsit58785.2023.10304968 · Cited by 3
“"Understanding Deep Learning Requires Rethinking Generalization" [9] addresses the challenges of overfitting in deep learning.”
BackgroundAntonietti, A. (2023) · IEEE Signal Processing in Medicine and Biology Symposium · doi:10.1109/spmb59478.2023.10372728 · Cited by 3
“Our findings demonstrate that these carefully chosen features capture the salient elements of the 3-axes acceleration data, enabling accurate classification without sacrificing computational efficiency [18, 19].”
BackgroundManipulating Feature Visualizations with Gradient Slingshots
Bareeva, D., Höhne, M., Warnecke, A., Pirch, L., Müller, K., Rieck, K., Lapuschkin, S., & Bykov, K. (2025) · neural information processing systems · doi:10.52202/085713-5560 · Cited by 5
Zhong, Y., Tang, H., Chen, J., & Wang, Y.-X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01496 · Cited by 6
Column Row Convolutional Neural Network: Reducing Parameters for Efficient Image Processing
Im, S., Jeong, J., Lee, J., Shin, C., Cho, J. H., & Ju, H. (2024) · Neural Computation · doi:10.1162/neco_a_01653 · Cited by 3
“It is well known that the performance of deep learning models can be improved by increasing the number of layers or nodes (Zhang et al., 2021).”
BackgroundKashiwagi, M., Maeda, K., Togo, R., Ogawa, T., & Haseyama, M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446840 · Cited by 3
“These annotation processes are prone to introduce mistaken labels called noisy labels into training datasets, which has been proven to result in the degradation of the generalization performance of DNNs [4].”
BackgroundDual-Mix for Cross-Modal Retrieval with Noisy Labels
Ding, F., Liu, X., Wang, X., & Zhong, F. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446808 · Cited by 5
“According to [5–7], Deep Neural Networks (DNN) can easily overfit to noisy labels within training.”
BackgroundLabel Noise Robust Crowd Counting with Loss Filtering Factor
Xu, Z., Lin, H. Y., Chen, Y., & Li, Y. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2329859 · Cited by 3
“The effectiveness of deep neural networks is dependent on having access to high-quality labeled training data because label mistakes (label noise) in training data can significantly impair model performance on clean test data (Zhang et al. 2021).”
High-Dimensional Analysis for Generalized Nonlinear Regression: From Asymptotics to Algorithm
Li, J., Liu, Y., & Wang, W. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29253 · Cited by 4
Mukherjee, S., & Huberman, B. A. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-47724-9_24 · Cited by 4
Zhang, H., Chen, Q., Xue, B., Banzhaf, W., & Zhang, M. (2024) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3638529.3654075 · Cited by 3
“In the context of feature construction, it has been observed that large feature construction layers in deep neural networks can generalize effectively [42].”
Open-vocabulary object detection via debiased curriculum self-training
Zhang, H., Guan, D., Ke, X., Saddik, A. E., & Lu, S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124762 · Cited by 4
Aggregation Strategy with Gradient Projection for Federated Learning in Diagnosis
Lin, H., Gao, Y., Li, H., Zhang, X., Yu, X., Chen, J., & Liu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5689-6_18 · Cited by 3
Natural low-illumination image enhancement based on dual-channel prior information
Wang, L. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e35831 · Cited by 5
“Better generalization [30] is still the focus of deep learning.”
BackgroundDynamic training for handling textual label noise
Cheng, S., Chen, W., Liu, W., Zhou, L., Zhao, H., Kong, W., Qu, H., & Fu, M. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05738-x · Cited by 4
“These noisy labels can easily lead to the overfitting of DNNs, resulting in poor generalization performance on test data or in real-world applications [52].”
Deep learning‐based gesture recognition for surgical applications: A data augmentation approach
Santiago, S. S., & Cifuentes, J. (2024) · Expert Systems · doi:10.1111/exsy.13706 · Cited by 4
“Data augmentation is a strategy employed to artificially expand the training dataset, making it appear as if the new data is generated from the same underlying distribution (Zhang, Bengio, et al., 2021).”
Weak Devices Matter: Model Heterogeneity in Resource-Constrained Federated Learning
Peng, T., Huang, C., Pang, A., & Chiu, T.-C. (2024) · International Conference on Communications · doi:10.1109/icc51166.2024.10622675 · Cited by 3
“Thus, effectively improving the contribution of weak devices is the key to solving this challenge. largest width of split models can prevent overfitting from complex models [18] [19].”
BackgroundBeheshti, S., Shamsi, M., Sadat-Nejad, Y., & Zhou, M. (2024) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds60720.2024.00009 · Cited by 3
“One key to generalizability lies in the model’s ability to capture the underlying patterns in the data without overfitting [7].”
BackgroundCausal debiasing for unknown bias in histopathology—A colon cancer use case
Correa, R., Pai, R. K., Ebare, K., Buchanan, D. D., Jenkins, M. A., Phipps, A. I., Newcomb, P. A., Gallinger, S., Grant, R. C., Marchand, L. L., et al. (2024) · PLoS ONE · doi:10.1371/journal.pone.0303415 · Cited by 3
Hermosillo-Reynoso, F., & Torres-Román, D. (2024) · Sensors · doi:10.3390/s24237463 · Cited by 3
Deep Learning Based Satellite Image Segmentation Analyzes for Coastal Flood Disaster
Hayatunnufus, H., Candra, A., Slamet, B., Jaya, I., Wilson, & Syadzwana, H. (2024) · International Conference on Electrical, Telecommunication and Computer Engineering (ELTICOM) · doi:10.1109/elticom64085.2024.10864384 · Cited by 3
“Artificial neural networks in deep learning try to mimic the human brain through a combination of input data, weights, and biases [4].”
Sufficient Learning for Label Noise with Dual-Regularization
Dong, Y., Wu, G., Wang, J., & Wang, Z. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889183 · Cited by 3
“During training, DNNs tend to memorize the noise in the training data rather than learning meaningful patterns [6], leading to a decrease in performance.”
Soto‐Ferrari, M. (2025) · Computers & Industrial Engineering · doi:10.1016/j.cie.2025.111071 · Cited by 4
Shenvi, R. (2024) · IEEE MIT Undergraduate Research Technology Conference (URTC) · doi:10.1109/urtc65039.2024.10937527 · Cited by 3
“Explicit regularization [6, 7] is a related approach in which additional constituent terms are added to the objective function.”
Combining data augmentation and model fine-tuning for learning from limited data
Shi, H., Zhang, Y., & Wan, B. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02611-2 · Cited by 3
“Single-hidden-layer neural networks were chosen because they can represent any function of the input instances if the number of parameters exceeds the number of instances [50].”
Safety Risks in Fine-Tuning Large Language Models
Chen, P., & Liu, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-76770-8_14 · Cited by 4
RoNFA: Robust neural field-based approach for few-shot image classification with noisy labels
Xiang, N., Xing, L., & Jin, D. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.09.009 · Cited by 3
A Hessian-aware stochastic differential equation for modelling SGD
Li, X., Shen, Z., Zhang, L., & He, N. (2026) · Mathematical Programming · doi:10.1007/s10107-026-02333-5 · Cited by 5
Instance-Dependent Early Stopping for Adaptive Data Pruning
Yuan, S., Lin, R., Azian, F., Feng, L., Han, B., Niu, G., Sugiyama, M., & Liu, T. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3693108 · Cited by 5
Bharadwaj, P. V., Jeevan, T. P., Suvin, P. S., & Jayaram, S. R. (2019) · Applied Mechanics and Materials · doi:10.4028/www.scientific.net/amm.895.52 · Cited by 3
On the theory of deep learning: A theoretical physics perspective (Part I)
de Lara, A. C. M. (2023) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2023.129308 · Cited by 3
Fetal Movement Identification from Multi-Accelerometer Measurements using Recurrent Neural Networks
Senanayaka, J. B., Somathilake, E., Delay, U. H., Gunarathne, S., Godaliyadda, R. I., Ekanayake, P., Wijayakulasooriya, J., & Rathnayake, C. (2021) · IEEE International Conference on Industrial and Information Systems (ICIIS) · doi:10.1109/iciis53135.2021.9660747 · Cited by 3
Determinant Factors of Entrepreneurship Learning on Entrepreneurial Intention in Higher Education
Arni, Y., Siswandari, S., Akhyar, M., & Asrowi, A. (2022) · International Conference on Economics, Business and Economic Education Science (ICE-BEES) · doi:10.4108/eai.27-7-2021.2316845 · Cited by 3
Ensemble learning via supervision augmentation for white matter hyperintensity segmentation
Guo, X., Ye, C., Yang, Y., Zhang, L., Li, L., Lu, S., Lv, H., Guo, C., & Ma, T. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.946343 · Cited by 4
“Since the high capacity of deep neural networks (DNN), the DL-based approaches are easy to overfit to labels with noise and uncertainty, which may lead to biased models with weak generalization ability (Lee et al., 2016; Kohl et al., 2018; Zhang et al., 2021).”
BackgroundOpenImages Cyclists: Expandindo a Generalização na Detecção de Ciclistas em Câmeras de Segurança
Nardi, E. E. D. S., Padilha, B., Kamaura, L. T., & Ferreira, J. E. (2022) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd.2022.224626 · Cited by 3
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.