Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 1,001–1,100 of 2,667
Augmentation Matters: A Mix-Paste Method for X-Ray Prohibited Item Detection Under Noisy Annotations
Chen, R., Yan, Y., Xue, J., Lu, Y., & Wang, H. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3516546 · Cited by 9
Large Language Models (LLMs): A systematic study in Administration and Business
Pessanha, G. R. G., Vieira, A. G., & Brandão, W. C. (2024) · RAM. Revista de Administração Mackenzie · doi:10.1590/1678-6971/eramd240059 · Cited by 9
Nishiyama, S., & Ohzeki, M. (2024) · Journal of the Physical Society of Japan · doi:10.7566/jpsj.94.014802 · Cited by 7
Pi-Vision: Leveraging YOLO V8 for Real-Time Person Detection and Recognition on Raspberry Pi
Kumar, P., Choudhary, S. K., Pallerla, S., Sangani, S., Thota, A., & Matta, V. (2024) · International Conference on Data Intelligence and Cognitive Informatics (ICDICI) · doi:10.1109/icdici62993.2024.10810836 · Cited by 7
Heterogeneous Feature Re-Sampling for Balanced Pedestrian Attribute Recognition
Zhou, Y.-B., Li, B., Hu, H.-M., Zhang, X.-K., Zhang, D.-P., & Wang, H.-Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3526930 · Cited by 10
End-to-end physics-informed multi-branch GAN for enhanced DoFP polarization image reconstruction
Liu, H., Gu, Z., Shi, S., Li, Z., Lei, X., Liu, Z., Song, M., Yang, J., Li, C., Zhou, Z., et al. (2025) · Optics Express · doi:10.1364/oe.547918 · Cited by 7
Spicker, D., Nazemi, A., Hutchinson, J. M., Fieguth, P., Kirkpatrick, S. I., Wallace, M., & Dodd, K. W. (2025) · Statistics in Medicine · doi:10.1002/sim.70013 · Cited by 7
“Previous work has shown that, for many architectures of neural networks, adding noise to the input data does not lead to an increased training loss [44].”
Carlson, D. E., Chavarriaga, R., Liu, Y.-L., Lotte, F., & Lu, B.-L. (2025) · Journal of Neural Engineering · doi:10.1088/1741-2552/adbfbd · Cited by 9
“In most real-world scenarios, a well-performing DL model has training error that is essentially zero [18].”
Achieving Robust Channel Estimation Neural Networks by Designed Training Data
Luan, D., & Thompson, J. (2025) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2025.3592299 · Cited by 7
“Recent discussions on neural network generalization are provided in [27]–[30].”
Few-Shot Learning on Edge Devices Using CLIP: A Resource-Efficient Approach for Image Classification
Lu, J. (2024) · Information Technology and Control · doi:10.5755/j01.itc.53.3.36943 · Cited by 9
Scaling deep learning for material imaging with a pseudo 3D model for domain transfer
Tang, K., Armstrong, R. T., Mostaghimi, P., Niu, Y., Meyer, Q. P.-G., Zhao, C., Finegan, D. P., Popeil, M., Singh, K., Menke, H., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-66449-6 · Cited by 11
Tensor network to learn the wave function of data
Dymarsky, A. Y., & Pavlenko, K. (2022) · Physical Review Research · doi:10.1103/physrevresearch.4.043111 · Cited by 8
Lin, G., Bai, H., Zhao, J., Yun, Z., Chen, Y., Pang, S., & Feng, Q. (2022) · Medical Physics · doi:10.1002/mp.15627 · Cited by 7
Migliorelli, L., Frontoni, E., & Moccia, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117458 · Cited by 7
Nessy: A Neuro-Symbolic System for Label Noise Reduction
Smirnova, A., Yang, J., Yang, D., & Cudré-Mauroux, P. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3199570 · Cited by 7
Soltane, I. B., Hallo, G., Lacombe, C., Lamaignère, L., Bonod, N., & Néauport, J. (2022) · Journal of the Optical Society of America A · doi:10.1364/josaa.462367 · Cited by 7
“Despite the growing amount of reviews highlighting this ability [24], the underlying mechanisms of such performance have not been unveiled [25].”
BackgroundProstate Cancer Histology Synthesis Using StyleGAN Latent Space Annotation
Daroach, G., Duenweg, S. R., Brehler, M., Lowman, A. K., Iczkowski, K. A., Jacobsohn, K. M., Yoder, J. A., & LaViolette, P. S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16434-7_39 · Cited by 7
Pseudo-shot Learning for Soil Classification With Laser-Induced Breakdown Spectroscopy
Huang, Y., Bais, A., & Hussein, A. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3262503 · Cited by 8
“It is demonstrated that deep neural networks easily fit false pseudolabels, which deteriorates the model’s test performance [41].”
BackgroundGeneration of Synthetic CPTs with Access to Limited Geotechnical Data for Offshore Sites
Shoukat, G., Michel, G., Coughlan, M., Malekjafarian, A., Thusyanthan, I. N., Desmond, C., & Pakrashi, V. (2023) · Energies · doi:10.3390/en16093817 · Cited by 7
The Implicit Regularization of Momentum Gradient Descent in Overparametrized Models
Wang, K., Fu, Z., Zhou, Y., & Yan, Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26209 · Cited by 9
“This finding, along with a series of studies (Gunasekar et al. 2018; Soudry et al. 2018; Var-di and Shamir 2021; Zhang et al. 2021), suggest that while the hypothesis space itself is extremely complex, the search strategy of optimization favors certain structured solution as if some explicit…”
BackgroundInformation theoretic perspective on sample complexity
Pereg, D. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.08.032 · Cited by 7
AI - Limits and Prospects of Artificial Intelligence
Klimczak, P., & Petersen, C. (2023) · AI critique · doi:10.1515/9783839457320 · Cited by 10
Label correction using contrastive prototypical classifier for noisy label learning
Xu, C., Lin, R., Cai, J., & Wang, S. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119647 · Cited by 8
“However, recent studies indicate that training deep neural networks with cross-entropy can cause a model to fit and memorize arbitrary labels, ultimately leading to decreased performance and generalization [5].”
BackgroundHeavy-Tailed Regularization of Weight Matrices in Deep Neural Networks
Xiao, X., Li, Z., Xie, C., & Zhou, F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44204-9_20 · Cited by 14
Design of Undersampled Seismic Acquisition Geometries via End-to-End Optimization
Hernandez-Rojas, A., & Argüello, H. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3339119 · Cited by 6
“This flexibility allows enhanced representation learning of latent weights, and model flexibility, and facilitates adaptability during training [49], [50], [51].”
BackgroundCoDC: Accurate Learning with Noisy Labels via Disagreement and Consistency
Dong, Y., Li, J., Wang, Z., & Jia, W. (2024) · Biomimetics · doi:10.3390/biomimetics9020092 · Cited by 7
“However, these methods can introduce noise to the labels, which can seriously affect the generalization of DNNs during the training process [16].”
Time-Domain Speech Super-Resolution With GAN Based Modeling for Telephony Speaker Verification
Kataria, S., Villalba, J., Moro-Velázquez, L., Żelasko, P., & Dehak, N. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3369536 · Cited by 7
Generative Quantile Regression with Variability Penalty
Wang, S., Shin, M., & Bai, R. (2023) · Journal of Computational And Graphical Statistics · doi:10.1080/10618600.2024.2327824 · Cited by 12
“…overparameterization improves the generalization and robustness of DNNs, since it greatly enhances the representation power of the DNN and simplifies the optimization landscape (Allen-Zhu, Li, and Liang 2019; Soltanolkotabi, Javanmard, and Lee 2019; Zhang et al. 2021; Montanari and Zhong 2022).”
Adaptive Learning for Soil Classification in Laser-Induced Breakdown Spectroscopy Streaming
Huang, Y., Mohajan, S., Beier, N. F., Wan, Y., Lamothe, S., Bais, A., Dyck, M. F., Hegmann, F. A., & Hussein, A. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3375260 · Cited by 9
“However, it is still limited because of inevitable false pseudolabels, which can deteriorate the model’s test performance [72].”
BackgroundOdio-BERT: Evaluating domain task impact in hate speech detection
Yigezu, M. G., Kolesnikova, O., Gelbukh, A. F., & Sidorov, G. (2024) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-219349 · Cited by 7
“Linear classifiers have consistently demonstrated their competitive perfor-ISSN mance, often rivaling, and in some cases surpassing, the effectiveness of neural networks across machine learning tasks [22–24].”
FormulAI: Designing Rule-Based Datasets for Interpretable and Challenging Machine Learning Tasks
Tissot, H. C. (2024) · Artificial Intelligence and Applications · doi:10.47852/bonviewaia42021781 · Cited by 10
Shen, J. G., Li, Z., Lu, Y., Pan, M., & Li, X. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00435-y · Cited by 7
“Mislabeled samples can significantly impact the performance of deep predictive models (Krause et al. 2016; Zhang et al. 2021).”
InfluentialKumar, D. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09678-y · Cited by 10
“For instance, [19] introduced a transfer learning-oriented method for autonomous driving under varying weather conditions.”
Feature Noise Boosts DNN Generalization Under Label Noise
Zeng, L., Chen, X., Shi, X., & Shen, H. T. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3394511 · Cited by 11
“It is worth noting that DNNs with a sufficient number of parameters can potentially fit random label data given adequate training [1].”
BackgroundIteratively Calibratable Network for Reliable EEG-Based Robotic Arm Control
Lee, B.-H., Cho, J.-H., Kwon, B.-H., Lee, M., & Lee, S. (2024) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2024.3434983 · Cited by 7
SoK: Unintended Interactions among Machine Learning Defenses and Risks
Duddu, V., Szyller, S., & Asokan, N. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00243 · Cited by 16
Iterative Filter Pruning for Concatenation-based CNN Architectures
Pavlitska, S., Bagge, O., Peccia, F. N., Mammadov, T., & Zöllner, J. M. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650334 · Cited by 8
SOAMC: A Semi-Supervised Open-Set Recognition Algorithm for Automatic Modulation Classification
Di, C., Ji, J., Sun, C., & Liang, L. (2024) · Electronics · doi:10.3390/electronics13214196 · Cited by 8
Generalization Guarantees of Gradient Descent for Shallow Neural Networks
Wang, P., Lei, Y., Wang, D., Ying, Y., & Zhou, D. (2024) · Neural Computation · doi:10.1162/neco_a_01725 · Cited by 15
Learning with noisy labels via clean aware sharpness aware minimization
Huang, B., Zhang, P., Xie, Y., & Xu, C. (2025) · Scientific Reports · doi:10.1038/s41598-025-85679-8 · Cited by 10
Legnaro, E., Guastavino, S., Piana, M., & Massone, A. M. (2025) · The Astrophysical Journal · doi:10.3847/1538-4357/adb41a · Cited by 7
Power Characterization of Noisy Quantum Kernels
Wang, Y., Qi, B., Wang, X., Liu, T., & Dong, D. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3545545 · Cited by 11
“To make accurate predictions on unseen data, both the training and generalization errors should be small [28].”
From Robustness to Improved Generalization and Calibration in Pre-trained Language Models
Jukić, J., & Šnajder, J. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00739 · Cited by 8
RL Perceptron: Generalization Dynamics of Policy Learning in High Dimensions
Patel, N., Lee, S., Mannelli, S. S., Goldt, S., & Saxe, A. (2025) · Physical Review X · doi:10.1103/physrevx.15.021051 · Cited by 14
“This approach focuses on worst-case analysis, which typically differs significantly from practice (at least in supervised learning [42]).”
Xu, Q., Wu, T., Wang, Y., Li, X., Yu, H., Cen, S., & Li, Z. (2025) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2025.101406 · Cited by 8
“This process enables LLMs to develop rich language understanding and generation skills, including vocabulary relationships, grammar, and context dependencies [53 – 55].”
Shell Theory: A Statistical Model of Reality
Lin, W.-Y., Liu, S.-Y., Ren, C., Cheung, N.-M., Li, H., & Matsushita, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3084598 · Cited by 8
An approach to improving sound-based vehicle speed estimation
Bulatović, N., & Djukanović, S. (2022) · Zooming Innovation in Consumer Technologies Conference (ZINC) · doi:10.1109/zinc55034.2022.9840552 · Cited by 7
Augmentation Pathways Network for Visual Recognition
Bai, Y., Zhou, M., Zhang, W., Zhou, B., & Mei, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3250330 · Cited by 7
“In some cases, deep models are likely to overfit the training data as well as its noisy signals [1], even on a large-scale dataset such as ImageNet [2], [3].”
BackgroundAziz, M. T., Mahmud, S., Goh, M. K. O., & Nandi, D. (2024) · Egyptian Informatics Journal · doi:10.1016/j.eij.2024.100479 · Cited by 8
Asymptotic Spectral Representation of Linear Convolutional Layers
Yi, X. (2022) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3140718 · Cited by 7
Learning curves for deep structured Gaussian feature models*
Zavatone-Veth, J. A., & Pehlevan, C. (2024) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ad642a · Cited by 8
“Characterizing how data structure and model architecture affect generalization performance is among the foremost goals of deep learning theory [1, 2].”
BackgroundFoolish Crowds Support Benign OverfittingPreprint
Chatterji, N. S., & Long, P. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02914 · Cited by 7
Task Affinity with Maximum Bipartite Matching in Few-Shot LearningPreprint
Le, C. P., Dong, J., Soltani, M., & Tarokh, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02399 · Cited by 15
“This is because neural networks are not invariant to the label’s permutation (Zhang et al., 2021).”
BackgroundUnveiling the Hessian's Connection to the Decision BoundaryPreprint
Sabanayagam, M., Behrens, F., Adomaityte, U., & Dawid, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.07104 · Cited by 12
“However, these results are extensively contested and discussed (Zhang et al., 2021), since flatness is not a well-defined concept in nonconvex landscapes of deep models (Dinh et al., 2017).”
BackgroundDeep Neural Network Estimation in Panel Data Models
Chronopoulos, I., Chrysikou, K., Kapetanios, G., Mitchell, J., & Raftapostolos, A. (2023) · Working paper · doi:10.26509/frbc-wp-202315 · Cited by 7
Hong, Z., Wang, H., Zada, Z., Gazula, H., Turner, D. D., Aubrey, B., Niekerken, L., Doyle, W., Devore, S., Dugan, P. C., et al. (2024) · eLife · doi:10.7554/elife.101204.1 · Cited by 21
The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text GenerationPreprint
Carlsson, F., Liu, F., Ward, D., Kurfalı, M., & Nivre, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.04318 · Cited by 10
“These phenomena, which seemingly break the foundations of statistical learning theory, are often attributed to the over-parameterization of large networks in relation to the training data volume He et al. (2016); Zhang et al. (2021).”
Tian, J., & Yang, W. (2024) · Entropy · doi:10.3390/e26110902 · Cited by 6
“The feature extractor comprises two convolutional neural network layers, each utilizing [ 10, 20 ] 3 × 3 convolutional kernels, followed by two fully connected layers containing [ 160, 80 ] neurons, respectively.”
Qian, X., Zhu, S., Chen, Q., Li, Y., Fu, Y., Wei, B., Huang, T., Ma, J., Wang, S., Zhang, Z., et al. (2025) · Spectrochimica Acta Part A Molecular and Biomolecular Spectroscopy · doi:10.1016/j.saa.2025.126748 · Cited by 6
Empirical Analysis of SHAP Stability Under Data Corruption Across Datasets and Model Architectures
Stow, M. T., & Stewart, A. A. (2025) · IARJSET · doi:10.17148/iarjset.2025.12810 · Cited by 7
GCSAM: Gradient Centralized Sharpness Aware Minimization
Hassan, M., Vakanski, A., Zhang, B., & Xian, M. (2025) · IEEE Access · doi:10.1109/access.2025.3624332 · Cited by 8
Superior resilience to poisoning and amenability to unlearning in quantum machine learning
Chen, Y., & Zhang, S. (2026) · Nature Communications · doi:10.1038/s41467-026-70420-4 · Cited by 11
“A further open question arises when considering the scaling of these models.”
Basha, S. S., Rao, A. N., & Fariz, T. K. N. (2025) · Frontiers in Environmental Science · doi:10.3389/fenvs.2025.1692076 · Cited by 6
Jo-SNC: Combating Noisy Labels Through Fostering Self- and Neighbor-Consistency
Sun, Z., Yao, Y., Liu, T., Li, Z., Shen, F., & JinHui, T. A. N. G. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3646737 · Cited by 6
Muhammad, J., Ghergherehchi, M., Ali, S., Song, H. S., & Rahim, N. (2026) · PLoS ONE · doi:10.1371/journal.pone.0342062 · Cited by 9
“Zhang et al. [40] advanced this work by integrating convolutional neural networks (CNNs) with long short-term memory (LSTM) models on the PPMI dataset, reporting an accuracy of 85.3 % and an F1-score of 84.5 %.”
InfluentialYang, Y., Franz, D., Esen, C., & Hellmann, R. (2023) · Journal of Laser Applications · doi:10.2351/7.0001169 · Cited by 6
Robust Image Classification With Noisy Labels by Negative Learning and Feature Space Renormalization
Wu, H., & Sun, J. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3387831 · Cited by 7
Generalized predictive analysis of reactions in paper devices via graph neural networks
Sun, H., Pan, Y., Dong, H., Liu, C., Yang, J., Tao, Y., & Jia, Y. (2024) · Sensors and Actuators B Chemical · doi:10.1016/j.snb.2024.136085 · Cited by 6
Subclass consistency regularization for learning with noisy labels based on contrastive learning
Sun, X., & Zhang, S. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128759 · Cited by 6
Bi-TTA: Bidirectional Test-Time Adapter for Remote Physiological Measurement
Li, H., Lü, H., & Chen, Y.-C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73247-8_21 · Cited by 14
Uncertainty-Informed Volume Visualization using Implicit Neural Representation
Saklani, S., Goel, C., Bansal, S., Wang, Z., Dutta, S., Athawale, T. M., Pugmire, D., & Johnson, C. R. (2024) · doi:10.1109/uncertaintyvisualization63963.2024.00013 · Cited by 6
“Despite the widespread popularity of deep learning techniques, there are significant concerns regarding their interpretability, robustness, and generalizability in real-world applications [66].”
BackgroundIntelligent Information Processing Technology for Digital Libraries Based on Deep Learning
Wang, L., & Yu, X. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.10.056 · Cited by 7
The Method of Gaps: Exact Expressions for the Generalization Error of Supervised Learning Algorithms
Perlaza, S. M., & Zou, X. (2026) · IEEE Transactions on Information Theory · doi:10.1109/tit.2026.3732982 · Cited by 11
Fu, Y., Liang, X., Yang, X., Li, L., Meng, L., Wei, Y., Huang, D., & Qin, Y. (2024) · APL Bioengineering · doi:10.1063/5.0207658 · Cited by 9
“24,25 To address these concerns, an ensemble machine learning approach can be utilized.”
Mining label-free consistency regularization for noisy facial expression recognition
Tan, Y., Xia, H., & Song, S. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01722-7 · Cited by 6
“Generally, sample selection-based methods treat samples with small losses as clean samples and abandon or relabel the remaining samples [46–49].”
Huang, P., Chen, K., Wang, C., & Liu, W. (2025) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2024.3522307 · Cited by 6
“To prevent overfitting and improve the generalization, reducing the model’s complexity is beneficial [30].”
SASD: Self-Attention for Small Datasets—A case study in smart villages
Bolaños-Martinez, D., Durán-López, A., Garrido, J. L. G., Delgado‐Márquez, B. L., & Bermúdez-Edo, M. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126245 · Cited by 7
Loci: Federated Continual Learning of Heterogeneous Tasks at Edge
Luopan, Y., Han, R., Zhang, Q., Zuo, X., Liu, C. H., Wang, G., & Chen, L. Y. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3531123 · Cited by 6
“Similarity measurement based on weight: Based on the parameter space theory [59], this method measures the similarity between two models in their parameter space.”
Possart, D., Mill, L., Vollnhals, F., Hildebrand, T., Suter, P., Hoffmann, M., Utz, J., Augsburger, D., Thies, M., Gu, M., et al. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01702-6 · Cited by 6
“However, their ability to generalize to novel, unseen data critically depends on the availability of representative training datasets 16 , as these datasets determine the data distribution from which diverse and class-de fi ning featuresare derived 17 .”
Huang, W., Wang, T., & Cui, W. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3541037 · Cited by 6
“Second, the quality of the training data plays a critical role in determining the mapping performance of the trained network [36].”
Representation and regression problems in neural networks: Relaxation, generalization, and numerics
Liu, K., & Zuazua, E. (2025) · Mathematical Models and Methods in Applied Sciences · doi:10.1142/s0218202525500228 · Cited by 7
DSDIR: A Two-Stage Method for Addressing Noisy Long-Tailed Problems in Malicious Traffic Detection
Li, G., Zhang, R., Sun, Z., Xing, L., & Zhang, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889365 · Cited by 10
“However, most malicious traffic datasets contain noisy-labeled and long-tailed problems simultaneously [7], [8], called Noisy Long-Tailed (NLT) problem [9], which leads to poor generalization of DL [7], [10].”
Revisiting Machine Unlearning with Dimensional Alignment
Seo, S., Kim, D.-W., & Han, B. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00317 · Cited by 15
“Despite their success, these machine learning models are prone to unintentionally memorize training data [1], making them vulnerable to various inference attacks that may compromise user privacy and reveal sensitive details in the training data.”
Toward Understanding the Generalizability of Delayed Stochastic Gradient Descent
Deng, X., Shen, L., Li, S., Sun, T., Li, D., & Tao, D. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3572251 · Cited by 6
A Comparative Review of the Next-Generation YOLO Models: YOLOv10 and YOLO11
Chen, F., Zhang, Y., Fu, L., Hua, R., Zhang, Q., & Bi, S. (2025) · Journal of Computer Science and Artificial Intelligence · doi:10.54097/22zsmc10 · Cited by 6
“In horizontal comparisons, YOLO11 not only maintains better accuracy and stability for medium and large object detection than YOLOv10, but also demonstrates improved generalization [20] capabilities in complex backgrounds and dense small-object scenarios.”
Granular-Ball Representation Learning for Deep CNN on Learning with Label Noise
Dai, D., Zhu, H., Xia, S., & Wang, G. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6585-3_4 · Cited by 9
Human Activity Recognition with Noise-Injected Time-Distributed AlexNet
Dutta, S. J., Boongoen, T., & Reyer Zwiggelaar, A. (2025) · Biomimetics · doi:10.3390/biomimetics10090613 · Cited by 6
OUI Need to Talk About Weight Decay: A New Perspective on Overfitting Detection
Fernández-Hernández, A., Mestre, J. I., Dolz, M. F., Duato, J. F., & Quintana–Ort́ı, E. S. (2025) · International Conference on Advanced Machine Learning and Data Science (AMLDS) · doi:10.1109/amlds63918.2025.11159348 · Cited by 6
“Indeed, understanding and enhancing generalization has become a central focus of contemporary research, as highlighted by works such as [6] and [7].”
Zhang, Y., Lu, Z., Yang, Y., Wang, X., Meng, Y., Yue, J., Wang, J., Wang, D., Sun, Y., Li, W., et al. (2025) · Materials Science in Semiconductor Processing · doi:10.1016/j.mssp.2025.110291 · Cited by 6
Information Bottleneck Theory Based Exploration of Cascade Learning
Du, X., Farrahi, K., & Niranjan, M. (2021) · Entropy · doi:10.3390/e23101360 · Cited by 6
Robust Product Classification with Instance-Dependent Noise
Nguyen, H. A., & Khatwani, D. (2022) · Workshop on e-Commerce and NLP (ECNLP) · doi:10.18653/v1/2022.ecnlp-1.20 · Cited by 6
“learning capacity make them highly susceptible to over-fitting to noise (Arpit et al., 2017; Zhang et al., 2021a).”
BackgroundMethodInfluentialEvent-set differential privacy for fine-grained data privacy protection
Wang, T., Yang, W., Ma, X., & Wang, B. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.10.006 · Cited by 6
Covert Task Embedding: Turning a DNN Into an Insider Agent Leaking Out Private Information
Li, L., Zhang, W., & Barni, M. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3216010 · Cited by 8
On the Importance of Domain Awareness in Classifier Interpretations in Medical Imaging
Major, D., Lenis, D., Wimmer, M., Berg, A., Neubauer, T., & Bühler, K. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3247659 · Cited by 9
“Furthermore, as quantitative measures about an ANN’s a-priori capability to generalize remain an open research topic [74], switching to prominent, less debatable reference methods like [50], [75] remains problematic for the medical domain.”
BackgroundThe deep neural network approach to the reference class problem
Buchholz, O. (2023) · Synthese · doi:10.1007/s11229-023-04110-9 · Cited by 8
“Many insights about the generalization ability of DNNs rely on empirical studies conducted with specific network architectures, but there is theoretical progress for some aspects of the problem (Zhang et al. 2021).”
BackgroundWeakly Supervised Medical Image Segmentation with Soft Labels and Noise Robust Loss
Felfeliyan, B., Hareendranathan, A. R., Kuntze, G., Wichuk, S. A., Forkert, N. D., Jaremko, J. L., & Ronsky, J. L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-37742-6_47 · Cited by 7
Zhang, X., Li, X., Chen, G., Liao, P., Wang, T., Yang, H., He, C., Zhou, W., & Sun, Y. (2023) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2023.3312591 · Cited by 6
“Conversely, fine-tuning large models on small datasets may result in the memorization of data distribution rather than adaptation to underlying patterns in the target domain, due to the imbalance between model complexity and limited data [10].”
BackgroundWang, X., Guo, B., & Shen, Y. F. (2023) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2023.029552 · Cited by 5
Data Provenance via Differential Auditing
Mu, X., Pang, M., & Zhu, F. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3334821 · Cited by 7
Overhead-free Noise-tolerant Federated Learning: A New Baseline
Lin, S., Zhai, D., Zhang, F., Jiang, J., Liu, X., & Ji, X. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1449-1 · Cited by 7
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.