Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 701–800 of 2,667
Teke, M., & Etem, T. (2025) · The European Physical Journal Special Topics · doi:10.1140/epjs/s11734-025-01842-2 · Cited by 13
“Models incorporating synthetic data and deep learning networks may also struggle with generalization [26].”
Sun, J., Liu, J., Hu, L., Wei, Z., & Xiao, L. (2022) · Remote Sensing · doi:10.3390/rs14041000 · Cited by 13
Label noise and self-learning label correction in cardiac abnormalities classification
Vázquez, C. G., Breuss, A., Gnarra, O., Portmann, J., Madaffari, A., & Poian, G. D. (2022) · Physiological Measurement · doi:10.1088/1361-6579/ac89cb · Cited by 15
Noise Models in Classification: Unified Nomenclature, Extended Taxonomy and Pragmatic Categorization
Sáez, J. A. (2022) · Mathematics · doi:10.3390/math10203736 · Cited by 15
Co-Learning Meets Stitch-Up for Noisy Multi-Label Visual Recognition
Liang, C., Yang, Z., Zhu, L., & Yang, Y. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3270103 · Cited by 15
“Deep neural networks are prone to fit noisy labels [45].”
BackgroundLearning for Online Mixed-Integer Model Predictive Control With Parametric Optimality Certificates
Russo, L., Nair, S. H., Glielmo, L., & Borrelli, F. (2023) · IEEE Control Systems Letters · doi:10.1109/lcsys.2023.3285778 · Cited by 13
“Indeed, prediction models may perform poorly for various reasons: insufficient richness of the model parameterization, distribution shift between the training and test sets, convergence of the training algorithm towards a sub-optimal minimum [14].”
BackgroundAttribute-Based Membership Inference Attacks and Defenses on GANs
Sun, H., Zhu, T., Li, J., Ji, S., & Zhou, W. (2023) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2023.3305591 · Cited by 16
“Additionally,theadversaryalso targetsthediscriminatorbecausethediscriminativemodelstend tooverfitaccordingtothepreviousworks[15],[22],[23],[24], [25].”
Enabling Binary Neural Network Training on the Edge
Wang, E., Davis, J. J., Moro, D., Zieliński, P., Lim, J. J., Coelho, C., Chatterjee, S., Cheung, P. Y. K., & Constantinides, G. A. (2023) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3626100 · Cited by 16
“, a network possess perfect expressivity once its number of parameters matches the number of data points used for its training [50].”
BackgroundOn the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensions
Beaglehole, D., Belkin, M., & Pandit, P. (2023) · SIAM Journal on Mathematics of Data Science · doi:10.1137/22m1499819 · Cited by 31
Park, J., Kim, T., Gu, C., Kang, Y., & Cheong, J. (2023) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2023.102692 · Cited by 13
Late Stopping: Avoiding Confidently Learning from Mislabeled Examples
Yuan, S., Feng, L., & Liu, T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01473 · Cited by 28
“Unfortunately, previous studies [1, 56] showed that DNNs can easily overfit random labels, resulting in poor generalization performance.”
BackgroundMethodImproving generalization of deep neural networks by leveraging margin distribution
Lyu, S.-H., Wang, L., & Zhou, Z. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.03.019 · Cited by 19
AI and Law: The Next GenerationPreprint
Lee, K., Cooper, A. F., Grimmelmann, J., & Ippolito, D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4580739 · Cited by 19
Position: Why We Must Rethink Empirical Research in Machine LearningPreprint
Herrmann, M., Lange, F. J. D., Eggensperger, K., Casalicchio, G., Wever, M., Feurer, M., Rügamer, D., Hüllermeier, E., Boulesteix, A.-L., & Bischl, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.02200 · Cited by 41
A Falsificationist Account of Artificial Neural Networks
Buchholz, O., & Raidl, É. (2022) · The British Journal for the Philosophy of Science · doi:10.1086/721797 · Cited by 17
“A general mathematical characterization of the phenomenon remains challenging, which is why many insights about the generalization ability of ANNs still rely on empirical studies (Zhang et al. 2021).”
Guo, Y., Guo, J., Li, Y., Zhang, P., Zhao, Y., Qiao, Y., Liu, B., & Wang, G. (2024) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-023-03047-1 · Cited by 13
“…the DL-based methods strengthen features relevant to diagnosis while suppressing those irrelevant features resulting from varying imaging conditions [14], effectively minimizing the negative impact of inconsistent imaging conditions on the diagnostic accuracy of panoramic dental radiographs [15].”
A Time-Consistency Curriculum for Learning From Instance-Dependent Noisy Labels
Wu, S., Zhou, T., Du, Y., Yu, J., Han, B., & Liu, T. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3360623 · Cited by 13
Improving Neural Network Generalization on Data-Limited Regression with Doubly-Robust Boosting
Wang, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i18.30071 · Cited by 16
“Consequently, the reduction of E out cannot be reliably achieved using these regularization methods alone (Zhang et al. 2021).”
BackgroundMethodPatchMix: patch-level mixup for data augmentation in convolutional neural networks
Hong, Y., & Chen, Y. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02141-3 · Cited by 12
Gan, T., Liu, X., Liu, R., Huang, J., Liu, D., Tu, W., Song, J., Cai, P., Shen, H., & Wang, W. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1354925 · Cited by 14
“Therefore, the generalization ability of these models in various populations and medical institutions is limited (25, 26).”
BackgroundAkhtom, D., Singh, M. M., & Chew, X. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10777-4 · Cited by 16
OR 2 Net: Online Re-weighting Relation Network for kinship verification
Li, H., Zhao, X. J., Wang, M., Song, H., & Sun, F. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124815 · Cited by 15
MSFN: a multi-omics stacked fusion network for breast cancer survival prediction
Zhang, G., Ma, C., Yan, C., Luo, H., Wang, J., Liang, W., & Luo, J. (2024) · Frontiers in Genetics · doi:10.3389/fgene.2024.1378809 · Cited by 14
Yang, Z., Yang, X., Tang, H., Shi, H., & Liu, Z. (2024) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2024.176423 · Cited by 12
Lee, Z., Wang, T., Zhao, L., Wang, D., Ye, X., Shang, S., & Yu, X. (2024) · Journal of Remote Sensing · doi:10.34133/remotesensing.0302 · Cited by 12
Self-supervised contrastive learning for implicit collaborative filtering
Song, S., 刘, 斌., Teng, F., & Li, T. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109563 · Cited by 14
Online learning discriminative sparse convolution networks for robust UAV object tracking
Qi, X., Xu, Z., Wang, H., Chen, Y., & Tao, L. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112742 · Cited by 13
S-JEPA: A Joint Embedding Predictive Architecture for Skeletal Action Recognition
Abdelfattah, M., & Alahi, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73411-3_21 · Cited by 34
A systematic review of deep learning for structural geological interpretation
Fernandes, G. L., Figueiredo, F., Hatushika, R. S., Leão, M. L., Mariano, B. A., Monteiro, B. A. A., de Oliveira, F. F., Panoutsos, T., Pires, J. P., Poppe, T. M., et al. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-024-01079-y · Cited by 15
Adamu, S., Iorliam, A., & Asilkan, Ö. (2025) · Journal of Future Artificial Intelligence and Technologies · doi:10.62411/faith.3048-3719-58 · Cited by 12
“Thus yielding competitive performance with far less complexity and equivalence to weighted techniques in ensembles of decision trees [47].”
Dual-Domain deep prior guided sparse-view CT reconstruction with multi-scale fusion attention
Wu, J., Lin, J., Jiang, X., Zheng, W., Zhong, L., Pang, Y., Meng, H., & Li, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-02133-5 · Cited by 12
“Second, these methods exhibit susceptibility to overfitting, demonstrated by substantial performance gaps between training and patient-separated test datasets 34 .”
Meta-analysis of H. pylori and the gut microbiome interactions and clinical outcomes
Wu, X., Zhu, H., Hu, Y., Zhang, L., & Huang, L. (2025) · Frontiers in Cellular and Infection Microbiology · doi:10.3389/fcimb.2025.1610523 · Cited by 13
“Detecting consistent microbiological fi ngerprints over different populations is complicated, because of the wide range of clinical presentations linked to H. pylori -related diseases (Zhang et al., 2021).”
InfluentialData Can Speak for Itself: Quality-guided Utilization of Wireless Synthetic Data
Gong, C., Bo, L., Gao, W., & Xu, C. (2025) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3711875.3729123 · Cited by 17
“The mislabeled data can distort training process and degrade performance [30, 31], and we observe performance degradation when trained with synthetic data deficient in both affinity and diversity in §7.4.”
InfluentialFuture Factories With 6G: Agentic AI and Cyber–Physical Digital Twins
Li, H., Madhukumar, H., Methley, N., Chen, X., Wu, Y., Parra-Ullauri, J. M., Sharma, V., Lee, J., Koblitz, A. R., Andrews, M., et al. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3623075 · Cited by 19
“Directly modeling these complex, nonlinear relationships with sophisticated methods (such as deep neural networks) typically results in poor generalization to new scenarios [28], [29], [30].”
Ireddy, R. K. (2024) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit24113395 · Cited by 12
Golilarz, N. A., Khatib, H. S. A., & Rahimi, S. (2026) · IEEE Access · doi:10.1109/access.2026.3689754 · Cited by 16
Neural Joint Entropy Estimation
Shalev, Y., Painsky, A., & Ben‐Gal, I. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3204919 · Cited by 20
Penalty based robust learning with noisy labels
Kong, K., Lee, J., Kwak, Y., Cho, Y., Kim, S., & Song, W. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.02.030 · Cited by 12
Arief, H. A., Thomas, P. J., & Wiktorski, T. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3154783 · Cited by 12
Integral representations of shallow neural network with rectified power unit activation function
Abdeljawad, A., & Grohs, P. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.09.005 · Cited by 15
Using model explanations to guide deep learning models towards consistent explanations for EHR data
Watson, M., Hasan, B. A. S., & Moubayed, N. A. (2022) · Scientific Reports · doi:10.1038/s41598-022-24356-6 · Cited by 14
“There’s also a significant lack of understanding around why and how models generalise 16 , especially with recent work showing that over-parameterised networks are absent of the classic U-shaped test error curve 17,18 .”
BackgroundHe, Z., Chen, P., & Huang, T. (2022) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre55969.2022.00014 · Cited by 16
Lim, P. K., Julca, I., & Mutwil, M. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.01.013 · Cited by 13
“Besides a large dataset size, credible labeling of the training data is also paramount as mislabeled data will ultimately degrade the model’s performance [30,31].”
Combining layered label correction and mixup supervised contrastive learning to learn noisy labels
Huang, B., Zhang, P., & Xu, C. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119242 · Cited by 14
Few-Shot Object Detection With Self-Supervising and Cooperative Classifier
Qi, D., Hu, J., & Shen, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3204597 · Cited by 15
PanorAMS: Automatic Annotation for Detecting Objects in Urban Context
Groenen, I., Rudinac, S., & Worring, M. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3279696 · Cited by 13
MetaASSIST: Robust Dialogue State Tracking with Meta Learning
Ye, F., Wang, X., Huang, J., Li, S., Stern, S., & Yılmaz, E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.76 · Cited by 12
LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classification
Mekala, D., Dong, C., & Shang, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.360 · Cited by 26
Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning
Wu, H., Lei, C., Sun, X., Wang, P., Chen, Q., Cheng, K.-T. T., Lin, S. C.-F., & Wu, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01494 · Cited by 22
“For supervised learning, data augmentation aids in reducing overfitting and regularization [84].”
Macaluso, A. (2024) · Künstliche Intell. · doi:10.1007/s13218-024-00856-7 · Cited by 16
“This empirically driven methodology presents a challenge to the traditional reliance on theoretical constructs such as VC dimension [29] or effective dimen-sion [30] to gauge a model’s generalization capacity [31].”
Ott, F., Heublein, L., Rügamer, D., Bischl, B., & Mutschler, C. (2024) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2024.104256 · Cited by 14
“Pre-training neural networks using simulated data can enhance their performance in various tasks, especially in the case of data scarcity, domain adaptation, data augmentation, and providing a good initialization for transfer learning to real data (Zhang et al., 2017; Shrivastava et al., 2017).”
BackgroundMulti-sentence Grounding for Long-Term Instructional Video
Li, Z., Chen, Q., Han, T., Zhang, Y., Wang, Y., & Xie, W. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72992-8_12 · Cited by 24
“…on noisy pseudo-labels ( i.e ., relying on weakly-aligned transcripts, and getting steps-transcript similarity from imperfect pre-trained text encoder), the model tends to learn the alignment patterns from procedural step to videos before overfitting to the noises, as also being observed in [44].”
BackgroundLi, P., He, X., Qiao, M., Cheng, X., Li, J., Guo, X., Zhou, T., Song, D., Chen, M., Miao, D., et al. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3128539 · Cited by 14
Effect of Deep Learning on College Students’ Career Planning
Gu, X. (2022) · Mathematical Problems in Engineering · doi:10.1155/2022/1573635 · Cited by 12
Aggregating Frame-Level Information in the Spectral Domain With Self-Attention for Speaker Embedding
Tu, Y., & Mak, M. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3153267 · Cited by 14
Towards a general and numerically efficient deposition model for wire-arc directed energy deposition
Glasder, M., Fabbri, M., Aschwanden, I., Bambach, M., & Wegener, K. (2023) · Additive manufacturing · doi:10.1016/j.addma.2023.103832 · Cited by 12
Delta Hedging of Derivatives using Deep Reinforcement LearningPreprint
Giurca, A., & Borovkova, S. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3847272 · Cited by 21
Ayana, H., Mereba, T., & Alemu, A. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1399350 · Cited by 11
Supervision Adaptation Balancing In-Distribution Generalization and Out-of-Distribution Detection
Zhao, Z., Cao, L., & Lin, K.-Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3321869 · Cited by 12
Su, H., Zhu, T., Lv, J.-Q., Wang, H., Zhao, J., & Xu, J. (2024) · Bioresource Technology · doi:10.1016/j.biortech.2024.130536 · Cited by 11
MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition
Schmitt, F., Huang, E. M., Hutter, H.-P., Stadelmann, T., & Darvishy, A. (2024) · IEEE Access · doi:10.1109/access.2024.3404834 · Cited by 15
“This leads to detrimental variability in the training data and therefore to inefficient learning, excessive training data needs, and, finally, suboptimal recognition performance due to unresolved ambiguity in the model [6], [7].”
BackgroundEffective and Robust Adversarial Training Against Data and Label Corruptions
Zhang, P.-F., Huang, Z. H., Xu, X.-S., & Bai, G. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3394677 · Cited by 14
“DNNs have considerable power to even memorize label noise patterns during the training process, resulting in inferior performance [16], [17], [18].”
BackgroundQu, J., Yousef, S., Faney, T., de Hemptinne, J.-C., & Gallinari, P. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.124025 · Cited by 13
Instance-Dependent Multilabel Noise Generation for Multilabel Remote Sensing Image Classification
Kim, Y., Kim, S., Ro, Y., & Lee, J. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3454157 · Cited by 14
“Since deep learning networks can overfit to incorrect (i.e., noisy) labels [19], [20], there is a significant risk that the generalization performance of the trained networks could be severely degraded.”
BackgroundEstimating Per-Class Statistics for Label Noise Learning
Luo, W., Chen, S., Liu, T., Han, B., Niu, G., Sugiyama, M., Tao, D., & Gong, C. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3466182 · Cited by 15
“It has also been shown that deep neural networks can easily be misled by such noisily labeled examples, leading to dramatic performance degradation [54].”
Generalization of neural network models for complex network dynamics
Vasiliauskaitė, V., & Antulov-Fantulin, N. (2024) · Communications Physics · doi:10.1038/s42005-024-01837-w · Cited by 13
Feasibility of Machine Learning-Enhanced Detection for QR Code Images in Email-based Threats
Ford, J., & Berry, H. S. (2024) · Cyber Awareness and Research Symposium (CARS) · doi:10.1109/cars61786.2024.10778732 · Cited by 11
“Zhang et al. [10] emphasize the importance of re-thinking generalization in deep learning, particularly the challenges of training models for diverse tasks.”
Marion, T. J., Yuan, C., & Moghaddam, M. (2025) · Research-Technology Management · doi:10.1080/08956308.2024.2444142 · Cited by 15
“We follow calls for further research into how AI can be applied to innovation systems (Verganti, Vendraminelli, and Iansiti 2020; Zhang et al. 2021) and contribute to the literature by highlighting the opportunities and challenges of integrating AI into the front end of the innovation process.”
CromSS: Cross-Modal Pretraining With Noisy Labels for Remote Sensing Image Segmentation
Liu, C., Albrecht, C. M., Wang, Y., & Zhu, X. X. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3552642 · Cited by 18
“Recent studies have demonstrated the resilience of deep learning models to a certain degree of label noise [23], [24], showing that the deep learning models trained on large-scale noisy labels can exhibit strong feature learning capabilities and transfer learning performance across various…”
Alhazbi, S., Hussain, A., Oligeri, G., & Papadimitratos, P. (2025) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2025.3577016 · Cited by 19
“This suggests that the neural network model struggles with generalization, as it overfits (memorizes) noisy training data but fails to perform well on unseen samples even though they are from the same distribution [39].”
A Guide to Machine Learning Epistemic Ignorance, Hidden Paradoxes, and Other Tensions
Naser, M. Z. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70038 · Cited by 17
“For example, Zhang et al. (2017, 2021) challenge conventional wisdom by demonstrating that deep networks can easily memo-rize random labels yet still generalize effectively on natural data.”
Towards Natural Machine Unlearning
He, Z., Li, T., Cheng, X., Huang, Z., & Huang, X. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3597350 · Cited by 11
LLM agents security duality: a comprehensive survey of self-security and empowered cybersecurity
Xu, Y., Zhuang, Y., Liu, X., Zhang, T. B., Xiao, B., Xu, X., Jiang, D., Wang, J., & Hu, H. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11563-0 · Cited by 19
Santikellur, P., & Chakraborty, R. S. (2021) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2021.3129112 · Cited by 11
Deep Learning in Network-Level Performance Prediction Using Cross-Layer Information
Cao, Q., Pun, M.-O., & Yi, C. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3163274 · Cited by 12
Research on performance predictions using single-hole film cooling based on PointNet
Zhi, R., Li, Z., Wen, F., Su, L., & Wang, S. (2023) · Physics of Fluids · doi:10.1063/5.0136442 · Cited by 11
Differentially Private Federated Continual Learning with Heterogeneous Cohort Privacy
Chathoth, A. K., Necciai, C. P., Jagannatha, A., & Lee, S. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10021082 · Cited by 24
Self-Training With Double Selectors for Low-Resource Named Entity Recognition
Fu, Y., Lin, N., Yu, X., & Jiang, S. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3250828 · Cited by 11
“However, one of the major challenges for self-training is error propagation [14] where wrong pseudo labels would impair the model performance.”
BackgroundA fundus image classification framework for learning with noisy labels
Hu, T., Yang, B., Guo, J., Zhang, W., Liu, H., Wang, N., & Li, H. (2023) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2023.102278 · Cited by 13
Schmid, J., Assassi, L., & Chênes, C. (2023) · European Radiology Experimental · doi:10.1186/s41747-023-00357-6 · Cited by 11
“Data augmentation is often seen as a type of regularization to improve the generalization of deep learning models [5] by avoiding overfitting and countering data imbalance [3].”
BackgroundHow Do Label Errors Affect Thin Crack Detection by DNNs
Xu, L., Zou, H., & Okatani, T. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00464 · Cited by 15
Boundary between noise and information applied to filtering neural network weight matrices
Staats, M., Thamm, M., & Rosenow, B. (2023) · Physical review. E · doi:10.1103/physreve.108.l022302 · Cited by 17
“…three hidden layers, denoted as MLP1024, with layer sizes [in, 1024, 512, 512, out], and (ii) modern convolutional neural networks (CNNs), called miniAlexNet [25], consisting of two convolutional layers followed by max-pooling, batch normalization, and fully connected layers with regularization.”
Investigating the Sample Weighting Mechanism Using an Interpretable Weighting Framework
Zhou, X., Wu, O., & Li, M. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3316168 · Cited by 17
ComCo: Complementary supervised contrastive learning for complementary label learning
Jiang, H., Sun, Z., & Tian, Y. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.10.013 · Cited by 14
AI-Assisted Cotton Grading: Active and Semi-Supervised Learning to Reduce the Image-Labelling Burden
Fisher, O. J., Rady, A. M., El-Banna, A. A. A., Emaish, H. H., & Watson, N. J. (2023) · Sensors · doi:10.3390/s23218671 · Cited by 13
“This can lead to a plateau in accuracy as the model is not gaining any new insights from the additional data.”
BackgroundAllure of Simplicity: On Interpretable Machine Learning Models in Healthcare
Grote, T. (2023) · Philosophy of Medicine · doi:10.5195/pom.2023.139 · Cited by 18
Melki, P., Bombrun, L., Diallo, B., Dias, J., & da Costa, J. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00068 · Cited by 19
“…proven in the lab and in the field, some important issues such as inter-pretability [31, 21, 7], generalization to new observations and domains [28, 29, 27], robustness to noise and out-of-distribution observations [15, 17, 4], and uncertainty quan-tification [1, 10, 11] are yet to be solved or…”
BackgroundA deep semantics-aware data augmentation method for fault localization
Hu, J., & Lei, Y. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107409 · Cited by 11
Yan, P., Teng, Q., Ma, Z., Wu, X., & He, X. (2024) · Computers and Geotechnics · doi:10.1016/j.compgeo.2024.106145 · Cited by 11
Kar, A., Dhara, S. K., Sen, D., & Biswas, P. K. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3370779 · Cited by 17
“Deep learning based methods are prone to fail in generalizing on datasets that are out of training distribution [63].”
Data-Driven Multi-Dimension Driving Safety Evaluation for Real-World Electric Vehicles
Hong, J., Liang, F., Zhang, H., Chen, Y., Li, R., Li, K., & Yang, J. (2024) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2024.3379677 · Cited by 13
Hassan, M. F., Lang, W. S., & Paramesran, R. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19245-1 · Cited by 12
“Additionally, they may suffer from issues such as local over-enhancement caused by uneven illumination during image acquisition [38, 39] and a failure to accurately represent the image depth information, particularly in edge areas [40].”
ECLB: Efficient contrastive learning on bi-level for noisy labels
Guan, J., Liu, J., Huang, S., & Yang, Y. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112128 · Cited by 12
Microfluidics with Machine Learning for Biophysical Characterization of Cells
Jeon, H., & Han, J. (2025) · Annual Review of Analytical Chemistry · doi:10.1146/annurev-anchem-061622-025021 · Cited by 14
On the Convergence of (Stochastic) Gradient Descent for Kolmogorov–Arnold Networks
Gao, Y., & Tan, V. Y. F. (2025) · IEEE Transactions on Information Theory · doi:10.1109/tit.2025.3588401 · Cited by 17
“While much of the literature focuses on convergence to local optima, numerical experiments have shown that both GD and SGD nearly achieves global optima in practice in which the mean-square error approaches zero [38].”
Learning From Noisy Labels via Dynamic Loss Thresholding
Yang, H., Jin, Y.-Z., Li, Z., Wang, D.-B., Geng, X., & Zhang, M.-L. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3313604 · Cited by 18
Multi-Head Relu Implicit Neural Representation Networks
Aftab, A., Morsali, A., & Ghaemmaghami, S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747352 · Cited by 18
Sato, M., Tateishi, R., Moriyama, M., Fukumoto, T., Yamada, T., Nakagomi, R., Kinoshita, M. N., Nakatsuka, T., Minami, T., Uchino, K., et al. (2022) · Gastro Hep Advances · doi:10.1016/j.gastha.2021.09.003 · Cited by 11
“45 This may be dif fi cult to achieve in clinical settings for ethical or methodological reasons.”
BackgroundCorrecting Imprecise Object Locations for Training Object Detectors in Remote Sensing Applications
Bernhard, M., & Schubert, M. (2021) · Remote Sensing · doi:10.3390/rs13244962 · Cited by 15
Noise-Robust Learning from Multiple Unsupervised Sources of Inferred Labels
Silva, A., Luo, L., Karunasekera, S. A., & Leckie, C. A. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20806 · Cited by 14
“As found by recent studies (Zhang et al. 2021; Reed et al. 2015), DNNs can overfit the noise in such noisy labels, which substantially degrades their performance.”
Background
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.