Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 1,101–1,200 of 2,667
Intriguing Equivalence Structures of the Embedding Space of Vision Transformers
Salman, S., Shams, M. M. B., Islam, C. M., Nishino, M., & Liu, X. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729551 · Cited by 8
“For example, Zhang et al. reveal that deep networks can memorize training sets yet still generalize [10], highlighting gaps in our understanding of how embeddings capture semantic relationships.”
An Outlook for AI Innovation in Multimodal Communication Research
Henlein, A., Bauer, A., Bhattacharjee, R., Ćwiek, A., Gregori, A., Kügler, F., Lemanski, J., Lücking, A., Mehler, A., Prieto, P., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61066-0_13 · Cited by 11
Learning with noisy labels for robust fatigue detection
Wang, M., Hu, R., Zhu, X., Zhu, D., & Wang, X. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112199 · Cited by 8
Multi-stage ensemble with refinement for noisy labeled data classification
Choi, C., Lee, W., & Son, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124672 · Cited by 6
Revisiting Deep Ensemble for Out-of-Distribution Detection: A Loss Landscape Perspective
Fang, K., Tao, Q., Huang, X., & Yang, J. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02156-x · Cited by 12
“Given a DNN f well-trained on data from some distribution P in , f also performs well on the test data, which is commonly regarded as being from the same distribution P in as the training data [2].”
MethodNon‐linear phylogenetic regression using regularised kernels
Rosas‐Puchuri, U., Santaquiteria, A., Khanmohammadi, S., Solís‐Lemus, C., & Betancur‐R, R. (2024) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.14385 · Cited by 6
“The use of a regularisation term in the error function enhances the model's generalisability and robustness to outliers (Jiang et al., 2022; Tibshirani, 1996; Zhang et al., 2022), provided it is properly tuned (Srivastava et al., 2014; Zhang et al., 2021).”
A class sensitivity feature guided T-type generative model for noisy label classification
Bai, Y., & Cui, H. (2024) · Machine Learning · doi:10.1007/s10994-024-06598-9 · Cited by 7
“Meanwhile, the existence of noisy label in real-world data is inevitable (Mahajan et al., 2018; Kuznetsova et al., 2020) and has long been an intractable problem, since DNNs can be easily affected and overfitted to noisy label dataset (Zhang et al., 2021a).”
Non-iterative disentangled unitary coupled-cluster based on lie-algebraic structure
Haidar, M., Adjoua, O., Badreddine, S., Peruzzo, A., & Piquemal, J. (2025) · Quantum Science and Technology · doi:10.1088/2058-9565/adb3c5 · Cited by 12
“Overparameterizing a neural network can enhance QNN performance, reduce training and generalization errors [70, 71], and even result in provable convergence [72, 73].”
Test-Time Poisoning Attacks Against Test-Time Adaptation Models
Cong, T., He, X., Shen, Y., & Zhang, Y. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00072 · Cited by 13
Instance-Dependent Noisy-Label Learning with Graphical Model Based Noise-Rate Estimation
Garg, A. K., Nguyen, C., Felix, R., Do, T.-T., & Carneiro, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73235-5_21 · Cited by 13
“This is a relevant issue because even slight inaccuracies in the labels can significantly affect the performance of DNN due to their inherent memorisation capabilities [34,53].”
BackgroundONDS: Optimum Node and Data Selection From Constrained IoT for Efficient Online Learning
Abdellatif, A. A., Shaban, K. B., & Massoud, A. M. (2024) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2024.3483295 · Cited by 8
Lu, X., Qiao, C., Wang, H., Li, Y., Wang, J., Wang, C., Wang, Y., & Qie, S. (2024) · Sensors · doi:10.3390/s24227258 · Cited by 6
“For example, the literature points out that deep learning models often exhibit instability when processing small-sample data, requiring regularization or data augmentation techniques for improvement [53].”
Federated semi-supervised learning based on truncated Gaussian aggregation
Zhu, S., Wang, Y., & Sun, G. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06798-z · Cited by 6
“…learning rate of Inter-client consistency loss based on truncated Gaussian: Since the model tends to forget past knowledge and overfit the most recent training data [25], averaging the parameters in each round provides more reliable parameters compared to using the latest local parameters directly.”
InfluentialLeveraging interpolation models and error bounds for verifiable scientific machine learning
Chang, T. H., Gillette, A., & Maulik, R. (2025) · Journal of Computational Physics · doi:10.1016/j.jcp.2025.113726 · Cited by 6
“…models that interpolate or nearly-interpolate training data often achieve state-of-the-art performance on standard benchmark problems [23], and interpolation techniques perform similarly to many of the machine learning techniques discussed above in high-dimensional regimes, so long…”
Misclassification bounds for PAC-Bayesian sparse deep learning
Mai, T. (2025) · Machine Learning · doi:10.1007/s10994-024-06690-0 · Cited by 6
“The statistical learning community has extensively investigated the generalization properties of Deep Neural Networks (DNNs), (Barron, 1994; Bartlett et al., 2017; Neyshabur et al., 2018; Zhang et al., 2021; Schmidt-Hieber, 2020; Suzuki, 2018; Imaizumi & Fukumizu, 2019; Suzuki, 2019).”
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0005 · Cited by 6
Machine Learning Techniques Improving the Box–Cox Transformation in Breast Cancer Prediction
Alshamrani, S. S. (2025) · Electronics · doi:10.3390/electronics14163173 · Cited by 9
Normalized Augmented Inverse Probability Weighting with Neural Network Predictions
Rostami, M., & Saarela, O. (2022) · Entropy · doi:10.3390/e24020179 · Cited by 6
“[1] argue that the fact that SGD-type algorithms control the complexity of the NN algorithm to some extent [2,10] is sufficient for the first step.”
BackgroundRegularizing Scale-Adaptive Central Moment Sharpness for Neural Networks
Chen, J., Guo, Z., Li, H., & Chen, C. L. P. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3210045 · Cited by 6
Jaiswal, A. K., Chen, T., Rousseau, J. F., Peng, Y., Ding, Y., & Wang, Z. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00496 · Cited by 10
“Many works [43, 1, 29, 42, 10, 41] have recently observed that DNNs tend to prioritize learning simple patterns.”
BackgroundIntra-Class Mixup for Out-of-Distribution Detection
Ravikumar, D., Kodge, S., Garg, I., & Roy, K. (2023) · IEEE Access · doi:10.1109/access.2023.3254920 · Cited by 9
Morabbi, S., Soltanizadeh, H., Mozaffari, S., & Fadaeieslam, M. J. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05448-0 · Cited by 6
LPR: learning point-level temporal action localization through re-training
Fang, Z., Jianping Fan, B., & Yu, J. (2023) · Multimedia Systems · doi:10.1007/s00530-023-01128-4 · Cited by 8
“However, the pseudo-labels contain a large amount of label noise, and as the training progresses, the model tends to over-fitting the noise data [18].”
BackgroundBiomedical Data Annotation: An OCT Imaging Case Study
Anderson, M., Sadiq, S. N., Solim, M. A. N., Barker, H., STEEL, D. H., Habib, M., & Obara, B. (2023) · Journal of Ophthalmology · doi:10.1155/2023/5747010 · Cited by 6
“As deep neural networks are vulnerable to overftting, accepting annotations as perfectly correct and reliable may result in biased models with poor generalization [12].”
BackgroundDomain Adaptation Using Class-Balanced Self-Paced Learning for Soil Classification With LIBS
Huang, Y., Bais, A., & Hussein, A. (2023) · IEEE Transactions on Plasma Science · doi:10.1109/tps.2023.3305559 · Cited by 6
“It is demonstrated that deep neural networks (DNNs) easily fit false pseudolabels, which deteriorate the model’s test performance [48].”
BackgroundWang, Q., Zhang, Z., Gao, M., Xie, J., Zhu, P., Li, P., Zuo, W., & Hu, Q. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3321392 · Cited by 10
Input Margins Can Predict Generalization Too
Mouton, C., Theunissen, M. W., & Davel, M. H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i13.29351 · Cited by 6
Robust returns ranking prediction and portfolio optimization for M6
Ai, H., Liu, C. H. B., & Lin, P. (2024) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2024.04.004 · Cited by 6
Do Highly Over-Parameterized Neural Networks Generalize Since Bad Solutions are Rare?
Martinetz, J., & Martinetz, T. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3529297 · Cited by 6
Cluster-Based Classification with Neural ODEs via Control
Álvarez-López, A., Orive, R., & Zuazua, E. (2025) · Journal of Machine Learning · doi:10.4208/jml.241114 · Cited by 7
Artificial Neural Nets and the Representation of Human Concepts
Freiesleben, T. (2025) · Synthese Library/Synthese library · doi:10.1007/978-3-032-03083-2_13 · Cited by 10
Balanced and Deterministic Weight-Sharing Helps Network Performance
Chang, O., & Lipson, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01424-7_5 · Cited by 6
Feature Mining: A Novel Training Strategy for Convolutional Neural Network
Xie, T., Deng, J., Cheng, X., Liu, M., Wang, X., & Liu, M. (2022) · Applied Sciences · doi:10.3390/app12073318 · Cited by 6
Realizable Learning is All You Need
Hopkins, M., Kane, D. M., Lovett, S., & Mahajan, G. (2024) · TheoretiCS · doi:10.46298/theoretics.24.2 · Cited by 11
“While uniform convergence was a very popular technique in the early years of learning, practitioners have since moved away from the paradigm which fails to capture learning rates seen in practice [44, 45].”
BackgroundLiu, C., Wang, Q., Si, W., & Ni, X. (2022) · Physics in Medicine and Biology · doi:10.1088/1361-6560/aca873 · Cited by 6
“It could be attributed to the implicit regularization of neural networks that drives the convergence to such smooth solutions (Zhang et al 2021), thoughwe have set a largem for the spatial positional encoding to allow spatial discontinuities.”
BackgroundLearning from Better Supervision: Self-distillation for Learning with Noisy Labels
Baek, K., Lee, S., & Shim, H. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956388 · Cited by 8
Combating with extremely noisy samples in weakly supervised slot filling for automatic diagnosis
Shi, X., & Che, W. (2023) · Frontiers of Computer Science · doi:10.1007/s11704-022-2134-1 · Cited by 6
“Essentially, noisy labels are corrupted from ground-truth labels, and thus they inevitably degenerate the robustness of learned models, especially for deep neural networks [21].”
BackgroundGraph Neural Network based Child Activity Recognition
Mohottala, S., Samarasinghe, P. D., Kasthurirathna, D., & Abhayaratne, C. (2022) · IEEE International Conference on Industrial Technology (ICIT) · doi:10.1109/icit48603.2022.10002799 · Cited by 6
“These results are consistent with the existing research done on over-parameterized neural networks and the resultant high generalization accuracy [55], [56].”
ResultCMW-Net: an adaptive robust algorithm for sample selection and label correction
Shu, J., Yuan, X., & Meng, D. (2023) · National Science Review · doi:10.1093/nsr/nwad084 · Cited by 6
Using Neural Networks to Guide Data-Driven Operational Decisions
Lagzi, S., Chen, N., & Milner, J. M. (2026) · Management Science · doi:10.1287/mnsc.2023.04141 · Cited by 8
Discussion and Author Responses
Engstrom, L., Ilyas, A., Mądry, A., Santurkar, S., Tran, B., & Tsipras, D. (2019) · Distill · doi:10.23915/distill.00019.7 · Cited by 6
Depth Without the Magic: Inductive Bias of Natural Gradient DescentPreprint
Kerekes, A., Mészáros, A., & Huszár, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.11542 · Cited by 7
“…fast convergence on a quadratic loss; Bernacchia et al. (2018) proved fast convergence for deep linear models under quadratic loss; more recently, Zhang et al. (2021) gave a proof of fast convergence which holds for a broad class of overparametrized networks and also extends to K-FAC; Rudner et…”
BackgroundA path-norm toolkit for modern networks: consequences, promises and challenges
Gonon, A., Brisebarre, N., Riccietti, E., & Gribonval, R. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.01225 · Cited by 11
“Networks that can fit random labels have already been found empirically (Zhang et al., 2021), and it is open whether this stays true with a constraint on the path-norm.”
Geometry of Decision Making in Language ModelsPreprint
Joshi, A., Bhatt, D., & Modi, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.20315 · Cited by 8
Li, A., Yang, B., Naganawa, M., Fontaine, K. M., Toyonaga, T., Carson, R. E., & Tang, J. (2023) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ad0535 · Cited by 5
A Statistical Model for Predicting Generalization in Few-Shot Classification
Bendou, Y., Gripon, V., Pasdeloup, B., Lioi, G., Mauch, L., Uhlich, S., Cardinaux, F., Hacene, G. B., & García, J. A. (2023) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco58844.2023.10289951 · Cited by 6
Chance-Constrained Abnormal Data Cleaning for Robust Classification With Noisy Labels
Shen, X., Luo, Z., Li, Y., Ouyang, T., & Wu, Y. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3375518 · Cited by 5
“Although deeplearningtechnologyhassignificantlyincreasedtheaccuracyof classification[10],deeplearningmodelscouldbemorevulnerableto overfittingthelabelnoiseinthetrainingset[11],[12],[13],adversely affectingtheclassificationaccuracy.”
Causal representation learning through higher-level information extraction
Silva, F., Oliveira, H. P., & Pereira, T. (2024) · ACM Computing Surveys · doi:10.1145/3696412 · Cited by 10
“…(very often surpassing the number of training samples by large ratios [ 62 ]) and the high test performances achieved, the interest in rethinking why previous definitions of generalization expectation did not seem to apply in these approaches has been renewed [ 4 , 15 , 21 , 122 , 186 , 274 ].”
Bearing fault diagnosis method based on improved meta-ResNet and sample weighting under noise label
Xie, S., Wang, J., Li, Y., & Yang, L. (2024) · Structural Health Monitoring · doi:10.1177/14759217241277243 · Cited by 7
“20,21 To address the issue of noisy labels, there are two main methods.”
Uncertainty Driven Adaptive Self-Knowledge Distillation for Medical Image Segmentation
Guo, X., Wang, M., Xiang, Y., Yang, Y., Ye, C., Wang, H., & Ma, T. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2025.3526259 · Cited by 8
“This can hinder the model’s ability to generalize to new data [8], [9].”
InfluentialPark, J.-G., Amangeldi, A., Fakhrutdinov, N., Karzhaubayeva, M., & Zorbas, D. (2025) · IEEE Access · doi:10.1109/access.2025.3536471 · Cited by 5
A Unified Open Adapter for Open-World Noisy Label Learning: Data-Centric and Learning-Based Insights
Zong, C.-C., Yang, P., Xie, M.-K., & Huang, S.-J. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3550899 · Cited by 6
“Regrettably, labels obtained through these means inevitably harbor numerous errors, and direct learning from them can markedly undermine the generalization performance of DNNs [14], [15].”
Federated semi-supervised learning with contrastive representations against noisy labels
Mao, W., Yu, B., Lv, Y., Xie, Y., & Zhang, C. B. C. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113421 · Cited by 6
Self-citationStackelberg Game-Based Hierarchical Incentive Mechanism for Clustered Vehicular Federated Learning
Wang, S., Xia, W., Zhao, H., Wei, K., Quek, T. Q. S., & Zhu, H. (2025) · IEEE Transactions on Communications · doi:10.1109/tcomm.2025.3582713 · Cited by 5
“Meanwhile, we think the model with lower accuracy in the last round has a greater potential for performance improvement in the current round [31].”
Li, C., Zhang, W., Li, F., Guo, M., & Fan, S. (2025) · Applied Sciences · doi:10.3390/app15137186 · Cited by 6
Multilabel Chest X-Ray Image Classification via Category Disentangled Causal Learning
Li, Q., Liu, M., Chang, R., Nie, W., Bai, S., & Liu, A.-A. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3591094 · Cited by 5
“Theoretically, causal relationships remain consistent in both training and testing datasets [44], [45], [46].”
Yu, D., An, T., Chen, Y., Wang, Y., Fan, X., Wang, C., Zhao, W., Xu, X., & Zhu, L. (2025) · ACS ES&T Water · doi:10.1021/acsestwater.5c00467 · Cited by 5
Sikosana, M., Maudsley-Barton, S., & Ajao, O. (2025) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa65407.2025.11166406 · Cited by 6
LLM4Load: An LLM Prompt-Driven Framework for Multi-Scale Workload Prediction
Ding, Z., Ding, D., Zhang, H., Cao, J., & Xue, G. (2025) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws67624.2025.00044 · Cited by 6
Farah, E., & Shahrour, I. (2025) · Water · doi:10.3390/w17192886 · Cited by 5
Karbalaie, A., Strong, A., Nordström, T., Schelin, L., Selling, J., Grip, H., Prorok, K., & Häger, C. K. (2025) · Journal of Sports Sciences · doi:10.1080/02640414.2025.2578584 · Cited by 7
“These stochastic factors can lead to different model performances even when trained on the same data (Bengio et al., 2016; Pineau et al., 2021; Zhang et al., 2021).”
LandSegmenter: Towards a flexible foundation model for Land Use and Land Cover mapping
Liu, C., Huang, W., & Zhu, X. X. (2026) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2026.04.056 · Cited by 5
Kumar, D., Suthar, N., Rodriguez, R. V., & Hemachandran, K. (2025) · Journal of Information Communication and Ethics in Society · doi:10.1108/jices-07-2025-0174 · Cited by 6
Theodosiou, T. C., & Rekatsinas, C. S. (2025) · F1000Research · doi:10.12688/f1000research.169129.1 · Cited by 5
Fagbohun, T. R., Okechukwu, V. O., Adelusi, O. A., & Okonkwo, J. O. (2025) · Frontiers in Environmental Science · doi:10.3389/fenvs.2025.1718081 · Cited by 5
Hassan, M., Islam, M. S. B., Mahmud, S., Elgamal, M., Sumon, M. S. I., Hssain, A. A., Ibrahim, E., Kumar, A., Zhao, Y., Kashem, S. B. A., et al. (2025) · Talanta · doi:10.1016/j.talanta.2025.129332 · Cited by 5
Machine learning-driven new paradigm for Co-based superalloys
Luo, J., liu, X., Ma, Q., Pei, C., Yao, H., Xiong, J., & Gao, Q. (2026) · Journal of Materials Informatics · doi:10.20517/jmi.2025.52 · Cited by 9
Learning to See with Less: A Survey on Computer Vision with Limited and Imperfect Data
Fawakherji, M., Alaoui, S., & Turjya, S. M. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10558-y · Cited by 9
“From a training perspective, deep neural networks are known to be highly flexible and capable of memorizing noisy labels, especially in over-parameterized settings [43].”
Attention to Training Set Partitioning in Machine Learning for Solid-State Hydrogen Storage Alloys
Li, J., Zhang, C., Li, M.-Z., Luo, S., Lu, X., Zhu, Y.-K., Gao, Z.-Y., & Yang, W.-J. (2026) · Journal of Alloys and Compounds · doi:10.1016/j.jallcom.2026.187957 · Cited by 6
Self-citationMeshram, S., Itankar, P. R., Prasad, S. K., Sheikh, S. M., Husain, U., Badnale, A., Ghagare, A., & Dhakare, S. P. (2026) · Current Pharmacology Reports · doi:10.1007/s40495-026-00452-3 · Cited by 6
Self-supervised learning from noisy and incomplete data
Tachella, J. A., & Davies, M. E. (2026) · Foundations and Trends® in Signal Processing · doi:10.1108/ftsig-10-2025-0133 · Cited by 11
The Details Matter: Preventing Class Collapse in Supervised Contrastive Learning
Fu, D. Y., Chen, M. F., Zhang, M., Fatahalian, K., & Ré, C. (2022) · doi:10.3390/cmsf2022003004 · Cited by 12
Point Cloud Instance Segmentation with Inaccurate Bounding-Box Annotations
Peng, Y., Feng, H., Chen, T., & Hu, B. (2023) · Sensors · doi:10.3390/s23042343 · Cited by 5
“Since deep neural networks are highly capable of learning any complex function, it is easy to overfit inaccurate labels and reduce the generalization performance [26].”
Huang, X., & Chong, K. F. E. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01815-9 · Cited by 7
“…ID or OOD, these NC instances may lead to significant performance degradation during training, which is not surprising since neural networks are capable of achieving zero training error even in the extreme case of completely noisy data (Arpit et al., 2017; Yao et al., 2021; Zhang et al., 2021).”
BackgroundMethodLei, Y., & Huang, R. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.409 · Cited by 5
“As revealed in (Zhang et al., 2021), it is due to fine tuning’s drawback of overfitting, making the model simply memorize the labeled samples and fail to learn generalizable features for new domains.”
BackgroundEvaluation of targeted dataset collection on racial equity in face recognition
Hong, R. S., Kohno, T., & Morgenstern, J. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604662 · Cited by 7
“This demonstrates the notion that large neural networks have extensive capacity to capture arbitrarily complex functions [48], which also applies to new samples from distinct demographic groups.”
BackgroundWalking path images from real-time location data predict degree of cognitive impairment
Faruk, T., Shum, L. C., Iaboni, A., & Khan, S. S. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102657 · Cited by 5
Automatic Noise Generation and Reduction for Text Classification
Chen, H., Sun, Y., Zhang, M., & Zhang, M. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3325135 · Cited by 9
“D ATAnoiseisinevitableinsupervisedmachinelearning[1], [2],[3],whichcouldleadtodegradedperformancefor targeted tasks.”
Dynamics and Perturbations of Overparameterized Linear Neural Networks
de Oliveira, A. C. B., Siami, M., & Sontag, E. D. (2023) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc49753.2023.10383478 · Cited by 9
Jiang, K., Cao, B., & Fan, J. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2023.046348 · Cited by 7
Learning Accurate Pseudo-Labels via Feature Similarity in the Presence of Label Noise
Wang, P., Wang, X., Wang, Z., & Dong, Y. (2024) · Applied Sciences · doi:10.3390/app14072759 · Cited by 7
Khagi, B., Belousova, T., Short, C. M., Taylor, A. A., Bismuth, J. X., Shah, D. J., & Brunner, G. (2024) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2024.03.035 · Cited by 6
Co–TES: Learning noisy labels with a Co-Teaching Exchange Student method
Shin, C., & Oh, S. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.04.001 · Cited by 5
Decoding class dynamics in learning with noisy labels
Tatjer, A., Nagarajan, B., Marques, R., & Radeva, P. I. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.04.012 · Cited by 5
MFNet: Meta‐learning based on frequency‐space mix for MRI segmentation in nasopharyngeal carcinoma
Li, Y., Chen, Q., Li, H., Wang, S., Chen, N., Han, T., Wang, K., Yu, Q., Cao, Z., & Tang, J. (2024) · Journal of Cellular and Molecular Medicine · doi:10.1111/jcmm.18355 · Cited by 5
“14 It helps prevent overfitting to the training data and enables better generalization to the test data.”
ReLU neural networks of polynomial size for exact maximum flow computation
Hertrich, C., & Sering, L. (2024) · Mathematical Programming · doi:10.1007/s10107-024-02096-x · Cited by 16
“Yet they usually do not suffer from overfitting and the generalization error is much lower than expected [75].”
MFPINC: prediction of plant ncRNAs based on multi-source feature fusion
Nie, Z., Gao, M., Jin, X., Rao, Y., & Zhang, X. (2024) · BMC Genomics · doi:10.1186/s12864-024-10439-3 · Cited by 6
“Therefore, in recent years, an increasing number of studies have explored feature fusion meth-ods, which aim to effectively integrate features from different sources or at different levels [26].”
BackgroundFetoML: Interpretable predictions of the fetotoxicity of drugs based on machine learning approaches
Jeong, M., & Yoo, S. (2024) · Molecular Informatics · doi:10.1002/minf.202300312 · Cited by 6
“Additionally, we used L2 regularization to prevent the model from overfitting the training set [71, 72].”
MethodDiving Into Sample Selection for Facial Expression Recognition With Noisy Annotations
Nie, W., Wang, Z., Wang, X., Chen, B., Zhang, H., & Liu, H. (2024) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2024.3435498 · Cited by 5
Xue, H., Cui, S., Ma, Q., Li, Z., Zhou, P., Li, Y., & Xie, L. (2024) · Engineering Applications of Computational Fluid Mechanics · doi:10.1080/19942060.2024.2394638 · Cited by 5
“A larger training set helps the model learn more effectively from the available data, improving its ability to generalize (Zhang et al., 2021).”
Multibranch Time-Frequency Spectrum Segmentation Approach for Gearbox Fault Diagnosis
Song, H., Sun, Y., Gao, H., Guo, L., & Chen, Y. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3457878 · Cited by 5
Yu, Z., Gong, H., Zhang, S., & Wang, W. (2024) · Remote Sensing · doi:10.3390/rs16183430 · Cited by 5
Zhu, J., Ma, X., Yu, D., Li, Y., & Zhao, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125349 · Cited by 5
Chen, T., Zhang, D., & Lee, D.-J. (2024) · Electronics · doi:10.3390/electronics13193891 · Cited by 6
“Sample selection methods are typically based on the small-loss assumption, which posits that neural networks tend to fit clean samples before overfitting to noisy labels [24].”
Xu, L., Gan, Y., & Jin, Y. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3491106 · Cited by 5
“Thisissueisalsocritical in deep learning-based expression recognition methods, as deep models tend to memorize noisy data [8], and the uncertainty often introduces noise [9].”
Combating Noisy Labels by Alleviating the Memorization of DNNs to Noisy Labels
Yuan, S. W., Li, X., Miao, Y., Zhang, H., Liu, X., & Deng, R. H. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521722 · Cited by 8
“However, recent studies [15], [48] have revealed that DNNs can easily fit even the random labels of training data.”
Unified regularity measures for sample-wise learning and generalization
Zhang, C., Yuan, M., Ma, X., Liu, Y., Lu, H., Wang, L., Su, Y., & Liu, Y. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00069-4 · Cited by 6
“Zhang et al. [11, 12] further reported that the deep neural network trained with the ordinary stochastic gradient descent (SGD) [13] method could easily fit random labels.”
Self-citationYetman, C. C. (2026) · Ergo an Open Access Journal of Philosophy · doi:10.3998/ergo.10324 · Cited by 5
“Zhang et al. (2021) show that the effective capacity of deep convolutional neural networks is often sufficient for memorizing a massive set of completely random data.”
Text Classification with Limited Training Data: Suicide Risk Detection on Social Media
Pasch, S., & Cutura, J. A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825333 · Cited by 4
CCT: A Cyclic Co-Teaching Approach to Train Deep Neural Networks With Noisy Labels
Kamassury, J. K. S., da Silva, H. P., Cordeiro, F. R., & Silva, D. (2025) · IEEE Access · doi:10.1109/access.2025.3548510 · Cited by 5
“The presence of noisy labels poses a significant challenge in DL due to the remarkable ability of models to memorize entire datasets, including incorrectly labeled samples [3].”
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.