Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 201–300 of 2,667
Algorithms going wild – A review of machine learning techniques for terrestrial ecology
Cipriano, C., Noce, S., Mereu, S., & Santini, M. (2025) · Ecological Modelling · doi:10.1016/j.ecolmodel.2025.111164 · Cited by 54
Winsorization for Robust Bayesian Neural Networks
Sharma, S., & Chatterjee, S. (2021) · Entropy · doi:10.3390/e23111546 · Cited by 58
“Such aspects of DL fitting have been observed earlier and are of considerable interest in studies on the properties of DL [4,5].”
BackgroundS3IM: Stochastic Structural SIMilarity and Its Unreasonable Effectiveness for Neural Fields
Xie, Z., Yang, X.-D., Yang, Y., Sun, Q., Jiang, Y.-X., Wang, H., Cai, Y., & Sun, M. (2023) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.01652 · Cited by 64
Synthesizing class labels for highly imbalanced credit card fraud detection data
Kennedy, R. K. L., Villanustre, F., Khoshgoftaar, T. M., & Salekshahrezaee, Z. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00897-7 · Cited by 58
“Noisy or inaccurately labeled data can drastically impact the effectiveness and usability of classification models [2].”
BackgroundDeep learning models for triaging hospital head MRI examinations
Wood, D. A., Kafiabadi, S., Busaidi, A. A., Guilhem, E., Montvila, A., Lynch, J., Townend, M., Agarwal, S., Mazumder, A., Barker, G. J., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102391 · Cited by 56
“Recent studies have shown that this ‘label noise’ can significantly impact the performance of deep learning models ( Karimi et al., 2020 ; Zhang et al., 2021 ).”
BackgroundUnveiling the Structure of Wide Flat Minima in Neural Networks
Baldassi, C., Lauditi, C., Malatesta, E. M., Perugini, G., & Zecchina, R. (2021) · Physical Review Letters · doi:10.1103/physrevlett.127.278301 · Cited by 66
A Parallel Feature Fusion Network Combining GRU and CNN for Motor Imagery EEG Decoding
Gao, S., Yang, J. L., Shen, T., & Jiang, W. (2022) · Brain Sciences · doi:10.3390/brainsci12091233 · Cited by 48
“More training data can be generated by rotation, translation, and cropping to alleviate the overfitting problem and improve the accuracy and robustness of the model [34].”
BackgroundA survey on membership inference attacks and defenses in machine learning
Niu, J., Liu, P., Zhu, X., Shen, K., Wang, Y., Chi, H., Shen, Y., Jiang, X., Ma, J., & Zhang, Y. (2024) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2024.02.001 · Cited by 81
Random neural networks in the infinite width limit as Gaussian processes
Hanin, B. (2023) · The Annals of Applied Probability · doi:10.1214/23-aap1933 · Cited by 82
Hullman, J. R., Kapoor, S., Nanayakkara, P., Gelman, A., & Narayanan, A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534196 · Cited by 59
“Recent results show how counter to classical expectations about overfitting, minimizing training error without explicit regularization over overparameterized models tends to result in good generalization [156, 191, 222], driving a new theoretical agenda aimed at disentangling optimization methods and statistical prope…”
BackgroundA Unified Linear Speedup Analysis of Federated Averaging and Nesterov FedAvg
Qu, Z., Lin, K., Li, Z., Zhou, J., & Zhou, Z. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15180 · Cited by 54
On the Generalization Mystery in Deep LearningPreprint
Chatterjee, S., & Zieliński, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.10036 · Cited by 59
“…why gradient descent generalizes well in the simple case of over-parameterized linear regression (such as Example 1) since the Hessian of the loss function is constant over the parameter space, and thus the same at both well and poorly generalizing minima [Zhang et al., 2021, Section 5].”
BackgroundCo-LDL: A Co-Training-Based Label Distribution Learning Method for Tackling Label Noise
Sun, Z., Liu, H., Wang, Q., Zhou, T., Wu, Q., & Tang, Z. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3116430 · Cited by 54
Zhang, H. K., Roy, D. P., & Luo, D. (2023) · Remote Sensing of Environment · doi:10.1016/j.rse.2023.113653 · Cited by 50
Revisiting the Performance-Explainability Trade-Off in Explainable Artificial Intelligence (XAI)
Crook, B., Schlüter, M., & Speith, T. (2023) · 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew57809.2023.00060 · Cited by 36
“However, recent theoretical work suggests that neural networks often do not overfit, even when their parameters outnumber the training samples [63].”
BackgroundSource Inference Attacks: Beyond Membership Inference Attacks in Federated Learning
Hu, H., Zhang, X., Salcic, Z. A., Sun, L., Choo, K. R., & Dobbie, G. (2023) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2023.3321565 · Cited by 65
“This enables DNNs to learn patterns effectively from the training data on one side while on the other side such models can have unnecessarily high capacities to memorize the details of the training data [53], [54], [56], which can lead to the overfitting of ML models.”
BackgroundPixel-Level Noise Mining for Weakly Supervised Salient Object Detection
Liu, K.-D., Feng, M.-T., Zhao, W., Sun, J.-T., Dong, W.-S., Wang, Y.-N., & Mian, A. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3575255 · Cited by 40
“However, network training is highly prone to overfit noisy supervision signals [16].”
Li, R., Wang, H., Dai, H., Hong, J., Tong, G., & Chen, X. (2022) · Energy · doi:10.1016/j.energy.2022.123853 · Cited by 48
Somepalli, G., Fowl, L., Bansal, A., Yeh-Chiang, P., Dar, Y., Baraniuk, R. G., Goldblum, M., & Goldstein, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01333 · Cited by 47
“It is known that neural networks can easily overfit complex datasets, and can even interpolate randomly labeled images [39].”
BackgroundInstance-Dependent Noisy Label Learning via Graphical Modelling
Garg, A. K., Nguyen, C. C., Felix, R., Do, T.-T., & Carneiro, G. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00232 · Cited by 63
OMG: Towards Effective Graph Classification Against Label Noise
Yin, N., Shen, L., Wang, M., Luo, X., Luo, Z., & Tao, D. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3271677 · Cited by 71
“After learning generalized patterns, noisy samples still do harm to the performance [15], [27], [28].”
BackgroundA practitioner’s guide to Kolmogorov–Arnold networks
Noorizadegan, A., Wang, S., Ling, L., & Dominguez-Morales, J. P. (2026) · Computer Science Review · doi:10.1016/j.cosrev.2026.100991 · Cited by 55
Impact of Training Dataset Size on Neural Answer Selection Models
Linjordet, T., & Balog, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-15712-8_59 · Cited by 53
SoK: Cryptographic Neural-Network Computation
Ng, L. K. L., & Chow, S. S. M. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179483 · Cited by 77
“Understanding more about deep learning still needs more research [61].”
BackgroundFederated Noisy Client Learning
Tam, K., Li, L., Han, B., Xu, C., & Fu, H. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3336050 · Cited by 61
“In deep learning, it has been empirically shown that deep networks can learn the clean pattern at the start of training [38], [41]–[43].”
BackgroundZhang, J., Luo, B., Li, F., Niu, X., Zhang, Q., Wang, Y., & Chen, X. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3185660 · Cited by 47
“their excellent performances highly rely on collecting largescale training samples that are not only time-consuming but also sensitive to imaging settings [23], [24].”
BackgroundPhysics-infused Machine Learning for Crowd Simulation
Zhang, G., Yu, Z., Jin, D., & Li, Y. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539440 · Cited by 61
“However, methods of this type are typically hard to generalize to different scenarios [37].”
MethodTraining Data Extraction From Pre-trained Language Models: A Survey
Ishihara, S. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.23 · Cited by 84
Problem-Dependent Power of Quantum Neural Networks on Multiclass Classification
Du, Y., Yang, Y., Tao, D., & Hsieh, M.-H. (2023) · Physical Review Letters · doi:10.1103/physrevlett.131.140601 · Cited by 72
“This is in line with prior results in the context of deep learning theory [112–114] and the generalization of quantum neural networks [54–56, 124 • From the perspective of trainability, over-parameterization in our work refers to the scenario where the adopted ansatz U ( θ ) forms a 2-design,…”
BackgroundResultInfluentialWang, Y., Wang, W., Liu, D., Hou, W., Zhou, T., & Ji, Z. (2023) · Genome biology · doi:10.1186/s13059-023-03054-0 · Cited by 55
“Training DNNs on such noisy labeled datasets may cause suboptimal performance because it has been proved that DNNs easily overfit to noisy labels given their powerful learning ability [16].”
BackgroundBadLabel: A Robust Perspective on Evaluating and Enhancing Label-Noise Learning
Zhang, J., Song, B., Wang, H., Han, B., Liu, T., Liu, L., & Sugiyama, M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3355425 · Cited by 62
Liu, H., Wu, Y., Yu, Z., & Zhang, N. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00120 · Cited by 82
Liu, P., Qian, W., Zhang, H., Zhu, Y., Hong, Q., Li, Q., & Yao, Y. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10926-9 · Cited by 69
Dou, Y., Han, X., & Lin, P. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110547 · Cited by 54
Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning
Zhu, K., Hu, X., Wang, J., Xie, X., & Yang, G. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00408 · Cited by 62
Perbet, P., Guindon, L., Côté, J.-F., & Béland, M. (2024) · Remote Sensing of Environment · doi:10.1016/j.rse.2024.114107 · Cited by 47
Learning From Multiple Expert Annotators for Enhancing Anomaly Detection in Medical Image Analysis
Le, K. H., Tran, T. V., Pham, H., Nguyen, H. T., Le, T., & Nguyen, H. Q. (2023) · IEEE Access · doi:10.1109/access.2023.3243845 · Cited by 56
“…of broader topics such as curriculum learning [29], which is biologically inspired by human gradual learning, with easier examples are preferred in early training stages; learning with noisy labels [30, 31], which prefers examples with smaller training losses as they are more likely to be clean.”
BackgroundChen, Z., Liu, X., Yang, Q., Wang, Y.-J., Miao, K., Gong, Z., Yu, Y., Leonov, A., Liu, C., Feng, Z., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.1671 · Cited by 43
Underwater object detection using collaborative weakly supervision
Cai, S., Li, G., & Yuan, S. (2022) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2022.108159 · Cited by 43
Zhu, E., Zong, Z., Li, E., Lu, Y., Zhang, J., Xie, H., Li, Y., Yin, W., & Wei, Z. (2025) · Nature Communications · doi:10.1038/s41467-025-57516-z · Cited by 48
Hadamard Product in Deep Learning: Introduction, Advances and Challenges
Chrysos, G. G., Wu, Y., Pascanu, R., Torr, P. H. S., & Cevher, V. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3560423 · Cited by 59
Mean-field inference methods for neural networks
Gabrié, M. (2020) · Journal of Physics A Mathematical and Theoretical · doi:10.1088/1751-8121/ab7f65 · Cited by 59
Liu, H., Zhang, C., Yao, Y., Wei, X.-S., Shen, F., Tang, Z., & Zhang, J. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3055024 · Cited by 44
Self-citationReview–A Survey of Learning from Noisy Labels
Liang, X., Liu, X., & Yao, L. (2022) · ECS Sensors Plus · doi:10.1149/2754-2726/ac75f5 · Cited by 50
Turnpike in optimal control of PDEs, ResNets, and beyond
Geshkovski, B., & Zuazua, E. (2022) · Acta Numerica · doi:10.1017/s0962492922000046 · Cited by 68
Zhu, H., Yang, J., & Huang, N. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c01149 · Cited by 47
Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning
Huang, L., Zhang, C., & Zhang, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3217792 · Cited by 50
Self-citationLearning to Detect Mobile Objects from LiDAR Scans Without Labels
You, Y., Luo, K. Z., Phoo, C. P., Chao, W., Sun, W. J., Hariharan, B., Campbell, M., & Weinberger, K. Q. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00120 · Cited by 73
“3We note that for detection problems, a neural network can hardly overfit the training data to achieve 100% accuracy [40], in contrast to classification problems [65].”
BackgroundTRGP: Trust Region Gradient Projection for Continual Learning
Lin, S., Yang, L., Fan, D., & Zhang, J. (2022) · arXiv · doi:10.48550/arxiv.2202.02931 · Cited by 43
“Based on the fact that stochastic gradient descent updates lie in the subspace spanned by the data input (Zhang et al., 2021; Saha et al., 2021), it follows that the gradient Therefore, the projection of ∇ W l L 2 onto the orthogonal direction to S l 1 is 0, which means that the model W l 1 will…”
BackgroundDeep Learning–Assisted Burn Wound Diagnosis: Diagnostic Model Development Study
Chang, C.-W., Lai, F., Christian, M., Chen, Y. C., Hsu, C., Chen, Y., Chang, D., Roan, T., & Yu, Y. C. (2021) · JMIR Medical Informatics · doi:10.2196/22798 · Cited by 49
DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition With Limited Annotations
Hu, P., Sun, X., Sclaroff, S., & Saenko, K. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3346405 · Cited by 54
“…of lightweight learnable prompts, the proposed methods effectively recognize different classes within complex visual scenes. attempt to use web supervision to automatically generate the pseudo labels, which unfortunately leads to poor performance as the web supervision is noisy and incomplete [56].”
BackgroundAutomatic detection of urban flood level with YOLOv8 using flooded vehicle dataset
Wan, J., Qin, Y., Shen, Y., Yang, T., Xu, Y., Zhang, S., Yang, G., Xue, F., & Wang, Q. J. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131625 · Cited by 42
A review of explainable AI methods and their application in manufacturing systems
Tzionis, G., Mouratidis, P., Kougka, G., Gialampoukidis, I., Vrochidis, S., Kompatsiaris, I. Y., & Vlachopoulou, M. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07908-z · Cited by 50
Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review
Le, M. T., Wolinski, P., & Arbel, J. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3798276 · Cited by 53
“Another crucial property of NNs, related to generalization in statistical learning theory [202, 205], is their ability to generalize to new data with fewer examples than parameters, even with very large models [102, 122], and they are even capable of labeling random data [222].”
Chen, J., Wan, J., Ye, G., & Wang, Y. (2024) · Bioresource Technology · doi:10.1016/j.biortech.2024.131362 · Cited by 42
Hsu, C.-Y., Li, W., & Wang, S. (2024) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2024.2397441 · Cited by 49
Privacy preserving large language models: ChatGPT case study based vision and framework
Ullah, I., Hassan, N., Gill, S. S., Suleiman, B., Ahanger, T. A., Shah, Z., Qadir, J., & Kanhere, S. S. (2024) · IET Blockchain · doi:10.1049/blc2.12091 · Cited by 46
Scalable Penalized Regression for Noise Detection in Learning with Noisy Labels
Wang, Y., Sun, X., & Fu, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00044 · Cited by 53
Multi-label Iterated Learning for Image Classification with Label Ambiguity
Rajeswar, S., Rodríguez, P., Singhal, S., Vázquez, D., & Courville, A. C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00474 · Cited by 56
“[66], who showed that noise memorization happens later in the training procedure.”
BackgroundDelving into Sample Loss Curve to Embrace Noisy and Imbalanced Data
Jiang, S., Li, J., Wang, Y., Huang, B., Zhang, Z., & Xu, T. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i6.20661 · Cited by 53
Effectiveness of Distillation Attack and Countermeasure on Neural Network Watermarking
Yang, Z., Tang, T., Dang, H., Wu, Z., & Chang, E.-C. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3609476 · Cited by 45
Granziol, D., Zohren, S., & Roberts, S. (2020) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.2006.09092 · Cited by 56
“Since DNNs have been shown to easily fit completely random data (Zhang et al., 2021) and have exponentially local minima in the training loss (Choromanska et al., 2015a), which are close to the global minimum of training loss, there are likely to be many regions of low training error/loss.”
BackgroundWang, G., Zhang, X., Peng, Z., Jia, X., Tang, X., & Jiao, L. (2023) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2023.01.011 · Cited by 42
Wang, K., Zhuang, S., Ran, Q.-Y., Zhou, P., Hua, J., Zhou, G., & He, X. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102832 · Cited by 48
Kiyani, E., Shukla, K., Karniadakis, G. E., & Karttunen, M. (2023) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2023.116258 · Cited by 44
Bodendorf, F., Sauter, M., & Franke, J. E. (2022) · International Journal of Production Economics · doi:10.1016/j.ijpe.2022.108708 · Cited by 40
Cheng, S., Wang, Z., Yang, B., & Nakano, K. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3265662 · Cited by 43
“Apart from the batch normalization and dropout techniques, the fine-tuned model structure and parameters are all defined for better generalization performance [47].”
MethodImproving Satellite Retrieval of Coastal Aquaculture Pond by Adding Water Quality Parameters
Hou, Y., Zhao, G., Chen, X., & Yu, X. (2022) · Remote Sensing · doi:10.3390/rs14143306 · Cited by 41
“, the adaptability of the classification approach in fresh samples [63].”
BackgroundLow Curvature Activations Reduce Overfitting in Adversarial Training
Singla, V., Singla, S., Feizi, S., & Jacobs, D. W. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01611 · Cited by 55
“The standard bias-variance trade-off from classical machine learning theory fails to explain why deep networks generalize well especially when they have far more parameters than the samples they are trained on [77].”
BackgroundInfluentialQuantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification
Zhang, Q., Xu, C., Chen, Y., & Rao, Z. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3200344 · Cited by 61
Improved Residual Network based on norm-preservation for visual recognition
Mahaur, B., Mishra, K. K., & Singh, N. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.10.023 · Cited by 40
Chisco: An EEG-based BCI dataset for decoding of imagined speech
Zhang, Z., Ding, X., Bao, Y., Zhao, Y., Liang, X., Qin, B., & Liu, T. (2024) · Scientific Data · doi:10.1038/s41597-024-04114-1 · Cited by 59
A Hybrid Improved Neural Networks Algorithm Based on L2 and Dropout Regularization
Xie, X., Xie, M., Moshayedi, A. J., & Skandari, M. H. N. (2022) · Mathematical Problems in Engineering · doi:10.1155/2022/8220453 · Cited by 46
Hammadi, S. S., Majeed, B. H., & Hassan, A. K. (2023) · International Journal of Emerging Technologies in Learning (iJET) · doi:10.3991/ijet.v18i06.38615 · Cited by 39
Identifying active retrogressive thaw slumps from ArcticDEM
Huang, L., Willis, M. J., Li, G., Lantz, T. C., Schaefer, K. M., Wig, E., Cao, G., & Tiampo, K. F. (2023) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2023.10.008 · Cited by 39
Learning With Imbalanced Noisy Data by Preventing Bias in Sample Selection
Liu, H., Sheng, M., Sun, Z., Yao, Y., Hua, X., & Shen, H. T. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3368910 · Cited by 44
Robust Node Classification on Graph Data with Graph and Label Noise
Zhu, Y., Feng, L., Deng, Z., Chen, Y., Amor, R., & Witbrock, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i15.29668 · Cited by 71
“It has been validated that deep learning models easily overfit to noisy labels with any ratio of corrupted labels (Zhang et al. 2021).”
BackgroundSVFAP: Self-Supervised Video Facial Affect Perceiver
Sun, L., Lian, Z., Wang, K., He, Y., Xu, M., Sun, H., Liu, B., & Tao, J. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3436913 · Cited by 52
“In the following sections, we elaborate on both pre-training and fine-tuning details.”
Xu, L., Ning, S., Xu, X., Wang, S., Chen, L., Long, R., Zhang, S., Zhou, Y., Zhang, M., & Thapa, B. R. (2024) · Agricultural Water Management · doi:10.1016/j.agwat.2024.109176 · Cited by 43
Self-supervised learning for label-free segmentation in cardiac ultrasound
Ferreira, D. L., Lau, C., Salaymang, Z., & Arnaout, R. (2025) · Nature Communications · doi:10.1038/s41467-025-59451-5 · Cited by 44
“Deep neural networks have been observed to be robust to label noise and fi rst fi t the training data with clean labels during the early learning phase, before eventually memorizing the examples with false labels or artifacts 58 – 62 .”
Latent Gaussian Model Boosting
Sigrist, F. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3168152 · Cited by 41
Gaussian universality of perceptrons with random labels
Gerace, F., Krząkała, F., Loureiro, B., Stephan, L., & Zdeborová, L. (2024) · Physical review. E · doi:10.1103/physreve.109.034305 · Cited by 61
“…the definition of Rademacher complexities [7, 20], in Wendel/Cover’s pioneering studies [21, 22] or in thought-provoking numerical experiments with deep learning [23, 24], In this work, we ask: how would these theories for random labels change if using a realistic data set instead of a Gaussian one?”
Towards Accurate and Robust Domain Adaptation Under Multiple Noisy Environments
Han, Z., Gui, X.-J., Sun, H., Yin, Y., & Li, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3215150 · Cited by 51
Machine Learning of Implicit Combinatorial Rules in Mechanical Metamaterials
van Mastrigt, R., Dijkstra, M., van Hecke, M., & Coulais, C. (2022) · Physical Review Letters · doi:10.1103/physrevlett.129.198003 · Cited by 37
“Conversely, can these problems help us understand why neural networks work so well [74]?”
BackgroundUnsupervised Spectral Demosaicing With Lightweight Spectral Attention Networks
Feng, K., Zeng, H., Zhao, Y., Kong, S. G., & Bu, Y. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3364064 · Cited by 41
“We attribute this outcome to HSA introducing an excessive number of learning parameters, rendering it susceptible to noise or irrelevant information affecting network fitting [43].”
BackgroundMeta-Learning Guided Label Noise Distillation for Robust Signal Modulation Classification
Hao, X., Feng, Z., Peng, T., & Yang, S. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3462544 · Cited by 39
“Similar to many computer vision tasks [39], [40], signal annotation can be expensive, time consuming, and prone to errors.”
Machine learning based eddy current testing: A review
Munir, N., Huang, J., Wong, C., & Song, S.-J. (2024) · Results in Engineering · doi:10.1016/j.rineng.2024.103724 · Cited by 51
Causal Interventional Training for Image Recognition
Qin, W., Zhang, H., Hong, R., Lim, E., & Sun, Q. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3136717 · Cited by 37
“Since deep neural networks are driven by data, biased data inevitably cause biased models, resulting in poor generalization for test domains [6].”
BackgroundInstruments, agents, and artificial intelligence: novel epistemic categories of reliability
Duede, E. (2022) · Synthese · doi:10.1007/s11229-022-03975-6 · Cited by 48
“As a matter of fact, scientists do not yet understand how deep learning models generalize nor the conditions under which they will generalize well rather than poorly Räz (2022), Zhang et al. (2021).”
BackgroundOnline Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries
Bang, J., Koh, H., Park, S., Song, H., Ha, J.-W., & Choi, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00906 · Cited by 81
“However, on the contaminated data stream, diversity-based memory construction may promote to include many examples with corrupted labels in the episodic memory, leading to poor performance thanks to very high capacity of deep models to overly fit all the falsely labeled examples [40].”
BackgroundAn AI toolkit to support teacher reflection
Phillips, T. M., Saleh, A., & Ozogul, G. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-022-00295-1 · Cited by 36
“Second, regularization via dropout, data augmentation, and other means can discour - age overfitting and increase the robustness of the model (Zhang et al., 2021 ).”
BackgroundYu, P., Zuo, Z., & Lian, D. (2024) · Journal of the Knowledge Economy · doi:10.1007/s13132-024-01793-4 · Cited by 44
“To further validate our main regression results, we used total factor productivity (TFP), an alternative metric to measure corporate high-quality development, in accordance with the methodology proposed by Zhang (2021).”
RetractedContrasting random and learned features in deep Bayesian linear regression
Zavatone-Veth, J. A., Tong, W. L., & Pehlevan, C. (2022) · Physical review. E · doi:10.1103/physreve.105.064118 · Cited by 54
A Fully Deep Learning Paradigm for Pneumoconiosis Staging on Chest Radiographs
Sun, W., Wu, D., Luo, Y., Liu, L., Zhang, H., Wu, S., Zhang, Y., Wang, C., Zheng, H., Jiang, S., et al. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3190923 · Cited by 39
Altarabichi, M. G., Nowaczyk, S., Pashami, S., Mashhadi, P. S., & Handl, J. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120500 · Cited by 39
“However, the regularizing effect of dropout does not always protect against overfitting, as shown by Zhang et al. [4], who were able to perfectly fit a network regularized with dropout to the CIFAR10 dataset even when all the image labels were replaced with random ones.”
BackgroundHiURE: Hierarchical Exemplar Contrastive Learning for Unsupervised Relation Extraction
Liu, S., Hu, X., Zhang, C., Li, S., Wen, L., & Yu, P. S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.437 · Cited by 49
Self-citationLearning from Pixel-Level Noisy Label : A New Perspective for Light Field Saliency Detection
Feng, M., Liu, K., Zhang, L., Yu, H., Wang, Y., & Mian, A. S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00180 · Cited by 38
“However, directly training the network with single noisy labelling will not work as it is well-known that network training is highly prone to noise in the supervision labels which may guide the network to overfit to the corrupted labels [47].”
BackgroundJiménez-Mesa, C., Arco, J. E., Valentí, M., Payo, B. F., Zea‐Sevilla, M. A., Ortíz, A., Ávila‐Villanueva, M., Castillo-Barnés, D., Ramírez, J., del Ser-Quijano, T., et al. (2023) · International Journal of Neural Systems · doi:10.1142/s0129065723500156 · Cited by 39
Understanding Transformer Memorization Recall Through Idioms
Haviv, A., Cohen, I., Gidron, J., Schuster, R., Goldberg, Y., & Geva, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.19 · Cited by 93
Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descentPreprint
Li, Y., & Wei, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09502 · Cited by 43
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.