Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 301–400 of 2,667
A Data-centric Framework to Endow Graph Neural Networks with Out-Of-Distribution Detection Ability
Guo, Y., Yang, C. H., Chen, Y., Liu, J., Shi, C., & Du, J. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599244 · Cited by 68
“However, training such an encoder may suffer from complexity in computation [19] and difficulty in optimization [56].”
BackgroundSalaciaML: A Deep Learning Approach for Supporting Ocean Data Quality Control
Mieruch, S., Demirel, S., Simoncelli, S., Reiner Schlitzer, E., & Seitz, S. (2021) · Frontiers in Marine Science · doi:10.3389/fmars.2021.611742 · Cited by 34
“…each), with in total 128 ∗ 64 = 8, 192 parameters (weights), we are dealing with a quite small network compared to the data, while often deep artificial neural networks have more parameters than data to be trained on, and still are skillful and generalize well (details e.g., in Zhang et al., 2016).”
BackgroundZhang, R., Yao, W., Shi, Z., Ai, X., Tang, Y., & Wen, J. (2022) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2022.3220569 · Cited by 39
“In the application scenario, considering that there will be a portion of mislabels due to operational errors (the ratio should be less than 1%), the training of deep learning models will still not be significantly affected [44].”
BackgroundIntegrative conformal p-values for out-of-distribution testing with labelled outliers
Liang, Z., Sesia, M., & Sun, W. (2024) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1093/jrsssb/qkad138 · Cited by 68
On the computational complexity of ethics: moral tractability for minds and machines
Stenseke, J. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10732-3 · Cited by 37
Can Neural Quantum States Learn Volume-Law Ground States?
Passetti, G., Hofmann, D., Neitemeier, P., Grunwald, L., Sentef, M. A., & Kennes, D. M. (2023) · Physical Review Letters · doi:10.1103/physrevlett.131.036502 · Cited by 45
“Since it is well known that deep (and, in fact, already two-layer) networks are able to memorize even completely random data once the number of network parameters exceeds the number of data points [43], these results provide evidence that our FFNN ansatz does not learn to utilize any of this structure but only manages…”
BackgroundGligorić, Z., Görçün, Ö. F., Gligorić, M., Pamucar, D. S., Šimić, V., & Küçükönder, H. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102079 · Cited by 31
“In that regard, Zhang et al. (2021) pointed out the weakness of the ability to generalize as one of the most critical gaps in studies on deep learning.”
CIPHER: Cybersecurity Intelligent Penetration-Testing Helper for Ethical Researcher
Pratama, D., Suryanto, N., Adiputra, A. A., Le, T.-T.-H., Kadiptya, A. Y., Iqbal, M. J., & Kim, H. (2024) · Sensors · doi:10.3390/s24216878 · Cited by 49
“Larger models with more parameters can significantly enhance generalization ability [74].”
BackgroundInterspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNs
Wimmer, P., Mehnert, J., & Condurache, A. P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01220 · Cited by 41
Paolini, G., Escorihuela, M., Merlin, O., Pàmies-Sans, M., & Bellvert, J. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3222884 · Cited by 38
Efficient neural codes naturally emerge through gradient descent learning
Benjamin, A. S., Zhang, L.-Q., Qiu, C., Stocker, A. A., & Kording, K. P. (2022) · Nature Communications · doi:10.1038/s41467-022-35659-7 · Cited by 40
“It is now recognized that what neural network learn about their inputs is constrained not just by their connectivity but also implicitly by their learning algorithm (Zhang et al, 2021).”
BackgroundPolynomial‐time universality and limitations of deep learning
Abbe, E. A., & Sandon, C. (2023) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.22121 · Cited by 34
Bridging the Gap Between Model Explanations in Partially Annotated Multi-Label Classification
Kim, Y., Kim, J. M., Jeong, J., Schmid, C., Akata, Z., & Lee, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00332 · Cited by 50
“It is well known that neural networks can memorize wrong labels due to their large model capacities [45].”
Unsupervised Deep Learning for Ground Roll and Scattered Noise Attenuation
Liu, D., Sacchi, M. D., Wang, X., & Chen, W. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3325324 · Cited by 42
“Zhang et al. [55] draw considerable attention to a non-overfitting puzzle in deep neural networks, contradicting traditional generalization theory which suggests that models with excessive parameters easily overfit the data.”
BackgroundNoisy Recurrent Neural NetworksPreprint
Lim, S. H., Erichson, N. B., Hodgkinson, L., & Mahoney, M. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.04877 · Cited by 45
“For neural network (NN) models, implicit regularization towards wider minima is conjectured to be a prominent ingredient in the success of stochastic optimization [67, 28].”
BackgroundA Lightweight Block With Information Flow Enhancement for Convolutional Neural Networks
Bao, Z., Yang, S., Huang, Z., Zhou, M., & Chen, Y. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3237615 · Cited by 35
Lee, H.-T., Cheon, H.-R., Lee, S., Shim, M., & Hwang, H. (2023) · Scientific Reports · doi:10.1038/s41598-023-43542-8 · Cited by 33
“For example, a 10-s EEG epoch results in a series of five 2-s segments with a window length of 2 s without overlap, and then each segment is used as an independent trial (sample) to train a deep-learning model as well as estimate a diagnostic accuracy 7,8 .”
MethodDual Knowledge Distillation on Multiview Pseudo Labels for Unsupervised Person Re-Identification
Zhu, W., Peng, B., & Yan, W. Q. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3366395 · Cited by 42
“Research outcome has shown that deep neural networks have the capability to fit any ratio of noisy labels [35].”
BackgroundXu, S., Jiang, W., Liu, Q., Wang, H., Zhang, J., Li, J., Huang, X. L., & Bo, Y. (2024) · Underground Space · doi:10.1016/j.undsp.2024.07.003 · Cited by 33
Palanisamy, M., Vincent, M. T. P., & Hossain, M. B. (2025) · Journal of risk and financial management · doi:10.3390/jrfm18030165 · Cited by 32
“In the financial trade context, trust is of vital concern and encompasses various dimensions including trust in the issuing authority, the technology underlying the digital currency and the overall financial system (Zhang et al., 2021).”
LiDAR-based localization using universal encoding and memory-aware regression
Yu, S., Wang, C., Wen, C., Cheng, M., Liu, M., Zhang, Z., & Li, X. S. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108685 · Cited by 39
Trusting our machines: validating machine learning models for single-molecule transport experiments
Bro‐Jørgensen, W., Hamill, J. M., Bro, R., & Solomon, G. C. (2022) · Chemical Society Reviews · doi:10.1039/d1cs00884f · Cited by 34
Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space
Shao, W., Zhao, X., Ge, Y., Zhang, Z., Yang, L., Wang, X., Shan, Y., & Luo, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19830-4_17 · Cited by 52
LSTM autoencoders based unsupervised machine learning for transmission line protection
Rafique, F., Fu, L., & Mai, R. (2023) · Electric Power Systems Research · doi:10.1016/j.epsr.2023.109432 · Cited by 32
Effects of noise on the overparametrization of quantum neural networks
García-Martín, D., Larocca, M., & Cerezo, M. (2024) · Physical Review Research · doi:10.1103/physrevresearch.6.013295 · Cited by 41
“When a neural network is over-parametrized, it has a capacity which is larger than the number of training points [1].”
BackgroundA novel ensemble learning framework based on a genetic algorithm for the classification of pneumonia
Kaya, M., & KAYA, Y. Ç. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108494 · Cited by 32
Speech Imagery Decoding Using EEG Signals and Deep Learning: A Survey
Zhang, L., Zhou, Y., Gong, P., & Zhang, D. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3431224 · Cited by 35
Kallstenius, T., Capusan, A. J., Andersson, G., & Williamson, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-08031-0 · Cited by 34
“Overfitting occurs when a model becomes too closely aligned with the training data, capturing noise or irrelevant details, which reduces its ability to generalize to unseen data 18 .”
Li, F., Zhang, H., Lü, Z., Li, Y., Wei, Y., Li, Z., Bao, F., Zhang, J., Zhou, Y., & Chi, N. (2025) · Opto-Electronic Advances · doi:10.29026/oea.2025.250013 · Cited by 35
KTBoost: Combined Kernel and Tree Boosting
Sigrist, F. (2021) · Neural Processing Letters · doi:10.1007/s11063-021-10434-9 · Cited by 34
Adaptive ensemble of classifiers with regularization for imbalanced data classification
Wang, C., Deng, C., Yu, Z., Hui, D., Gong, X., & Luo, R. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.10.017 · Cited by 31
Wasserstein Adversarial Regularization for Learning With Label Noise
Fatras, K., Damodaran, B. B., Lobry, S., Flamary, R., Tuia, D., & Courty, N. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3094662 · Cited by 42
Low Dimensional Trajectory Hypothesis is True: DNNs can be Trained in Tiny Subspaces
Li, T., Tan, L., Huang, Z., Tao, Q., Liu, Y., & Huang, X. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3178101 · Cited by 63
“Since DNNs are working in the over-parameterization regime, they could easily fit any label, even incorrect or meaningless ones [20].”
BackgroundCompressing Features for Learning With Noisy Labels
Chen, Y., Hu, S. X., Shen, X., Ai, C., & Suykens, J. A. K. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3186930 · Cited by 33
Kim, B., Shim, H., & Baek, J. (2022) · Medical Physics · doi:10.1002/mp.15885 · Cited by 31
Towards Open Set Video Anomaly Detection
Zhu, Y., Bao, W., & Yu, Q. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19830-4_23 · Cited by 62
Adaptive Hierarchical Similarity Metric Learning With Noisy Labels
Yan, J., Luo, L., Deng, C. D., & Huang, H. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3242148 · Cited by 38
“Learning with noisy labels is challenging in deep learning as deep neural networks (DNNs) have the high capacity to memorize the noisy labels [38], [39].”
BackgroundEfficient Augmentation for Imbalanced Deep Learning
Dablain, D. A., Bellinger, C., Krawczyk, B., & Chawla, N. V. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00114 · Cited by 35
Evaluating the Generalization Ability of Super-Resolution Networks
Liu, Y., Zhao, H., Gu, J., Qiao, Y., & Dong, C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3312313 · Cited by 43
“In high-level vision tasks, like classification, they usually use the prediction accuracy on an unseen dataset or category as the generalization measure [15], [16], [17], [18], [19].”
BackgroundDynamic adaptive threshold based learning for noisy annotations robust facial expression recognition
Gera, D., Kumar, B. V. R., Badveeti, N. S. K., & Balasubramanian, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17510-3 · Cited by 32
“These methods need to know the noise rate beforehand and they may confuse hard samples with noisy ones due to small loss assumption [18, 19].”
InfluentialDistilling Large Vision-Language Model with Out-of-Distribution Generalizability
Li, X., Fang, Y., Liu, M., Ling, Z., Tu, Z., & Su, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00236 · Cited by 68
Practical Attribute Reconstruction Attack Against Federated Learning
Chen, C., Lyu, L., Yu, H., & Chen, G. (2024) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2022.3159236 · Cited by 30
“This privacy leakage cannot be evaded even when sensitive information like gender, age, location, and password are not explicitly included in the training data, as modern deep learning model owns more capacity than needed to perform well on their tasks [24].”
BackgroundEnhancing Heterogeneous Knowledge Graph Completion with a Novel GAT-based Approach
Wei, W., Song, Y., & Yao, B. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3639472 · Cited by 31
“In deep learning, overfitting is a fundamental problem characterized by a model that can match the training set with high accuracy but performs poorly when modeling other data.”
BackgroundTwo-path target-aware contrastive regression for action quality assessment
Ke, X., Xu, H., Lin, X., & Guo, W. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120347 · Cited by 39
Procrustean Training for Imbalanced Deep Learning
Ye, H.-J., Zhan, D., & Chao, W. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00016 · Cited by 44
Trustable Co-Label Learning From Multiple Noisy Annotators
Li, S., Liu, T., Tan, J., Zeng, D., & Ge, S. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3137752 · Cited by 40
NLIP: Noise-Robust Language-Image Pre-training
Huang, R., Long, Y., Han, J., Xu, H., Liang, X., Xu, C., & Liang, X. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25172 · Cited by 40
MI-SegNet: Mutual Information-Based US Segmentation for Unseen Domain Generalization
Bi, Y., Jiang, Z., Clarenbach, R., Ghotbi, R., Karlas, A., & Navab, N. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43901-8_13 · Cited by 44
An Efficient and Dynamic Privacy-Preserving Federated Learning System for Edge Computing
Tang, X., Guo, C., Choo, K. R., & Liu, Y. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3320611 · Cited by 33
“An adversary can exploit plaintext gradients or two consecutive snapshots of the model weight to recover information about the sensitive data [6].”
BackgroundHamid, O. H. (2024) · IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA) · doi:10.1109/cogsima61085.2024.10553755 · Cited by 29
“Most notably, the expansive neural networks, often featuring up to a trillion parameters, defy classical statistical expectations related to overfitting [20].”
BackgroundRethinking Deep CNN Training: A Novel Approach for Quality-Aware Dataset Optimization
Rusyn, B., Lutsyk, O., Kosarevych, R., Kapshii, O., Karpin, O., Maksymyuk, T., & Gazda, J. (2024) · IEEE Access · doi:10.1109/access.2024.3414651 · Cited by 33
“In [17], Zhang et al. explored a remarkable phenomenon of deep learning models.”
BackgroundDAR-DRL: A dynamic adaptive routing method based on deep reinforcement learning
Rao, Z., Xu, Y., Yao, Y., & Meng, W. (2024) · Computer Communications · doi:10.1016/j.comcom.2024.107983 · Cited by 35
Qin, B., Yang, Z., Mu, M., Wei, Y., Cui, Y., Fang, X., Dai, G., & Yuan, S. (2024) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.4882 · Cited by 29
“…learns hierarchical representations of multivariate data through neural networks with numerous layers (Bengio et al., 2013; Schmidhuber, 2015; Zhang et al., 2021), bridging a complex nonlinear mapping between historical and future states from simulations, reanalysis, and/or observations.”
Singh, P., Panda, S. S., Dash, J. C., Riscob, B., Pathak, S. K., & Hegde, R. S. (2025) · IEEE journal on multiscale and multiphysics computational techniques · doi:10.1109/jmmct.2025.3544270 · Cited by 31
“The model is trained using a batch size of 1024 samples and an early stopping strategy (with validation error as monitor), having implicit regularization ability [36] is used to control the number of training epochs to prevent over-training.”
An instance-dependent simulation framework for learning with label noise
Gu, K., Masotto, X., Bachani, V., Lakshminarayanan, B., Nikodem, J., & Yin, D. (2022) · Machine Learning · doi:10.1007/s10994-022-06207-7 · Cited by 35
Deep learning, stochastic gradient descent and diffusion maps
Fjellström, C., & Nyström, K. (2022) · Journal of Computational Mathematics and Data Science · doi:10.1016/j.jcmds.2022.100054 · Cited by 33
On the Robustness of Average Losses for Partial-Label Learning
Lv, J., Liu, B., Feng, L., Xu, N., Xu, M., An, B., Niu, G., Geng, X., & Sugiyama, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275249 · Cited by 47
Predicting rare outcomes in abdominal wall reconstruction using image-based deep learning models
Ayuso, S. A., Elhage, S. A., Zhang, Y., Aladegbami, B. G., Gersin, K. S., Fischer, J. P., Augenstein, V. A., Colavita, P. D., & Heniford, B. T. (2022) · Surgery · doi:10.1016/j.surg.2022.06.048 · Cited by 30
General Greedy De-Bias Learning
Han, X., Wang, S., Su, C., Huang, Q., & Tian, Q. C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3240337 · Cited by 37
Dynamic adaptive vehicle re-routing strategy for traffic congestion mitigation of grid network
Wang, C., Atkison, T., & Park, H. (2023) · International Journal of Transportation Science and Technology · doi:10.1016/j.ijtst.2023.04.003 · Cited by 35
Semi-Supervised Modified-UNet for Lung Infection Image Segmentation
Upadhyay, A. K., & Bhandari, A. K. (2023) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2023.3272209 · Cited by 29
“In addition to early stopping, weight decay regularization is also used, which is equivalent to L 2 regularization [64] and it is found effective with Adam optimizer [65].”
MethodGNN Cleaner: Label Cleaner for Graph Structured Data
Xia, J., Lin, H., Xu, Y., Tan, C., Wu, L., Li, S., & Li, S. Z. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3288002 · Cited by 48
On the Robustness of Random Forest Against Untargeted Data Poisoning: An Ensemble-Based Approach
Anisetti, M., Ardagna, C. A., Balestrucci, A., Bena, N., Damiani, E., & Yeun, C. Y. (2023) · IEEE Transactions on Sustainable Computing · doi:10.1109/tsusc.2023.3293269 · Cited by 35
Homogenization of SGD in high-dimensions: exact dynamics and generalization properties
Paquette, C., Paquette, E., Adlam, B., & Pennington, J. (2024) · Mathematical Programming · doi:10.1007/s10107-024-02171-3 · Cited by 49
Convex Formulation of Overparameterized Deep Neural Networks
Fang, C., Gu, Y., Zhang, W., & Zhang, T. (2022) · IEEE Transactions on Information Theory · doi:10.1109/tit.2022.3163341 · Cited by 37
Effective dimension of machine learning modelsPreprint
Abbas, A., Sutter, D., Figalli, A., & Woerner, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.04807 · Cited by 39
“For more details on this transformation, see [2].”
BackgroundMethodInfluentialA new deep learning model combining CNN for engine fault diagnosis
Sönmez, E., Kaçar, S., & Uzun, S. (2023) · Journal of the Brazilian Society of Mechanical Sciences and Engineering · doi:10.1007/s40430-023-04537-8 · Cited by 28
“This increases the number of samples in each fault type approximately 55 times with data augmentation using the sliding window method.”
MethodYadav, A., Hussain, R., Shukla, M., Jayaprakash, B., Kalia, R., Mary, S. P., Hsu, C., Mishra, M. K., Saleem, K., & El‐Meligy, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-93765-0 · Cited by 30
“One primary reason is the scarcity of training data, which can lead the model to over-learn specific characteristics of the limited dataset, hindering its ability to generalize to new, unseen examples 53 .”
Watson, M., Hasan, B. A. S., & Moubayed, N. A. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00159 · Cited by 37
Zhang, Y., Wang, Q., Tian, T., & Yang, Y. (2022) · Resources Policy · doi:10.1016/j.resourpol.2022.102584 · Cited by 28
“In addition, the studies of (Albulescu, 2020; Bourghelle et al., 2021; Jiang et al., 2021; Narayan, 2020; Zhang et al., 2021) empirically Y.”
BackgroundManaging Randomness to Enable Reproducible Machine Learning
Ahmed, H., & Lofstead, J. (2022) · International Workshop on Practical Reproducible Evaluation of Computer Systems (P-RECS) · doi:10.1145/3526062.3536353 · Cited by 36
“Zhang et. al [25] investigate why deep neural network (DNN) models successfully generalize and avoid overfitting to training data.”
BackgroundAutomated detection of schizophrenia using deep learning: a review for the last decade
Sharma, M. D., Patel, R. K., Garg, A., SanTan, R., & Acharya, U. R. (2023) · Physiological Measurement · doi:10.1088/1361-6579/acb24d · Cited by 32
RankDNN: Learning to Rank for Few-Shot Learning
Guo, Q., Haotong, G., Wei, X., Fu, Y., Yu, Y., Zhang, W., & Ge, W. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25150 · Cited by 36
A Novel Ensemble Framework for Multi-Classification of Brain Tumors Using Magnetic Resonance Imaging
KAYA, Y. Ç., & Kaya, M. (2024) · Diagnostics · doi:10.3390/diagnostics14040383 · Cited by 28
“Complex models with more learning parameters than the amount of data show high performance in the training phase due to overfitting; however, they perform poorly on test data they have not seen before [11].”
BackgroundHoang, T., Rana, S., Gupta, S., & Venkatesh, S. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00475 · Cited by 71
Turrisi, R., Verri, A., & Barla, A. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1360095 · Cited by 32
“Although the widespread notion for which deeper neural networks better generalize in a general framework, this result is in line with other studies (Zhang et al., 2021; Vento and Fanfarillo, 2019) in which authors showed that smaller models perform better when only a limited amount of data is…”
ResultSix lectures on linearized neural networks
Misiakiewicz, T., & Montanari, A. (2024) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ad292a · Cited by 37
AdapterGNN: Parameter-Efficient Fine-Tuning Improves Generalization in GNNs
Li, S., Han, X., & Bai, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29264 · Cited by 55
“Prompt tuning (Lester, Al-Rfou, and Constant 2021) (Belkin et al. 2019; Nakkiran et al. 2021; Zhang et al. 2021; Sun et al. 2016; Hardt, Recht, and Singer 2016; Mou et al. 2018).”
BackgroundPredicting the generalization gap in neural networks using topological data analysis
Ballester, R., Clemente, X. A., Casacuberta, C., Madadi, M., Corneanu, C., & Escalera, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2024.127787 · Cited by 23
“One of the most influential papers in the last few years has been [9], in which classical theories on the generalization capabilities of machine learning models were shown to fail to explain why neural networks generalize well in practice.”
BackgroundBeyond supervision: Harnessing self-supervised learning in unseen plant disease recognition
Chai, A. Y. H., Lee, S. H., Tay, F. S., Bonnet, P., & Joly, A. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128608 · Cited by 27
Domain-guided conditional diffusion model for unsupervised domain adaptation
Zhang, Y., Chen, S., Jiang, W., Zhang, Y., Lu, J., & Kwok, J. T.-Y. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107031 · Cited by 35
Physics-informed learning in artificial electromagnetic materials
Deng, Y., Fan, K., Jin, B., Malof, J. M., & Padilla, W. J. (2025) · Applied Physics Reviews · doi:10.1063/5.0232675 · Cited by 29
Mmileng, O. P., Whata, A., Olusanya, M. O., & Mhlongo, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0313734 · Cited by 30
Frequency Principle in Broad Learning System
Chen, G., Gan, M., Chen, C. L. P., Zhu, H., & Chen, L. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3081568 · Cited by 30
Ding, N., Chen, X., Levinboim, T., Changpinyo, S., & Soricut, R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19830-4_15 · Cited by 49
Semi-Supervised CT Lesion Segmentation Using Uncertainty-Based Data Pairing and SwapMix
Qiao, P., Li, H., Song, G., Han, H., Gao, Z., Tian, Y., Liang, Y., Li, X., Zhou, S. K., & Chen, J. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3232572 · Cited by 33
“Existing work has shown that models trained on unreliable data have high generalization errors [11].”
BackgroundSimultaneous Estimation of Hand Configurations and Finger Joint Angles Using Forearm Ultrasound
Bimbraw, K., Nycz, C. J., Schueler, M., Zhang, Z., & Zhang, H. K. (2023) · IEEE Transactions on Medical Robotics and Bionics · doi:10.1109/tmrb.2023.3237774 · Cited by 27
“There is ongoing research in the deep learning community on understanding generalization and its trade-off with model complexity [58].”
Meng, N., Wong, K.-Y. K., Zhao, M., Cheung, J. P. Y., & Zhang, T. G. (2023) · EClinicalMedicine · doi:10.1016/j.eclinm.2023.102050 · Cited by 27
Peivaste, I., Ramezani, S., Alahyarizadeh, G., Ghaderi, R., Makradi, A., & Belouettar, S. (2024) · Scientific Reports · doi:10.1038/s41598-023-50893-9 · Cited by 27
“These aspects play a critical role in determining the performance and generalization capabilities of ML models 50 .”
BackgroundA survey of deep network techniques all classifiers can adopt
Ghods, A., & Cook, D. J. (2020) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-020-00722-8 · Cited by 30
Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation
Wang, K., Muthukumar, V., & Thrampoulidis, C. (2023) · IEEE Transactions on Information Theory · doi:10.1109/tit.2023.3320098 · Cited by 30
Transport Analysis of Infinitely Deep Neural NetworkPreprint
SonodaSho, & MurataNoboru (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.02832 · Cited by 28
Javid-Hosseini, S.-H., Ghazanfarianpoor, P., Nayyeri, V., & Colantonio, P. (2024) · IEEE Transactions on Microwave Theory and Techniques · doi:10.1109/tmtt.2024.3374092 · Cited by 26
“It is important to note that the efficacy of this approach depends on several factors, including the type of neural network, noise characteristics, and the specific task for which the network is trained [34], [35].”
BackgroundLearning from open-set noisy labels based on multi-prototype modeling
Zhang, Y., Chen, Y., Fang, C., Wang, Q., Wu, J., & Xin, J. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110902 · Cited by 32
Robust Multimodal Learning via Representation Decoupling
Wei, S., Luo, Y., Wang, Y., & Luo, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72946-1_3 · Cited by 44
Guideline-Based Additive Explanation for Computer-Aided Diagnosis of Lung Nodules
Zhu, P., & Ogino, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33850-3_5 · Cited by 26
Deep Learning Is Singular, and That’s Good
Wei, S., Murfet, D., Gong, M., Li, H., Gell-Redman, J., & Quella, T. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3167409 · Cited by 26
Cramér–Rao bound‐informed training of neural networks for quantitative MRI
Zhang, X., Duchemin, Q., Liu, K., Gultekin, C., Flassbeck, S., Fernandez‐Granda, C., & Assländer, J. (2022) · Magnetic Resonance in Medicine · doi:10.1002/mrm.29206 · Cited by 29
Chebyshev Pooling: An Alternative Layer for the Pooling of CNNs-Based Classifier
Chan, K., Pau, G., & Im, S. (2021) · IEEE International Conference on Computer and Communication Engineering Technology (CCET) · doi:10.1109/ccet52649.2021.9544405 · Cited by 26
“The learning model can use redundancy to describe random errors or noise, rather than strict underlying data distribution [15].”
MethodQuantum Neural Networks and Topological Quantum Field Theories
Marcianò, A., Chen, D., Fabrocini, F., Fields, C., Greco, E., Gresnigt, N. G., Jinklub, K., Lulli, M., Terzidis, K., & Zappala, E. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.05.028 · Cited by 31
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.