Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 501–600 of 2,667
Robust Contrastive Cross-modal Hashing with Noisy Labels
Wang, L. E. I., Qin, Y., Sun, Y., Peng, D., Peng, X., & Hu, P. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680564 · Cited by 36
Chat-based Person Retrieval via Dialogue-Refined Cross-Modal Alignment
Bai, Y., Ji, Y., Cao, M., Wang, J., & Ye, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00374 · Cited by 20
“Although LLMs facilitate the dialogue data collection, the generated data inevitably contains noise, which would significantly impair the optimization of the retrieval process [57].”
Comparison of different ML methods concerning prediction quality, domain adaptation and robustness
Goodarzi, P., Schütze, A., & Schneider, T. (2022) · tm - Technisches Messen · doi:10.1515/teme-2021-0129 · Cited by 24
Temporal deep learning framework for retinopathy prediction in patients with type 1 diabetes
Rabhi, S., Blanchard, F., Diallo, A. M., Zeghlache, D., Lukas, C., Berot, A., Delemer, B., & Barraud, S. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102408 · Cited by 22
Bootstrapping the Relationship Between Images and Their Clean and Noisy Labels
Smart, B., & Carneiro, G. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00531 · Cited by 31
Highway icing time prediction with deep learning approaches based on data from road sensors
Wang, S., Wang, T., Pei, X., Wang, H., Zhu, Q., Tang, T., & Hou, T. (2023) · Science China Technological Sciences · doi:10.1007/s11431-022-2230-8 · Cited by 20
“…vector classification (SVC) to judge whether the pavement would freeze in a certain time in Nowadays, deep learning, a multilayer neural network method better than traditional machine learning methods in the generalization ability [19–21], can theoretically predict icing events more accurately.”
MethodA Survey of Dataset Refinement for Problems in Computer Vision Datasets
Wan, Z., Wang, Z., Chung, C., & Wang, Z. (2023) · ACM Computing Surveys · doi:10.1145/3627157 · Cited by 32
“Since deep neural networks have the capacity to essentially memorize any characteristics of the data [169], those problematic data can drastically inhibit the performance of model training.”
BackgroundFedSW: Federated learning with adaptive sample weights
Zhao, X., & Shen, D. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119873 · Cited by 22
Li, K., Chen, H., Wan, J., & Yu, S. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3376197 · Cited by 21
“Due to the high capacity of DNNs of memorizing the entire training data [7], decision boundaries transferred from the old model overfit the limited exemplars [2], [6], decreasing the effectiveness of KD in retaining long-term knowledge.”
BackgroundZhao, C., Zhao, J., Wang, W., Yuan, C., & Tang, J. (2024) · Ore Geology Reviews · doi:10.1016/j.oregeorev.2024.106001 · Cited by 20
Open-Set Domain Adaptation via Feature Clustering and Separation for Fault Diagnosis
Wang, X., Shi, Z., Sun, S., & Li, L. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3381929 · Cited by 20
Crusius, D., Cipcigan, F. S., & Biggin, P. C. (2024) · Faraday Discussions · doi:10.1039/d4fd00091a · Cited by 28
Cordeiro, F. R., & Carneiro, G. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111132 · Cited by 26
“In fact, Zhang et al. [7] showed that deep learning models can easily memorize label noise, and this process is independent of the model architecture or optimization method [8].”
MethodLoss Spike in Training Neural Networks
Li, X., Xu, Z.-Q. J., & Zhang, Z. (2025) · Journal of Computational Mathematics · doi:10.4208/jcm.2412-m2024-0083 · Cited by 22
Robust color medical image segmentation on unseen domain by randomized illumination enhancement
Zhang, Z., Li, Y., & Shin, B. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105427 · Cited by 21
Spectral classification by generative adversarial linear discriminant analysis
Cao, Z., Zhang, S., Liu, Y., Smith, C. J., Sherman, A. M., Hwang, Y., & Simpson, G. J. (2023) · Analytica Chimica Acta · doi:10.1016/j.aca.2023.341129 · Cited by 21
Extended multiphysics-informed neural network for conjugate heat transfer problems
Lee, J., Shin, S., Choi, H. J., Lee, A., Park, B., & Lee, S. (2025) · International Journal of Heat and Mass Transfer · doi:10.1016/j.ijheatmasstransfer.2025.127098 · Cited by 20
Xia, G., Ghrairi, Z., Heuermann, A., & Thoben, K. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.06.022 · Cited by 20
Abbas, S. K., Hussain, M., & Rimal, Y. (2025) · SN Computer Science · doi:10.1007/s42979-025-04214-8 · Cited by 21
“For instance, Zhang et al. [31] emphasized the role of algorithmic competence and benevo-lence in shaping trust, while Chen et al. [32] linked trust and loyalty to the continued use of robo-advisory services [33, 34].”
Correntropy meets cross-entropy: A robust loss against noisy labels
Zhou, N., Deng, Q., Luo, W., Huang, X., Du, Y., Chen, B., & Witold Pedrycz, C. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.113830 · Cited by 26
Automatic annotation correction for wearable EEG based epileptic seizure detection
Zhang, J., Chatzichristos, C., Vandecasteele, K., Swinnen, L., Broux, V., Cleeren, E., Paesschen, W. V., & Vos, M. D. (2022) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac54c1 · Cited by 22
“Even if some methods are considered to be robust to the imperfect annotations [36, 37], overfitting could not be avoided owing to their large network capacity and learning ability [19, 20].”
Gaussian and non-Gaussian universality of data augmentation
Huang, K. H., Orbanz, P., & Austern, M. (2026) · The Annals of Statistics · doi:10.1214/26-aos2642 · Cited by 20
Representation learning from noisy user-tagged data for sentiment classification
Chen, L., Wang, F., Yang, R., Xie, F., Wang, W., Xu, C. X., Zhao, W. Z., & Guan, Z. (2022) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01622-7 · Cited by 19
“Deep learning models will eventually memorize these wrongly given labels, which leads to the poor generalization performance [18].”
BackgroundCOVID-19 chest X-ray image classification in the presence of noisy labels
Xiao-qing, Y. I. N. G., Liu, H., & Huang, R. (2023) · Displays · doi:10.1016/j.displa.2023.102370 · Cited by 20
Riedl, K., Klock, T., Geldhauser, C., & Fornasier, M. (2026) · SIAM Journal on Mathematics of Data Science · doi:10.1137/25m1733409 · Cited by 24
A New Data Augmentation Method Based on Mixup and Dempster-Shafer Theory
Zhang, Z., Wang, H., Geng, J., Deng, X., & Jiang, W. R. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3330106 · Cited by 19
“It has been found that ERM allows deep neural networks to memorize training data [20].”
BackgroundPerovskite-based optoelectronic systems for neuromorphic computing
Cao, Y., Yin, L., Zhao, C., Zhao, T., Li, T., Kong, S., Shi, L., Zhou, J., Zhang, Z., Ke, Y., et al. (2023) · Nano Energy · doi:10.1016/j.nanoen.2023.109169 · Cited by 19
Contactless Pulse Estimation Leveraging Pseudo Labels and Self-Supervision
Li, Z., & Yin, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01882 · Cited by 32
“Likewise, ϵ The networks tend to learn simple and generalized patterns at the starting epochs [1, 54, 58, 59].”
BackgroundDeepfake Detection Fighting Against Noisy Label Attack
Qiao, T., Xie, S., Chen, Y., Retraint, F., Shi, R., & Luo, X. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3385286 · Cited by 21
“Meanwhile, in such scenario, the DNNs based clas-sifiers are probably unreliable for Deepfake detection [23], [24], [25].”
MethodBotCL: a social bot detection model based on graph contrastive learning
Yan, L., Li, Z., Gong, D., Hu, Q., & Lu, H. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02116-4 · Cited by 23
Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding
Shah, S. B., Pradhan, P., Pu, W., Randhi, R., Rodrigues, M. R. D., & Eldar, Y. C. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3412981 · Cited by 22
Improving acoustic species identification using data augmentation within a deep learning framework
MacIsaac, J., Newson, S. E., Ashton‐Butt, A., Pearce, H., & Milner, B. P. (2024) · Ecological Informatics · doi:10.1016/j.ecoinf.2024.102851 · Cited by 21
Conservative SPDEs as fluctuating mean field limits of stochastic gradient descent
Gess, B., Gvalani, R. S., & Konarovskyi, V. (2025) · Probability Theory and Related Fields · doi:10.1007/s00440-024-01353-6 · Cited by 32
Wang, H., Wu, Z., & He, J. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635844 · Cited by 33
“In Section 5.3, we showcase the difference in classification accuracy between the original and the retrained models remains minimal when the model converges.”
BackgroundMethodInfluentialRobust Bayesian attention belief network for radar work mode recognition
Du, M., Zhong, P., Cai, X., Bi, D., & Jing, A. (2022) · Digital Signal Processing · doi:10.1016/j.dsp.2022.103874 · Cited by 18
“The early stopping mechanism may improve its generalization [49] but to a very limited extent as shown in Fig.”
Skellam mixture mechanism: a novel approach to federated learning with differential privacy
Bao, E., Zhu, Y., Xiao, X., Yang, Y., Ooi, B. C., Tan, B. H. M., & Aung, K. M. M. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3551793.3551798 · Cited by 18
“Deep neural networks, especially large-scale ones such as GPT3 [11], are known for their excellent memorization capabilities [25, 47, 56].”
BackgroundSSVMR: Saliency-Based Self-Training for Video-Music Retrieval
Cheng, X., Zhu, Z., Li, H., Li, Y., & Zou, Y. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096514 · Cited by 26
“As mentioned in [14, 15], noisy samples usually have higher losses in the early stage.”
BackgroundKorbar, J., Ocepek, D., Čepon, G., & Boltežar, M. (2023) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2023.110426 · Cited by 18
“As Zhang et al. have shown, a sufficiently large neural network is capable of learning any dataset perfectly, even if the labels are assigned randomly [41].”
The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictions
Meyer, A. P., Albarghouthi, A., & D’Antoni, L. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593988 · Cited by 33
“Checking robustness to dataset multiplicity has parallels to adversarial machine learning, especially data poisoning, where an attacker modifies a small portion of the training dataset to reduce test-time accuracy [3, 52, 66, 68].”
BackgroundEvaluating and comparing bagging and boosting of hybrid learning for breast cancer screening
Zizaan, A., & Idri, A. (2023) · Scientific African · doi:10.1016/j.sciaf.2023.e01989 · Cited by 18
Knockoffs-SPR: Clean Sample Selection in Learning With Noisy Labels
Wang, Y., Fu, Y., & Sun, X. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3338268 · Cited by 23
Rethinking precision of pseudo label: Test-time adaptation via complementary learning
Zeng, L., Han, J., Du, L., & Ding, W. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.12.001 · Cited by 19
Assessing deep learning: a work program for the humanities in the age of artificial intelligence
Segessenmann, J., Stadelmann, T., Davison, A., & Dürr, O. M. (2023) · AI and Ethics · doi:10.1007/s43681-023-00408-z · Cited by 24
“…explain the success of DL to a large extent [133] and gave rise to numerous algorithmic advances, which account for another large part [110], a unified theory that fully justifies the remarkable performance of DL models is still missing [17, 22, 23], although progress seems to be made [134, 135].”
Driving Behavior Prediction Based on Combined Neural Network Model
Li, R., Shu, X., & Li, C. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3350199 · Cited by 18
“Considering the feature of accuracy and computational cost with LSTM-based baseline model [18] and the ability of each vehicle around the target vehicle modeled by LSTM to extract the temporal characteristics of their historical trajectories, LSTM neural networks is chosen in this article [19].”
MethodUnlocking the Power of Open Set: A New Perspective for Open-Set Noisy Label Learning
Wan, W., Wang, X., Xie, M.-K., Li, S.-Y., Huang, S.-J., & Chen, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29469 · Cited by 30
Low-Rank Optimal Transport for Robust Domain Adaptation
Xu, B., Yin, J., Lian, C., Su, Y., & Zeng, Z. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124344 · Cited by 18
Source-free cross-domain fault diagnosis of rotating machinery using the Siamese framework
Ma, C., Tu, X., Zhou, G., Huang, Y., & Ding, X. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112179 · Cited by 20
An analytic formulation of convolutional neural network learning for pattern recognition
Zhuang, H., Lin, Z., Yang, Y., & Toh, K. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121317 · Cited by 18
Learning Discriminative Dynamics with Label Corruption for Noisy Label Detection
Kim, S. Y., Lee, D., Kang, S., Chae, S., Jang, S., & Yu, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02121 · Cited by 28
“Unfortunately, these approaches have led to the emergence of noise in real-world datasets, with reported noise rates ranging from 8.0% to 38.5% [25, 27, 54], which severely degrades the model’s performance [1, 58].”
BackgroundInfluentialZhu, X., Sun, S., Lin, L., Wu, Y., & Ma, X. (2024) · Reviews in the Neurosciences · doi:10.1515/revneuro-2024-0088 · Cited by 21
Trained without My Consent: Detecting Code Inclusion in Language Models Trained on Code
Majdinasab, V., Nikanjam, A., & Khomh, F. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3702980 · Cited by 21
“To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.”
BackgroundMavridis, C., Vagenas, T. P., Economopoulos, T. L., Vezakis, I., Πετροπούλου, Ο., Kakkos, I., & Matsopoulos, G. K. (2024) · Electronics · doi:10.3390/electronics13244919 · Cited by 23
Fast binary logistic regression
Saran, A. N., & Nar, F. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2579 · Cited by 18
Weight decay induces low-rank attention layers
Kobayashi, S., Akram, Y., & von Oswald, J. (2024) · neural information processing systems · doi:10.52202/079017-0146 · Cited by 51
CoSQA + : Enhancing Code Search Evaluation With a Multi-Choice Benchmark and Test-Driven Agents
Gong, J., Wu, Y., Liang, L., Wang, Y., Chen, J., Liu, M., & Zheng, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3631886 · Cited by 25
Label Noise Modeling and Correction via Loss Curve Fitting for SAR ATR
Wang, C., Shi, J., Zhou, Y., Li, L., Yang, X., Zhang, T., Wei, S., Zhang, X., & Tao, C. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3121397 · Cited by 24
Implicit Kinematic Policies: Unifying Joint and Cartesian Action Spaces in End-to-End Robot Learning
Ganapathi, A., Florence, P., Varley, J., Burns, K., Goldberg, K. Y., & Zeng, A. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812165 · Cited by 23
Zhang, X., Cao, X., Zhang, P., Song, F., Zhang, J., Zhang, L., & Zhang, G. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3167809 · Cited by 18
“Therefore, the diversity and quantity of the data set have a great influence on the generalization ability of neural networks [49].”
BackgroundAdaptive sequencing using nanopores and deep learning of mitochondrial DNA
Danilevsky, A., Polsky, A. L., & Shomron, N. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac251 · Cited by 21
Guided by Meta-Set: A Data-Driven Method for Fine-Grained Visual Recognition
Zhang, C., Lin, G., Wang, Q., Shen, F., Yao, Y., & Tang, Z. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3181439 · Cited by 19
Self-citationJalota, S., & Suthar, M. (2023) · Soft Computing · doi:10.1007/s00500-023-08339-x · Cited by 19
“…associated with the Marshall mix design such as: voids in the mineral aggregates (VMA), voids filled with bitumen (VFB), air voids, theoretical and observed specific gravity of the bitumen mix have to be worked out analytically by trial and error process requiring human effort (Zhang et al. 2021).”
Energy-Efficient Spiking-CNN-Based Cross-Patient Seizure Detection
Muneeb, A., & Kassiri, H. (2023) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas46773.2023.10181879 · Cited by 23
DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networks
Jain, S., Addepalli, S., Sahu, P. K., Dey, P., & Babu, R. V. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01540 · Cited by 33
“The overparameterization of Deep networks leads to the existence of multiple optimal solutions to any given loss function [33, 45, 76].”
BackgroundFlat minima generalize for low-rank matrix recovery
Ding, L., Drusvyatskiy, D., Fazel, M., & Harchaoui, Z. (2024) · Information and Inference A Journal of the IMA · doi:10.1093/imaiai/iaae009 · Cited by 42
Stochastic collapse: how gradient noise attracts SGD dynamics towards simpler subnetworks*
Chen, F., Kunin, D., Yamamura, A., & Ganguli, S. (2024) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ad6429 · Cited by 28
“Contrary to traditional statistical learning theory, these highly expressive models exhibit impressive generalization capabilities on natural tasks, even without explicit regularization, despite having the capacity to memorize random data [1].”
Applying AI in Practice: Key Challenges and Lessons Learned
Fischer, L., Ehrlinger, L., Geist, V., Ramler, R., Sobieczky, F., Zellinger, W., & Moser, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-57321-8_25 · Cited by 21
A Close Look into the Calibration of Pre-trained Language Models
Chen, Y., Yuan, L., Cui, G., Liu, Z., & Ji, H. (2022) · arXiv · doi:10.48550/arxiv.2211.00151 · Cited by 18
An efficient catalyst screening strategy combining machine learning and causal inference
Song, C., Shi, Y., Meng, L., Wu, L., Xiong, X., Liu, J., & Xia, D. (2025) · Journal of Environmental Management · doi:10.1016/j.jenvman.2025.124665 · Cited by 18
A small object detection algorithm for mine environment
Liu, D., Zhao, X., & Fan, W. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110936 · Cited by 17
Shah, J., Alam, M., Javed, F., Umar, M., & Ahmad, F. (2025) · Scientific Reports · doi:10.1038/s41598-025-13926-z · Cited by 22
“Additionally, the training process dynamics plays a crucial role in determining overall model performance 49,50 .”
Uncertainty Modeling for Robust Domain Adaptation Under Noisy Environments
Zhuo, J., Wang, S., & Huang, Q. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3205457 · Cited by 21
Review: 2D material property characterizations by machine-learning-assisted microscopies
Si, Z., Zhou, D., Yang, J., & Lin, X. (2023) · Applied Physics A · doi:10.1007/s00339-023-06543-y · Cited by 19
Combining machine learning and molecular simulations to predict the stability of amorphous drugs
Barnard, T., & Sosso, G. C. (2023) · The Journal of Chemical Physics · doi:10.1063/5.0156222 · Cited by 18
Chaotic neural network quantization and its robustness against adversarial attacks
Osama, A., Gadallah, S. I., Said, L. A., Radwan, A. G., & Fouda, M. E. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111319 · Cited by 18
Dufumier, B., Gori, P., Petiton, S., Louiset, R., Mangin, J., Grigis, A., & Duchesnay, É. (2024) · NeuroImage · doi:10.1016/j.neuroimage.2024.120665 · Cited by 23
Aliferis, C., & Simon, G. (2024) · Health informatics · doi:10.1007/978-3-031-39355-6_12 · Cited by 19
Don, A. K. K., & Khalil, I. (2024) · ACM Transactions on Quantum Computing · doi:10.1145/3660647 · Cited by 23
“Model overfitting causes deep learning models to become excessively reliant on the training data, reducing generalizability [ 64 ].”
Liu, K., Wei, J., Zou, J., Wang, P., Yang, Y., & Shen, H. T. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3410133 · Cited by 22
Prediction of Ionospheric Electron Density Distribution Based on CNN-LSTM Model
Mao, S., Li, H., Zhang, Y., & Shi, Y. (2024) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2024.3437650 · Cited by 17
“…of layers modeled for the same dataset tends to be positively correlated with model complexity [15], which affects the model performance on this batch of datasets, as well as training time [16], computational resources, risk of overfitting [17], generalization ability [18], and other factors.”
BackgroundMalekjani, N., Kharaghani, A., & Tsotsas, E. (2025) · Chemical Engineering Science · doi:10.1016/j.ces.2025.121214 · Cited by 20
Chen, S., Peng, C., Chen, L., & Hao, K. (2025) · ISA Transactions · doi:10.1016/j.isatra.2025.01.041 · Cited by 17
Hasan, T. M., Abdar, M., Khosravi, A., Aickelin, U., Lió, P., Hossain, I., Rahman, A., & Nahavandi, S. (2025) · ACM Computing Surveys · doi:10.1145/3727633 · Cited by 25
“Furthermore, DNNs can also memorize the entire real dataset [111].”
Dunkin, F., Li, X., Zhang, Z., Wang, K., Gao, T., Wu, G., & Li, Z. (2025) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2025.3567264 · Cited by 17
“(cid:4) When the noise level becomes substantial, the training process is driven away from the underlying data distribution, leading to misplaced classification boundaries and degraded generalization [10].”
Deep Unlearn: Benchmarking Machine Unlearning for Image Classification
Cadet, X. F., Borovykh, A., Malekzadeh, M., Ahmadi‐Abhari, S., & Haddadi, H. (2025) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp63326.2025.00058 · Cited by 27
An Ensemble Model for Combating Label Noise
Lu, Y., Yang, B., & He, W. (2022) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3488560.3498376 · Cited by 20
“However, as noisy label ˆ 𝒚 𝑖 is likely to be wrong, the model gradually memorizes the training samples with wrong labels when optimizing 𝑐𝑒 .”
BackgroundA Novel Generative Convolutional Neural Network for Robot Grasp Detection on Gaussian Guidance
Li, Y., Liu, Y., Ma, Z., & Huang, P. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3203118 · Cited by 24
“145 The deep learning method has been shown to be effective for 146 a wide range of perceptual problems [15], [16], which allows 147 the perceptual system to learn mappings from some feature 148 sets to various visual features.”
MethodEFFECT OF LABEL NOISE IN SEMANTIC SEGMENTATION OF HIGH RESOLUTION AERIAL IMAGES AND HEIGHT DATA
Maiti, A., Elberink, S. O., & George Vosselman, A. (2022) · ISPRS annals of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprs-annals-v-2-2022-275-2022 · Cited by 17
“Moreover, overparameterized complex models are susceptible to memorization instead of learning generalization (Zhang et al., 2021).”
BackgroundLiu, Y., & Zhang, L. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3199028 · Cited by 19
Qi, Y., Qiu, M.-Z., Jing, H., Wang, Z.-Q., Yu, C.-L., Zhu, J., Wang, F., & Wang, T. (2022) · Ceramics International · doi:10.1016/j.ceramint.2022.08.033 · Cited by 20
Comparison of Different Features and Neural Networks for Predicting Industrial Paper Press Condition
Rodrigues, J. A., Farinha, J. T., Mendes, M., Mateus, R. J. G., & Cardoso, A. J. M. (2022) · Energies · doi:10.3390/en15176308 · Cited by 17
MAN: Mining Ambiguity and Noise for Facial Expression Recognition in the Wild
Zhang, Z., Sun, X., Li, J., & Wang, M. (2022) · Pattern Recognition Letters · doi:10.1016/j.patrec.2022.10.016 · Cited by 19
Rokem, A., Wu, Y., & Lee, A. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/196659 · Cited by 19
On the Optimal Memorization Power of ReLU Neural NetworksPreprint
Vardi, G., Yehudai, G., & Shamir, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03187 · Cited by 22
“…function and data samples (see e.g. Huang and Babri (1998); Huang (2003); Baum (1988); Vershynin (2020); Daniely (2019; 2020); Bubeck et al. (2020); Park et al. (2020); Hardt and Ma (2016); Yun et al. (2019); Zhang et al. (2021); Nguyen and Hein (2018); Rajput et al. (2021); Sontag (1997)).”
BackgroundInterpreting Single-Cell and Spatial Omics Data Using Deep Neural Network Training Dynamics
Karin, J., Mintz, R., Raveh, B., & Nitzan, M. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00721-5 · Cited by 16
An Efficient Algorithm for Generating Interactive Visualization of Large Datasets
Sharma, C., Chilluri, V. S. B., Kumar, A., & Goyal, S. (2025) · International Conference on Advances in Computer Science, Electrical, Electronics, and Communication Technologies (CE2CT) · doi:10.1109/ce2ct64011.2025.10941393 · Cited by 16
“Instances of Network information remember collaborations among individuals for interpersonal organizations, availability of pages in Internet diagrams, and protein-protein interaction (PPI) in natural organizations [18].”
Ding, C., Wang, S., Wang, Y., Wu, Z., Sun, J., & Mao, Y. (2023) · Physical Review A · doi:10.1103/physreva.107.062422 · Cited by 16
Liu, Z., Chen, Y., Song, G., Song, W., & Xu, J. (2023) · Mathematics · doi:10.3390/math11194147 · Cited by 16
The deep neural network solver for B-spline approximation
Wen, Z., Luo, J., & Kang, H. (2023) · Computer-Aided Design · doi:10.1016/j.cad.2023.103668 · Cited by 19
Single-Shot Pruning for Pre-trained Models: Rethinking the Importance of Magnitude Pruning
Kohama, H., Minoura, H., Hirakawa, T., Yamashita, T., & Fujiyoshi, H. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00155 · Cited by 24
Montin, E., Deniz, C. M., Kijowski, R., Youm, T. J., & Lattanzi, R. (2024) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101444 · Cited by 17
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.