Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 601–700 of 2,667
Searching to Exploit Memorization Effect in Deep Learning With Noisy Labels
Yang, H., Yao, Q., Han, B., & Kwok, J. T.-Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3394552 · Cited by 23
Methodology for Evaluating the Generalization of ResNet
Du, A., Qing, Z., & Dai, Y. (2024) · Applied Sciences · doi:10.3390/app14093951 · Cited by 17
Choudhary, P. S., & Dandapat, S. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3403205 · Cited by 17
“Such an approach often leads to limited generalizability of the model [57].”
BackgroundTransformer-Based Weakly Supervised Learning for Whole Slide Lung Cancer Image Classification
An, J., Wang, Y., Cai, Q., Zhao, G., Dooper, S., Litjens, G. J. S., & Gao, Z. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3425434 · Cited by 18
“However, due to the weak labels, if we directly match the predictions to noisy token labels, the deep model may even fit noisy samples due to strong fitting capabilities [39].”
Baglioni, P., Pacelli, R., Aiudi, R., Renzo, F. D., Vezzani, A., Burioni, R., & Rotondo, P. (2024) · Physical Review Letters · doi:10.1103/physrevlett.133.027301 · Cited by 26
“Introduction— Understanding how deep learning [1] works, given the enormous success of its technological applications in the last decade [2–4], provides a timely challenge to theorists working in physics, mathematics and computer science [5, 6].”
BackgroundJalota, S., & Suthar, M. (2024) · International Journal of Transportation Science and Technology · doi:10.1016/j.ijtst.2024.08.001 · Cited by 16
Data augmentation with attention framework for robust deepfake detection
Mamarasulov, S., Chen, L., Chen, C., Li, Y., & Wang, C. (2024) · The Visual Computer · doi:10.1007/s00371-024-03690-y · Cited by 17
“Augmentation as Regularization Beyond diversifying training data, augmentation plays a key role in model regularization, promoting broader generalization [23–26].”
Unsupervised ground-roll attenuation via implicit neural representations
Li, J., Liu, D., & Sacchi, M. D. (2024) · Geophysics · doi:10.1190/geo2024-0148.1 · Cited by 16
Full error analysis of the random deep splitting method for nonlinear parabolic PDEs and PIDEs
Neufeld, A., Schmocker, P., & Wu, S. (2024) · Communications in Nonlinear Science and Numerical Simulation · doi:10.1016/j.cnsns.2024.108556 · Cited by 16
Coding schemes in neural networks learning classification tasks
van Meegen, A., & Sompolinsky, H. (2025) · Nature Communications · doi:10.1038/s41467-025-58276-6 · Cited by 23
Deep Component Analysis via Alternating Direction Neural Networks
Murdock, C., Chang, M.-F., & Lucey, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_50 · Cited by 17
Task-Specific Loss for Robust Instance Segmentation With Noisy Class Labels
Yang, L., Li, H., Meng, F., Wu, Q., & Ngan, K. N. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3109084 · Cited by 20
Yang, S., Wang, G., Sun, H., Luo, X., Sun, P., Li, K., Wang, Q., & Zhang, S. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3172978 · Cited by 18
A Dataset Auditing Method for Collaboratively Trained Machine Learning Models
Huang, Y., Huang, C.-Y., Li, X., & Li, K. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3220706 · Cited by 23
“ML models such as DNNs are often overparameterized, which means that they have sufficient capacity to memorize information about their training datasets [10], [11], [12].”
BackgroundA Synergic Approach of Deep Learning towards Digital Additive Manufacturing: A Review
Pratap, A., Sardana, N., Utomo, S., Ayeelyan, J., Karthikeyan, P., & Hsiung, P. (2022) · Algorithms · doi:10.3390/a15120466 · Cited by 16
Stability-Based Generalization Analysis of the Asynchronous Decentralized SGD
Deng, X., Sun, T. S., Li, S., & Li, D. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25894 · Cited by 26
“In practice, the solution founded by SGD has not only very small training error but also generalizes surprisingly well to the test data (Zhang et al. 2021).”
BackgroundSoltanolkotabi, M., Stöger, D., & Xie, C. (2025) · IEEE Transactions on Information Theory · doi:10.1109/tit.2025.3530335 · Cited by 25
Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset DistillationPreprint
Loo, N., Hasani, R., Lechner, M., Amini, A., & Rus, D. L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.01428 · Cited by 17
“Neural networks have been shown to perform well and even generalize on a range of tasks, despite achieving zero loss on training data (Zhang et al., 2017; 2021).”
BackgroundAssessing the performance of QSP models: biology as the driver for validation
Singh, F. A., Afzal, N., Smithline, S. J., & Thalhauser, C. J. (2024) · Journal of Pharmacokinetics and Pharmacodynamics · doi:10.1007/s10928-023-09871-x · Cited by 15
“Belkin’s observation is consistent with the Universal Approximation Theorem [40], which states that a neural network can approximate any function sufficiently close given enough hidden layers and a deterministic relation between the input variables and the target function.”
Mutual Information-Based Generalisation Gap Analysis Using Deep Learning Model
Bhuyan, H. K., Unhelkar, B., Shankar, S. S., & Chakrabarti, P. (2024) · Journal of Information & Knowledge Management · doi:10.1142/s0219649225500017 · Cited by 15
Mai, Z., Zhang, P., Tu, C.-H., Chen, H.-Y., Nguyen, Q.-H., Zhang, L., & Chao, W. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01383 · Cited by 41
“But in-terestingly, PEFT methods, with only 2% 5% of tunable parameters, achieve similar accuracy, suggesting that its de-sign principle does offer sufficient effective capacity for the model to learn [105].”
Optimized preprocessing and Tiny ML for Attention State Classification
Wang, Y., Nahon, R., Tartaglione, E., Mozharovskyi, P., & Nguyen, V.-T. (2023) · IEEE Statistical Signal Processing Workshop (SSP) · doi:10.1109/ssp53291.2023.10207930 · Cited by 16
“We also use a deeper network, DNN 6, that consists of DNN 4 with two supplementary internal layers (of 128 neurons) and two dropouts layers before the last hidden layer to address the increased risk of overfitting known to be inherent to larger neural networks [21].”
MethodTeacher–student complementary sample contrastive distillation
Bao, Z., Huang, Z., Gou, J., Du, L., Liu, K., Zhou, J., & Chen, Y. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.11.036 · Cited by 16
Application of machine learning regression models to inverse eigenvalue problems
Pallikarakis, N., & Ntargaras, A. (2023) · Computers & Mathematics with Applications · doi:10.1016/j.camwa.2023.11.038 · Cited by 18
“In general, using a much larger amount of nodes than the size of training samples, can lead to overfitting [54].”
BackgroundFeng, J., He, Y., Pan, Y., Zhou, Z., Chen, S., & Gong, W. (2024) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2024.3352239 · Cited by 17
AutoSMC: An Automated Machine Learning Framework for Signal Modulation Classification
Wang, Y., Bai, J., Xiao, Z. Q., Chen, Z., Xiong, Y., Jiang, H., & Jiao, L. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3414249 · Cited by 15
“Zhang et al. [17] confirmed that the neural network can still fit the dataset with noisy labels, but the generalization performance of the network will be reduced.”
BackgroundThe role of objective sleep in implicit and explicit affect regulation: A comprehensive review
Straus, L. D., ten Brink, M., Sikka, P., Srivastava, R., Gross, J. J., & Colvonen, P. J. (2024) · Neurobiology of Stress · doi:10.1016/j.ynstr.2024.100655 · Cited by 16
“…and no regulation conditions (i.e., reacting naturally), or the effectiveness of using affect regulation strategies before and after sleep, will help clarify the role of sleep in affect regulation and vice-versa (e.g., see Vandekerckhove et al., 2012; Zhang et al., 2019; Zhang et al., 2021).”
BackgroundAddressing label noise for electronic health records: insights from computer vision for tabular data
Yang, J., Triendl, H., Soltan, A. A. S., Prakash, M., & Clifton, D. A. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02581-5 · Cited by 20
“Similar to previous studies (38; 42; 55), we noticed that deep neural networks, when trained without these noise-mitigation methods, have the capability to memorize random and noisy labels and exhibit poor generalization on unseen test sets (see Supplementary Table 6 and Figure 1).”
BackgroundResultHaddouti, S. E., Khaldoune, M., Ayache, M., & Kettani, M. D. E.-C. E. (2024) · Computing · doi:10.1007/s00607-024-01314-w · Cited by 16
“Additionally, the large number of trainable parameters increases the risk of overfitting, particularly in scenarios involving small or noisy datasets [30].”
Digital twin-based non-destructive testing method for ultimate load-carrying capacity prediction
Huang, L., Xu, Z., Gao, T., Liu, X., Bi, Q., Wang, B., & Tian, K. (2024) · Thin-Walled Structures · doi:10.1016/j.tws.2024.112223 · Cited by 17
Classifying Brain Tumours: A Deep Learning Approach with Explainable AI
Lin, L. P., & Seow, Z. H. (2024) · International Conference on Biomedical Engineering and Technology (ICBET) · doi:10.1145/3678935.3678953 · Cited by 15
“Training accuracy gauges a model’s ability to learn and predict accurately from training data, while training loss represents the disparity between predicted and true output [20].”
Gatheeshgar, P., Ranasinghe, R. S. S., Simwanda, L., Meddage, D. P. P., & Mohotti, D. (2024) · Structures · doi:10.1016/j.istruc.2024.108079 · Cited by 16
Unsupervised feature selection and class labeling for credit card fraud
Kennedy, R. K. L., Villanustre, F., & Khoshgoftaar, T. M. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01154-1 · Cited by 15
“The quality of the labels is an important factor for high performing machine learning models as noisy or inaccurately labeled data can severely impact the effectiveness and reliability of classification models [2].”
Benchmarking machine learning methods for the identification of mislabeled data
Nazaretyan, L., Leser, U., & Kircher, M. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11293-9 · Cited by 16
“They tend to (over)fit the entire training set including the corrupted samples and have eventually lower generalization power (Zhang et al. 2021; Algan and Ulusoy 2021; Ghiassi et al. 2023 ).”
DA-Flow: Dual Attention Normalizing Flow for Skeleton-Based Video Anomaly Detection
Wu, R., Chen, Y., Xiao, J. T., Li, B., Fan, J., Frederic Dufaux, A., Zhu, C., & Liu, Y. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3607708 · Cited by 15
“In addition, models with a smaller number of parameters have better generalization ability [5] and the capacity to better cope with unbalanced samples [6].”
Multi-domain semantic segmentation with overlapping labels
Bevandić, P., Oršić, M., Grubišić, I., Šarić, J., & Šegvić, S. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00248 · Cited by 21
Big Data and Deep Learning‐Based Video Classification Model for Sports
Wang, L., Zhang, H., & Yuan, G. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/1140611 · Cited by 15
“As shown in Figure 5, firstly, the free gymnastics decomposition action dataset is constructed; for the video data, feature extraction is performed by the convolutional neural network, and for the labeled text, a dictionary is constructed to extract the corresponding dictionary features; Figure 5 shows basic flow char…”
MethodHarmonizing Pathological and Normal Pixels for Pseudo-Healthy Synthesis
Zhang, Y., Lin, X., Zhuang, Y., Sun, L., Huang, Y., Ding, X., Wang, G., Yang, L., & Yu, Y. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3164095 · Cited by 15
Boosting Robust Learning Via Leveraging Reusable Samples in Noisy Web Data
Sun, Z., Yao, Y., Wei, X.-S., Shen, F., Zhang, J., & Hua, X. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3158001 · Cited by 15
Symmetry-based regularization in deep breast cancer screening
Castro, E., Pereira, J. C., & Cardoso, J. S. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102690 · Cited by 17
Multi-Stream Acoustic Modelling Using Raw Real and Imaginary Parts of the Fourier Transform
Loweimi, E., Yue, Z., Bell, P., Renals, S. J., & Cvetković, Z. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3237167 · Cited by 17
Interpretability for reliable, efficient, and self-cognitive DNNs: From theories to applications
Kang, X., Guo, J., Song, B., Cai, B., Sun, H., & Zhang, Z. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126267 · Cited by 16
DiscrimLoss: A Universal Loss for Hard Samples and Incorrect Samples Discrimination
Wu, T., Ding, X., Zhang, H., Gao, J., Tang, M., Du, L., Qin, B., & Liu, T. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3290477 · Cited by 15
Kilbride, J. B., Poortinga, A., Bhandari, B., Thwal, N. S., Quyen, N. H., Silverman, J., Tenneson, K. R., Bell, D. M., Gregory, M. J., Kennedy, R. E., et al. (2023) · Remote Sensing · doi:10.3390/rs15215223 · Cited by 18
A Bregman–Kaczmarz method for nonlinear systems of equations
Gower, R. M., Lorenz, D. A., & Winkler, M. (2023) · Computational Optimization and Applications · doi:10.1007/s10589-023-00541-9 · Cited by 15
“This assumption is strong, but can be fulfilled e.g. by modern machine learning applications such as non-parametric regression or over-parametrized deep neural networks [40, 63].”
MethodSparsity in long-time control of neural ODEs
Esteve, C., & Geshkovski, B. (2022) · Systems & Control Letters · doi:10.1016/j.sysconle.2022.105452 · Cited by 27
“Ultimately, this translates to simultaneously interpolating the above dataset through fapprox, whilst ensuring generalization/extrapolation, namely reliable prediction on points in X which are outside of said dataset ([47]).”
BackgroundMethodLarge Learning Rate Tames Homogeneity: Convergence and Balancing EffectPreprint
Wang, Y., Chen, M., Zhao, T., & Tao, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03677 · Cited by 49
“Empirical results indicate an intimate connection between training algorithms and the performance of trained models (Le et al., 2011; Bottou et al., 2018; Zhang et al., 2021; Soydaner, 2020; Zhou et al., 2020).”
BackgroundTran, V. N., Zhou, W., Kim, T., Mazepa, V. S., Valdayskikh, V., & Ivanov, V. Y. (2024) · Scientific Data · doi:10.1038/s41597-024-03483-x · Cited by 15
“To optimize effectiveness, an adaptive early stopping technique 78 was implemented.”
MethodExplainable Neural Rule Learning
Shi, S., Xie, Y., Wang, Z., Ding, B., Li, Y., & Zhang, M. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512023 · Cited by 19
“Moreover, neural networks might exploit spurious patterns as shortcuts to fit the training data instead of learning the causal patterns that consistently work on both in-distribution and out-of-distribution (OOD) data [28, 53].”
BackgroundA Convergence Path to Deep Learning on Noisy Labels
Liu, D., Tsang, I. W.-H., & Yang, G. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3202752 · Cited by 16
Enhancing Federated Learning With Server-Side Unlabeled Data by Adaptive Client and Data Selection
Xu, Y., Wang, L., Xu, H., Liu, J., Wang, Z., & Huang, L. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3265010 · Cited by 18
“The operator ◦ denotes function composition, i.e., ( Note that the transformation from a conventional model to the multi-exit model can be easily applied on any CNN-based model [41], e.g., VGG [43] and ResNet [44], which have been widely used in various FL applications [45], [46].”
MethodSun, W., Wu, D., Luo, Y., Liu, L., Zhang, H., Wu, S., Zhang, Y., Wang, C., Zheng, H., Shen, J., et al. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120710 · Cited by 14
SENSE: Self-Evolving Learning for Self-Supervised Monocular Depth Estimation
Li, G., Huang, R., Li, H., You, Z., & Chen, W. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3338053 · Cited by 20
Domain Generalization Guided by Gradient Signal to Noise Ratio of Parameters
Michalkiewicz, M., Faraki, M., Yu, X., Chandraker, M. K., & Baktashmotlagh, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00568 · Cited by 21
“A number of works study the generalization ability of gradient-based methods [41, 66, 28].”
BackgroundLocal Context-Aware Active Domain Adaptation
Sun, T., Lu, C. Y., & Ling, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01708 · Cited by 14
“It can also be utilized to update models and reduces the tendency to only memorize these newly queried data during fine-tuning [43].”
BackgroundParajuli, U., & Shin, S. (2024) · AQUA - Water Infrastructure Ecosystems and Society · doi:10.2166/aqua.2024.264 · Cited by 17
Deep learning in tropical leaf disease detection: advantages and applications
Yao, Z., & Huang, M. (2024) · Tropical Plants · doi:10.48130/tp-0024-0018 · Cited by 14
A Causality-Informed Graph Intervention Model for Pancreatic Cancer Early Diagnosis
Li, X., Guo, R., Zhu, H., Chen, T., & Qian, X. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3395586 · Cited by 16
“…many tasks [29], [30], [31], these data-driven architectures can hardly differentiate between causal and noncausal correlated factors, i.e., such models may learn spurious correlations from the training data as a shortcut to prediction, and hence show poor generalization on testing data [23], [32].”
BackgroundSummerfield, N., Morris, E. D., Banerjee, S., He, Q., Ghanem, A. I., Zhu, S., Zhao, J., Dong, M., & Glide‐Hurst, C. K. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2024.05.013 · Cited by 14
Resurrecting Label Propagation for Graphs with Heterophily and Label Noise
Cheng, Y., Shan, C., Shen, Y., Li, X., Luo, S., & Li, D. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671774 · Cited by 31
“It is well known that deep neural networks are very sensitive to noise, thereby degrading their generalization ability [33].”
On the Predictability of Fine-Grained Cellular Network Throughput Using Machine Learning Models
Basit, O., Dinh, P., Khan, I., Kong, Z. J., Hu, Y., Koutsonikolas, D., Lee, M., & Liu, C. (2024) · IEEE International Conference on Mobile Ad-Hoc and Sensor Systems (MASS) · doi:10.1109/mass62177.2024.00018 · Cited by 14
Noise-robust consistency regularization for semi-supervised semantic segmentation
Zhang, H., Li, H., Zhang, X., Yang, G., Li, A., Du, W., Xue, S., & Liu, C. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107041 · Cited by 18
Reinforcement Learning of Chaotic Systems Control in Partially Observable Environments
Weissenbacher, M., Borovykh, A., & Rigas, G. (2025) · Flow Turbulence and Combustion · doi:10.1007/s10494-024-00632-5 · Cited by 17
“…internal workings of neural networks is complicated by their non-linear structures and the interactions between layers, which can obscure the relationships between inputs and outputs, making it challenging to pinpoint the exact factors influencing model behavior (Lipton 2018; Zhang et al. 2021).”
Cerasa, A., & Zani, A. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.125357 · Cited by 18
Magnitude Attention-based Dynamic Pruning
Back, J., Ahn, N., & Kim, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126957 · Cited by 17
Jiao, Z., & Tao, R. (2025) · Transactions in GIS · doi:10.1111/tgis.70029 · Cited by 15
PLReMix: Combating Noisy Labels with Pseudo-Label Relaxed Contrastive Representation Learning
Liu, X., Zhou, B., Yue, Z., & Cheng, C. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00635 · Cited by 22
“These corrupted noisy labels severely interfere with the model’s training process due to over-fitting, which leads to worse generalization performance [2,59].”
Lu, Q., Ding, Z., Li, M., & Gu, J. (2025) · Ecological Indicators · doi:10.1016/j.ecolind.2025.113736 · Cited by 19
TSQP: Safeguarding Real-Time Inference for Quantization Neural Networks on Edge Devices
Sun, Y., Xiong, G., Liu, J., Liu, Z., & Cui, J. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00001 · Cited by 19
Cao, C., Zhao, W., Guo, J., Wang, Z., Lü, A., & Liang, C. C. (2025) · The Innovation Medicine · doi:10.59717/j.xinn-med.2025.100154 · Cited by 14
Fu, B., Kirchbuchner, F., & Kuijper, A. (2020) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3389189.3392606 · Cited by 25
Exploring Structural Sparsity of Deep Networks Via Inverse Scale Spaces
Fu, Y., Liu, C., Li, D., Zhong, Z., Sun, X., Zeng, J., & Yao, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3168881 · Cited by 23
Liu, H., Zhang, H., Lu, J., & Tang, Z. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3118216 · Cited by 16
Robust Learning From Noisy Web Images Via Data Purification for Fine-Grained Recognition
Zhang, C., Wang, Q., Xie, G.-S., Wu, Q., Shen, F., & Tang, Z. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3134156 · Cited by 14
Self-citationContrastive Active Learning Under Class Distribution Mismatch
Du, P., Chen, H., Zhao, S., Chai, S., Chen, H., & Li, C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3188807 · Cited by 20
“In real-world scenarios, the data distribution may not accord with the artificial distribution as assumed, just as mentioned in the works [49], [50].”
MethodZhu, M., Zhang, L., Wang, L., Li, D., Zhang, J., & Zhang, Y. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-022-02799-6 · Cited by 17
“Moreover, recent studies have shown that DNNs have suffi-B Lei Zhang leizhang@scu.edu.cn 1 College of Computer Science, Sichuan University, Chengdu 610065, Sichuan, China cient capacity to memorize the entire dataset [5].”
BackgroundBackward recursive Class Activation Map refinement for high resolution saliency map
Englebert, A., Cornu, O., & Vleeschouwer, C. D. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956711 · Cited by 15
“Some works study and attempt to explain the good generalization of learnt models to new samples [2], [3], [4].”
BackgroundLAFD-Net: Learning With Noisy Pseudo-Labels for Semisupervised Bearing Fault Diagnosis
Jian, Y., Chen, Z., Lei, Y., He, Z., Zhao, Y., He, L., Luo, W., & Chen, X. (2023) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3233957 · Cited by 14
“Our proposed method is mainly inspired by two facts: 1) recent studies [50], [51] have noticed that the deep-learning models generally have the memorization effect , that is, deep neural networks absorb knowledge from clean samples at the early stage but eventually memorizing (overfitting) the noisy…”
BackgroundMethodSupervised learning approaches to modeling pedestal density
Kit, A., Järvinen, A., Frassinetti, L., & Wiesen, S. (2023) · Plasma Physics and Controlled Fusion · doi:10.1088/1361-6587/acb3f7 · Cited by 14
“models would increase the risk of overfitting by having the capacity to memorize samples, overparameterized deep learning networks tend to still maintain generalization capabilities [13].”
BackgroundKumar, M. P., Gao, Z.-J., & Chen, K.-C. J. (2023) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2023.3248642 · Cited by 14
“will cause overfit or underfit problems [28].”
Surrogate Forward Models for Population Inference on Compact Binary Mergers
Riley, J. A., & Mandel, I. (2023) · The Astrophysical Journal · doi:10.3847/1538-4357/accf90 · Cited by 16
The severity level classification of Fusarium wilt of chickpea by pre-trained deep learning models
Hayıt, T., Endes, A., & HAYIT, F. (2023) · Journal of Plant Pathology · doi:10.1007/s42161-023-01520-z · Cited by 14
“CNNs also have some limitations that are important to consider, such as the requirement of a large amount of labeled training data and significant computational resources, including high-performance GPUs or specialized hardware (Ba and Caruana 2014; Marcus 2018; Lipton 2018; Zhang et al. 2021).”
Data Isotopes for Data Provenance in DNNs
Wenger, E., Li, X., Zhao, B. Y., & Shmatikov, V. (2023) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2024-0024 · Cited by 22
Improve Noise Tolerance of Robust Loss via Noise-Awareness
Ding, K., Shu, J., Meng, D., & Xu, Z. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3457029 · Cited by 19
Impact of Noisy Supervision in Foundation Model Learning
Chen, H., Wang, Z., Tao, R., Wei, H., Xie, X., Sugiyama, M., Raj, B., & Wang, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3552309 · Cited by 15
“For OOD tasks, the robustness performance is negatively correlated with the LSVR.”
MLR-SNet: Transferable LR Schedules for Heterogeneous Tasks
Shu, J., Zhu, Y., Zhao, Q., Meng, D., & Xu, Z. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3184315 · Cited by 14
Top interpretable neural network for handwriting identification
Marcinowski, M. (2022) · Journal of Forensic Sciences · doi:10.1111/1556-4029.14978 · Cited by 16
de Miguel-Rodríguez, J., Morales‐Esteban, A., Requena-García-Cruz, M.-V., Zapico-Blanco, B., Segovia-Verjel, M.-L., Romero-Sánchez, E., & Estêvão, J. M. C. (2022) · Sustainability · doi:10.3390/su14095274 · Cited by 14
“904, which lies within an acceptable range [52].”
BackgroundMethodde Oliveira, N. A., & Basso, L. F. C. (2025) · Risks · doi:10.3390/risks13060116 · Cited by 16
“This consideration is particularly important given that a deeper understanding of generalization remains critical for the reliable deployment of deep-learning models (Zhang et al. 2021).”
AI-assisted preoperative surgical planning for dental implant
Chen, F., Chen, H., Hai, D., Yang, Y., Xu, P., Yang, D., Wang, R., Bi, Z., Yuan, C., Wang, Y., et al. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-025-07584-y · Cited by 13
“Ultimately, prospective external validation on completely independent datasets from different geographic regions, patient populations, and equipment vendors remains the definitive benchmark for assessing true generalization capability [45].”
Functional network: A novel framework for interpretability of deep neural networks
Zhang, B., Dong, Z., Zhang, J., & Lin, H. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.11.035 · Cited by 17
DIOR: Learning to Hash With Label Noise Via Dual Partition and Contrastive Learning
Wang, H., Jiang, H., Sun, J., Zhang, S., Chen, C., Hua, X., & Luo, X. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3312109 · Cited by 16
“It has been widely recognized that deep neural networks tend to memorize noisy samples after learning generalized patterns [31].”
BackgroundInfluentialGradient-aware learning for joint biases: Label noise and class imbalance
Zhang, S., Zhu, C., Li, H., Cai, J., & Yang, L. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.12.028 · Cited by 14
High-risk event prone driver identification considering driving behavior temporal covariate shift
Zhang, R., Wen, X., Cao, H., Cui, P., Chai, H., Hu, R., & Yu, R. (2024) · Accident Analysis & Prevention · doi:10.1016/j.aap.2024.107526 · Cited by 15
On the Privacy Effect of Data Enhancement via the Lens of Memorization
Li, X., Li, Q., Hu, Z., & Hu, X. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3381477 · Cited by 26
Coupled Confusion Correction: Learning from Crowds with Sparse Annotations
Zhang, H., Li, S., Zeng, D., Yan, C. C., & Ge, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i15.29613 · Cited by 25
Competitive Model Selection in Algorithmic Targeting
Iyer, G. V., & Ke, T. T. (2024) · Marketing Science · doi:10.1287/mksc.2023.0175 · Cited by 13
Ashebir, H., Babaee, S., Diale, P. P., Worku, A., Msagati, T. A. M., & Fito, J. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.105170 · Cited by 13
Deep learning in vibrational spectroscopy: Benefits, limitations, and recent progress
Cai, Y., Lin, Y., Cai, H., & Ni, H. (2025) · Journal of the Chinese Chemical Society · doi:10.1002/jccs.70031 · Cited by 16
Bridging deep learning force fields and electronic structures with a physics-informed approach
Qi, Y., Gong, W., & Yan, Q. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01668-5 · Cited by 16
“This choice is motivated by the absence of atoms in the inter-layer gap affecting the Hamiltonian, and the hopping integral mostly depends on the relative positions of the atomic pair [38].”
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.