Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 1,501–1,600 of 2,667
Mensah, P. K., Ayidzoe, M. A., Opoku, A. A., Adu, K., Weyori, B. A., Nti, I. K., & Nimbe, P. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/4984490 · Cited by 3
“In addition, model complexity poses a threat of overfitting [34] that ultimately leads to poor performance.”
BackgroundCorrect Twice at Once: Learning to Correct Noisy Labels for Robust Deep Learning
Li, J., & Sun, H. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547861 · Cited by 7
“Previous works [76, 85] have shown that DNNs with powerful memorization will completely fit the noisy training data resulting in poor generalization over test data.”
BackgroundDeep Boosting Multi-Modal Ensemble Face Recognition with Sample-Level Weighting
Malakshan, S. R., Saadabadi, M. S. E., Najafzadeh, N., & Nasrabadi, N. M. (2023) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb57857.2023.10448893 · Cited by 3
“However, current SOTA performance demonstrates limited generalization on low-quality and Low-Resolution (LR) inputs, such as images captured by surveillance cameras or captured from long ranges [57].”
BackgroundSpectral Pruning for Recurrent Neural NetworksPreprint
Furuya, T., Suetake, K., Taniguchi, K., Kusumoto, H., Saiin, R., & Daimon, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.10832 · Cited by 3
“We observed that our method outperforms existing methods, and gets benefits from over-parameterization (Chang et al., 2020; Zhang et al., 2021) (see Sections 5.”
MethodSGD: The Role of Implicit Regularization, Batch-size and Multiple-epochsPreprint
Kale, S., Sekhari, A., & Sridharan, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05074 · Cited by 24
“The final empirical observation we consider is the fact that multiple epochs of SGD tends to further continually decrease not only the training error but also the test error (Zhang et al., 2021; Ma et al., 2018; Bottou and Bousquet, 2011).”
BackgroundGeneralization Properties of Stochastic Optimizers via Trajectory Analysis.Preprint
Hodgkinson, L., Şimşekli, U., Khanna, R., & Mahoney, M. W. (2021) · arXiv (Cornell University) · Cited by 3
Lower Bounds for the Minimum Mean-Square Error via Neural Network-based EstimationPreprint
Díaz, M., Kairouz, P., & Sankar, L. (2021) · arXiv (Cornell University) · Cited by 3
Anti-Neuron Watermarking: Protecting Personal Data Against Unauthorized Neural Model Training.Preprint
Zou, Z., Gong, B., & Wang, L. (2021) · arXiv (Cornell University) · Cited by 3
Investigating Temporal Convolutional Neural Networks For Satellite Image Time Series ClassificationPreprint
Abdallah, Z. S., & Brock, J. (2022) · doi:10.36227/techrxiv.19590580 · Cited by 3
A Simple Weakly-Supervised Contrastive Learning Framework for Few-shot Sentiment ClassificationPreprint
Chen, L., Ren, Z., Lu, S., Huang, X., Wang, W., Xu, C. X., Zhao, W. Z., & Guan, Z. (2023) · Research Square · doi:10.21203/rs.3.rs-2699220/v1 · Cited by 3
Compressing the Backward Pass of Large-Scale Neural Architectures by Structured Activation PruningPreprint
Barley, D., & Fröning, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.16883 · Cited by 3
“Overparameterization : Although reasons for this are not yet fully understood, overparameterization of networks, in combination with regularization, seems to lead to better generalization [15]–[17].”
BackgroundDeep Learning with DAGsPreprint
Balgi, S., Daoud, A., Pena, J., Wodtke, G. T., & Zhou, J. (2024) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/6375y · Cited by 4
Explore-Go: Leveraging Exploration for Generalisation in Deep Reinforcement LearningPreprint
Weltevrede, M., Kaubek, F., Spaan, M. T. J., & Böhmer, W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.08069 · Cited by 3
“…commonly used to improve generalisation performance in a wide variety of settings and applications (Shorten & Khoshgoftaar, 2019; Feng et al., 2021; Zhang et al., 2021a; Miao et al., 2023) and are thought to work by reducing overfitting to spurious correlations (Shen et al., 2022), inducing model…”
BackgroundPaluh, J. L., Sunil, S., Rajeev, R., Chatterjee, A., Pilitsis, J. G., & Mukherjee, A. (2024) · Research Square · doi:10.21203/rs.3.rs-4087784/v1 · Cited by 4
Fragile Giants: Understanding the Susceptibility of Models to Subpopulation AttacksPreprint
Gupta, I., Lycklama, H., Opel, E., Rose, E., & Hithnawi, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.08872 · Cited by 3
Benchmarking noisy label detection methodsPreprint
da Silva, H. P., Kamassury, J. K. S., & Silva, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.16211 · Cited by 6
Computational Convergence of Adaptive Immunity and Artificial IntelligencePreprint
Reddy, S. T. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.03.703525 · Cited by 4
“Both must generalize to novel instances never seen during training ( 26, 27 ).”
An Effective Data Refinement Approach for Upper Gastrointestinal Anatomy Recognition
Quan, L., Li, Y., Chen, X., & Zhang, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-59710-8_5 · Cited by 2
Free Dynamics of Feature Learning Processes
Furtlehner, C. (2023) · Journal of Statistical Physics · doi:10.1007/s10955-022-03064-5 · Cited by 2
“The rapid developments of the last decades inmachine learning (ML), in particular around the deep neural networks (DNN), have led statisticians to revisit the mathematical foundations of the field [1, 2], where at odds with intuition, highly over-parameterized models (typically DNN) perform well in generalization [3,…”
BackgroundSIAR: An Effective Model for Predicting Game Propagation
Wang, T., Ye, G., Liu, X., Zhou, R., Li, J., & Wang, T. (2024) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-9342-0_31 · Cited by 2
Unraveling Convolution Neural Networks: A Topological Exploration of Kernel Evolution
Yang, L., Xu, M., & He, Y. (2024) · Applied Sciences · doi:10.3390/app14052197 · Cited by 3
Learning Fixed Points of Recurrent Neural Networks by Reparameterizing the Network Model
Zhu, V., & Rosenbaum, R. (2024) · Neural Computation · doi:10.1162/neco_a_01681 · Cited by 4
“(32) in the overparameterized case is well understood [25, 26, 27, 28].”
BackgroundPerformance analysis of quantum convolutional layers for image classification
Xiang, Q., Li, D., Sun, Y., Hu, Z., Yuan, Y.-H., Hua, X., Zhu, Y.-H., Fu, Y., & Jiang, Y. (2024) · Physica Scripta · doi:10.1088/1402-4896/ad8d17 · Cited by 2
Improving Time Series Classification with Representation Soft Label Smoothing
Ma, H., & Chen, W. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0840-9_21 · Cited by 3
Fukunaga, R., Yoshida, S., Higashimoto, R., & Muneyasu, M. (2024) · ITE Transactions on Media Technology and Applications · doi:10.3169/mta.13.91 · Cited by 2
Modeling the Possible: Perspectives from Philosophy of Science
Knuuttila, T., Grüne‐Yanoff, T., Koskinen, R., & Wirling, Y. S. (2025) · Routledge eBooks · doi:10.4324/9781003342816 · Cited by 3
Integrity Verifiable Privacy-preserving Federated Learning for Healthcare-IoT
Li, J., & Yu, S. (2024) · IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom60970.2024.10880846 · Cited by 2
Prototype-Oriented Clean Subset Extraction for Noisy Long-Tailed Classification
Li, Z., Zhao, H., Gao, A., Guo, D., Chang, T., & Wan, X. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3546031 · Cited by 2
“These noisy labels impair the generalization of models, leading to suboptimal performance [8], [9], [10], [11].”
Robust Weakly Supervised Bird Species Detection via Peak Aggregation and PIE
Ghaffari, H., & Devos, P. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3552983 · Cited by 4
“Consider that a neural network can associate random numbers with meaningful objects after memorizing datasets [28] or achieve perfect accuracy on a dataset with unrelated input-output pairs [29].”
Zhang, S., Chu, S., Qiang, Y., Zhao, J., Wang, Y., & Wei, X. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108734 · Cited by 2
Park, H., Hong, S., Song, C., Kim, J. W., & Yi, M. Y. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00068 · Cited by 3
“According to the memorization effect [39], DNNs typically learn simpler samples early in training and then fit harder ones later.”
Liang, H., Zheng, H., Wang, H., He, L., Lin, H., & Liang, Y. (2025) · Mathematics · doi:10.3390/math13081259 · Cited by 3
Wang, Y., Chang, Y., Qin, Y. H., Zhao, Y., & Wei, S. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3562561 · Cited by 4
“The underlying principle relies on the memory e ff ect of neural networks [21], [22], which tend to learn and fit simpler samples that exhibit relatively small loss.”
Zhou, H., Xu, Z., Chen, Y., Yan, Y., Zhang, S., Lin, X., Cui, D., & Yang, J. (2025) · Journal of Economic Entomology · doi:10.1093/jee/toaf087 · Cited by 4
Almohab, H., Sidik, A. D. W. M., & Junfithrana, A. P. (2024) · IEEE International Conference on Computing, Engineering and Design (ICCED) · doi:10.1109/icced64257.2024.10983686 · Cited by 2
Neural network matrix product operator: A multi-dimensionally integrable machine learning potential
Hino, K., & Kurashige, Y. (2025) · Physical Review Research · doi:10.1103/physrevresearch.7.023217 · Cited by 4
Ratchatorn, T., & Tanaka, M. (2025) · IEEE Access · doi:10.1109/access.2025.3578265 · Cited by 2
“This issue, known as overfitting, causes a notable gap in performance between the training and testing datasets [1].”
Enhanced NILM Based on an Improved RGB Color-Encoded V–I Trajectory Representation
Meng, F., Lu, L., & Yu, M. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3577680 · Cited by 2
Higashi, T., & Nakai, T. (2025) · doi:10.1145/3709018.3736329 · Cited by 2
“Zhang et al. have elucidated that conventional neural networks possess the capability to memorize datasets labeled entirely at random [27].”
Chang, Z., McGinnity, C. J., Hinz, R., Wang, M., Dunn, J., Liu, R., Yakubu, M., Marsden, P. K., & Hammers, A. (2025) · EJNMMI Research · doi:10.1186/s13550-025-01251-5 · Cited by 2
“Neural networks can learn complex nonlinear relationships, but unlike the Fine Tree, they are not suitable for handling sparse data as they tend to overfit non-significant features, leading to performance degradation [42, 43].”
A Deep Learning Method for Pneumoconiosis Staging on Chest X-Ray Under Label Noise
Sun, W., Wu, D., Jiang, S., Luo, Y., Wang, H., Li, M., & Luo, C. (2025) · IEEE Access · doi:10.1109/access.2025.3590783 · Cited by 2
“1) The memory of deep neural networks is so powerful that it can even remember datasets with completely random labels [16].”
Semi-supervised Self-learning Enhanced Music Emotion Recognition
Sun, Y., Zhang, X., Zhou, M., & Li, W. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-4783-5_8 · Cited by 2
FedGraft: Memory-Aware Heterogeneous Federated Learning via Model Grafting
Liu, R., Hu, M., Xia, Z., Xie, X., Xia, J., Zhang, P., Huang, Y., & Chen, M. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3591537 · Cited by 4
“…is built on top of the following two facts: i) a given set of heterogeneous models with diverse architectures can be partitioned into functionally equivalent and interchangeable building blocks [30], and ii) the early layers of models are used to capture common features of the input samples [31].”
Domain-Division Based Progressive Learning for Source-Free Domain Adaptation
Liu, P., Li, J., Zhao, M. Y., Xue, W., Hu, Q., & Chen, S. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3590903 · Cited by 4
Chasing Shadows: Solving Deepfake Detection Benchmarks Using Irrelevant Features Only
Xu, Y., Terhörst, P., Pedersen, M., & Raja, K. B. (2025) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg61629.2025.11099163 · Cited by 2
ADU: Adaptive Detection of Unknown Categories in Black-Box Domain Adaptation
Lai, Y., Li, G., Liang, H., Zheng, J., & Ye, Z. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02848 · Cited by 2
“Deep learning models often overfit on noisy labels, leading to poor generalization [60].”
Salleh, M. F. S., Samsudin, A. H., Ismail, M. S., & Mahmud, J. (2025) · IEEE International Conference on Automatic Control and Intelligent Systems (I2CACIS) · doi:10.1109/i2cacis65476.2025.11101549 · Cited by 2
Huy, T. G., Nhan, L. N. T., & Cam, N. T. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111638 · Cited by 3
Fang, R., Sun, Y., Yuan, J., & Cao, H. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3602959 · Cited by 2
Modality-AGnostic image Cascade (MAGIC) for multi-modality cardiac substructure segmentation
Summerfield, N., He, Q., Kuo, A., Ruff, C., Pan, J., Ghanem, A. I., Zhu, S., Kumar, A., Nagpal, P., Zhao, J., et al. (2025) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2025.111296 · Cited by 2
Rapid federated unlearning with tuning parameters based on fisher information matrix
Zhao, F., Xu, Q., Wang, H., & Guo, D. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06593-0 · Cited by 2
“When a model is trained on a dataset, its parameter vector is updated along the negative gradient direction to converge to a minimum of the loss surface [35, 36].”
Khan, R., Saleh, H. H., Mefgouda, B., Alhussein, O., & Muhaidat, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-20305-1 · Cited by 2
“Moreover, the sensitivity of neural networks to such ablations may vary across different layers, with deeper layers often exhibiting higher vulnerability due to their role in representing more abstract features 28,29 .”
On the Limitations of Fractal Dimension as a Measure of Generalization
Tan, C., García‐Redondo, I., Wang, Q., Bronstein, M. M., & Monod, A. (2024) · neural information processing systems · doi:10.52202/079017-1928 · Cited by 2
Wang, N., Jin, W., Bi, H., & Yang, G. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131998 · Cited by 2
A predictive and prescriptive analytics approach for sustainable cellphone return management
Ibrahim, A. A., & Abdul‐Kader, W. (2025) · Decision Analytics Journal · doi:10.1016/j.dajour.2025.100656 · Cited by 3
Privacy-Preserving Residential Short-Term Load Forecasting With Missing Data Tolerance
Liang, S., Pang, K., Feng, C., Yang, Y., & Wen, F. (2025) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2025.3628876 · Cited by 3
When Interpretability Meets Noise: An LLM-Assisted Hybrid Deep Logical Rule Learning Framework
Chen, Y., Yuan, B., Chen, C., Li, Z., & Liao, B. (2025) · Machine Learning · doi:10.1007/s10994-025-06931-w · Cited by 2
“Notably, an increasing number of self-explanatory approaches are built on logical rules, offering globally consistent explanations by incorporating interpretability into neural networks (Zhang et al., 2021c; Benamira et al., 2023).”
InfluentialDu, R., Bu, T., Huang, Z., Morley, J. D., Xing, Y., Li, M., Xu, H., Feng, Y., Zhou, X., Long, D., et al. (2025) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2025.146996 · Cited by 3
Energy-Aware Anomaly Detection in Wind Turbine SCADA Systems
Ciocan, A. V., Hamrioui, S., Lorenz, P., Courboulay, V., & Ciocan, A. (2025) · International Conference on Software, Telecommunications and Computer Networks (SoftCOM) · doi:10.23919/softcom66362.2025.11197377 · Cited by 2
Reconstruction of dance movements using evolved parallel recurrent and wavelet models
Zhang, L., Al-Shamasneh, A. R., Karim, F. K., Saidani, T., Khishe, M., & Smerat, A. (2025) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2025.108276 · Cited by 2
El‐Dalahmeh, M., Siddhapura, O., Apachitei, G., Capener, M. J., Marco, J., Todd, A., Goodwin, K., & Niri, M. F. (2025) · Energy and AI · doi:10.1016/j.egyai.2025.100661 · Cited by 2
Bio-Inspired Generative Network with Knowledge Integration
Batbaatar, E., & Ryu, K. H. (2025) · Applied Sciences · doi:10.3390/app152412918 · Cited by 2
Hippocampal-prefrontal orchestration supports higher-order learning for creative ideation
Zhang, Z., Li, Z., Ding, R., Wu, G., Xie, X., Wu, J., Qin, S., & Luo, J. (2025) · Nature Communications · doi:10.1038/s41467-025-67157-x · Cited by 3
“Dimensionality reduction is cru-cialforfosteringthegeneralizationabilityinhigher-levelcognitionand deep learning models 26,27 .”
Chen, K., Yang, K., Deng, Z., Chen, J., Huang, H., & Yang, J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.113351 · Cited by 2
Qiwang, L., Shao, M., Cai, Y., Li, D., Hong, S., Qi, J., & Sun, H. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3606299 · Cited by 2
DQSA: Dynamic Quantized Self-Attention for Multi-Task Encrypted Network Traffic Classification
Chen, Y., Lan, H., Gwee, B., Cao, Q. P., Razul, S. G., & Lin, Z. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3647211 · Cited by 2
“First, incorporating fine-grained tra ffi c features into attention analysis improves fidelity but increases the risk of overfitting due to irrelevant details [16].”
Representation Optimal Matching for Federated Learning With Noisy Labels in Remote Sensing
Jiang, X., Wen, T., Sun, S., Yuan, J., Liu, H., Li, P., Wu, L., Wang, Y., & Liu, M. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3649179 · Cited by 4
Chen, L., Tang, N., Wu, X., Wang, Y., He, C., He, Z., Xie, L., Zhang, X., Chen, X., & Zhou, T. (2026) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1728353 · Cited by 3
“This is likely a consequence of the limited sample size (N = 90), which is insuf fi cient for training a complex model with a large number of parameters without severe over fi tting (Zhang et al., 2021).”
Towards boundary confusion for volumetric medical image segmentation
You, X., Ding, M., Zhang, M., Zhang, H., Wu, J., Yu, Y., Yang, J., & Gu, Y. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.103961 · Cited by 4
When Does Wasm Malware Detection Fail? A Systematic Analysis of Their Robustness to Evasion
Kim, T.-Y., Oh, S., Lee, K., Wang, W., Kwon, Y., Hong, S., & Kim, H. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00243 · Cited by 2
Bits for Privacy: Evaluating Post-Training Quantization via Membership Inference
Zhang, C., Qu, T., Li, Z., Zhang, T., Pang, J., & Mauw, S. (2025) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom66490.2025.00099 · Cited by 3
Self-citationFederated learning with dynamics-aware loss for label noise
Ouyang, C., Mao, J., Liu, Z., Zhu, D., Zhou, C., Hu, G., & Li, T. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131523 · Cited by 3
Benítez‐Andrades, J. A., Laballos-González, L., Pujante-Otalora, L., Nieto-Marcos, S., Pinto‐Carral, A., & Álvarez, M. J. Á. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03384-6 · Cited by 2
“High-capacity ensemble and gradient-boosting methods typically require larger datasets to achieve stable estimation of tree structures and to support the extensive hyperparameter optimisation needed for reliable generalisation [36, 37].”
Liu, J. (2026) · Computers in Human Behavior · doi:10.1016/j.chb.2026.108988 · Cited by 7
Hybrid Spatio-Temporal Deep Learning Models for Multi-Task Forecasting in Renewable Energy Systems
Tolegenova, G. B., Zakirova, A., Kalimoldayev, M., & Akhayeva, Z. (2026) · Computers · doi:10.3390/computers15030183 · Cited by 2
Fixing foundational concepts in machine learning: a methodological primer
Grote, T., & Huang, A. C. W. (2026) · Synthese · doi:10.1007/s11229-026-05510-3 · Cited by 2
Yang, J., Liu, Q., Cao, X., Li, F., & Yang, X. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115754 · Cited by 2
Agarwal, S., & Peta, S. B. (2025) · International Conference on Artificial Intelligence and Speech Technology (AIST) · doi:10.1109/aist68591.2025.11441565 · Cited by 2
A review of fairness challenges in natural language processing
Zadid, T. H., Jahin, S. M., Dhruba, A. R., Tanim, S. A., Hamja, A. M., Hasan, M. R., Miah, M. S. U., Islam, M. I., & Mansur, A. A. (2026) · Array · doi:10.1016/j.array.2026.100773 · Cited by 4
Exploring Hybrid Quantum-Classical Methods for Practical Time-Series Forecasting
Dimitrijevs, M., Kālis, M., & Repko, I. (2026) · Baltic Journal of Modern Computing · doi:10.22364/bjmc.2026.14.1.09 · Cited by 2
Zhao, F., Zhou, Z., Zhai, H., Zhang, P. F., & Li, X. (2026) · Computers & Geosciences · doi:10.1016/j.cageo.2026.106168 · Cited by 2
Reflections on predictive modeling for infectious diseases
Zhao, D. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1811213 · Cited by 2
From enclosure to foreclosure and beyond: opening AI’s totalizing logic
Schwerzmann, K. (2026) · AI & Society · doi:10.1007/s00146-026-03024-7 · Cited by 2
“Such models can even learn random, noisy data—that is, memorize them—even when this distribution doesn’t entail any meaningful pattern (Zhang et al. 2021, p. 107).”
Tie, Z., Meng, X., Chen, C., Xu, C., & Zuo, S. (2026) · Bioresource Technology · doi:10.1016/j.biortech.2026.134730 · Cited by 2
Guiding Noisy Label Conditional Diffusion Models with Score-Based Discriminator Correction
Cong, D. N., Bao, H. T., & Hoang-Thanh, T. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01722 · Cited by 4
“Patel and Sastry [37], Zhang et al. [58] showed that the model eventually memorizes noisy data given sufficient training time, however, the model typically remembers clean data before remembering noisy data.”
Rajaram, G., & Jayani, B. V. (2026) · IEEE International Conference on Emerging Computing and Intelligent Technologies (ICoECIT) · doi:10.1109/icoecit68303.2026.11496919 · Cited by 2
Advances in artificial intelligence techniques for diagnosis of cardiac diseases
Houssein, E. H., Mostafa, H., Ali, A. A., Shaban, H., & Hassanein, M. S. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01323-y · Cited by 2
“Advancing adaptive algorithms capable of generalizing across various healthcare settings and patient populations, along with large-scale external validation studies, is therefore crucial for improving model robustness and real-world applicability [92, 93].”
A Comprehensive Review on Deep Learning With Noisy Labels: Challenges, Solutions, and Frontiers
Fang, J., Wang, Z., Liu, W., Zhang, Y. J., He, Y., & liu, X. (2026) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2026.3683790 · Cited by 3
DGVAC: Disentangled Gaussian-vMF alignment for deep clustering
Ruan, T., Yu, N., Zhao, S., Chen, F., & Huang, Y. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109114 · Cited by 3
Tessema, M. C., Deresse, A. T., Firdi, M. D., & Dufera, T. T. (2026) · Discover Applied Sciences · doi:10.1007/s42452-026-08810-y · Cited by 2
AI-based UAV pest and disease detection: Time for a reset?
den Eeckhout, E. V., Spanoghe, P., & Maes, W. H. (2026) · Trends in Plant Science · doi:10.1016/j.tplants.2026.05.015 · Cited by 2
Gonçalves, H. O., Amaral, R., & Pereira, J. C. (2026) · Risk Analysis · doi:10.1111/risa.70297 · Cited by 2
HyperCore: Coreset Selection Under Noise via Hypersphere Models
Moser, B. B., Shanbhag, A. S., Nauen, T. C., Frolov, S., Raue, F., Folz, J., & Dengel, A. R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31335-5_15 · Cited by 2
Rethinking Federated Unlearning via the Lens of Memorization
Wei, J., Zhang, Y., Zhang, H., Zhang, L. Y., Chen, C., Ong, K., Zhang, J., & Xiang, Y. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817785 · Cited by 2
Memorization in 3D Shape Generation: An Empirical Study
Pu, S., Zeng, B., Zhou, K., Wang, M., & Liu, Z. (2026) · doi:10.1109/cvprf71909.2026.00180 · Cited by 2
Noise-resilient training of machine learning interatomic potentials
Lam, T. C. W., O’Neill, N., Schran, C., & Schaaf, L. L. (2026) · The Journal of Chemical Physics · doi:10.1063/5.0327092 · Cited by 2
Wallace, E. (2019) · Distill · doi:10.23915/distill.00019.6 · Cited by 2
Improving Hyperspectral Super-Resolution via Heterogeneous Knowledge Distillation
Liu, Z., Ma, Q., Jiang, J., & Liu, X. (2021) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3469877.3490610 · Cited by 2
“Since neural network is highly redundant [39], we believe that the idea is quite reasonable.”
BackgroundAl-Dmour, H. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00624-0 · Cited by 2
“These approaches encompass Monte-Carlo Dropout (MCDropout) [10], Maximum Class Probability (MCP) [17], Trust Score [20].”
MethodA Multi-Modal Fusion Network Guided by Feature Co-Occurrence for Urban Region Function Recognition
Zhang, N., Yongbin, W. A. N. G., Xiaoguang, W. A. N. G., & Yu, P. (2022) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2021edp7230 · Cited by 2
“Many studies demonstrated that deep learning network is prone to overfit to noise samples [19]–[21].”
Background
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.