Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 1,601–1,700 of 2,667
LCN: Label Correction Based on Network Prediction for Cross-Modal Retrieval with Noisy Labels
Okamura, D., Harakawa, R., & Iwahashi, M. (2022) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipaasc55919.2022.9980224 · Cited by 2
Hosseini, M. K., & Talebpour, A. (2023) · Journal of Transportation Engineering Part A Systems · doi:10.1061/jtepbs.teeng-7209 · Cited by 2
Liu, C., Raza, H., & Bhattacharyya, S. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-85955-4.00010-7 · Cited by 4
The implicit mathematical reasoning model combining self-attention and convolution
Yao, Z.-K., Zeng, B., Hu, H., & Wei, P.-F. (2023) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-224598 · Cited by 1
“However, it also leads the model to become overconfident, with test losses falling and then rising again on the test set, which is called over-fitting [21–23].”
Overview of artificial intelligence model watermarking
Wu, H., Zhang, J., Li, Y., Yin, Z., Zhang, X., Tian, H., Li, B., Zhang, W., & Yu, N. (2023) · Journal of Image and Graphics · doi:10.11834/jig.230010 · Cited by 4
fmLRE: A Low-Resource Relation Extraction Model Based on Feature Mapping Similarity Calculation
Wang, P., Shao, T., Ji, K. F., Li, G., & Ke, W. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26605 · Cited by 7
“However, poor classifier performance trained on few labeled data leads to insufficient precision of the pseudolabel, resulting in selection bias problem (Zhang et al. 2021).”
BackgroundRafanelli, A. (2023) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.385.51 · Cited by 2
Surrogate Deep Learning to Estimate Uncertainties for Driver Intention Recognition
Vellenga, K., Karlsson, A., Steinhauer, H. J., Falkman, G., & Sjögren, A. (2023) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3587716.3587758 · Cited by 2
“Thus, expressing how certain an underlying ML model is about the made predictions is vital for AI systems that are likely to encounter situations that differ from the ones it has been trained on [5, 43, 55, 58].”
Mallem, S., & Nakib, A. (2023) · Proceedings - International Conference on Image Processing · doi:10.1109/icip49359.2023.10222457 · Cited by 2
“This is because these models are often large enough to overfit on incorrect labels, leading to a lack of generalization and suboptimal results [1].”
BackgroundPicture This: A Deep Learning Model for Operational Real Estate Emissions
Gloria, B., & Höhn, B. (2023) · Journal of Sustainable Real Estate · doi:10.1080/19498276.2023.2251982 · Cited by 2
Class Imbalanced Semi-Supervised Learning With Meta-Learning
Shi, Y., Zhang, M., Zheng, X., & Chen, H. (2023) · International Conference on Cyber-Physical Social Intelligence (ICCSI) · doi:10.1109/iccsi58851.2023.10303813 · Cited by 3
“The research on the weak supervision problem in supervised learning has found that the label noise [10], class imbalance and other problems will lead to a large deviation between the distribution of weakly supervised data and the distribution of real data.”
BackgroundModerate Empiricism and Machine Learning
Buckner, C. J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197653302.003.0001 · Cited by 2
Distributed Continual Learning With CoCoA in High-Dimensional Linear Regression
Hellkvist, M., Özçelikkale, A., & Åhlén, A. (2024) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2024.3361714 · Cited by 4
Attentive Learning Facilitates Generalization of Neural Networks
Lei, S., He, F., Chen, H., & Tao, D. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3356310 · Cited by 2
“Although NNs are extremely overparameterized and can easily interpolate training data [33], [34], [35], they demonstrate excellent generalizability in many application domains, and the reason has been investigated from multiple perspectives, such as network parameters and learning dynamics.”
Multi-label noisy samples in underwater inspection from the oil and gas industry
Sousa, V. B., Pereira, S. A., Koher, T. M., & Pacheco, F. M. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09434-2 · Cited by 2
“Although the SLAM itself may enhance the generalization ability of networks in the presence of noisy samples, it still suffers from memorization effects on noisy labels [27].”
Carneiro, G. (2024) · Elsevier eBooks · doi:10.1016/b978-0-44-315441-6.00010-1 · Cited by 2
Introducing vacancy defects by rapid quenching in FCC metal for deep learning micrograph dataset
Hu, S., Li, Q., Sun, H., Lv, X., Song, M., Zhao, Y., Zhang, R., Zhang, C., Shen, Y., Liu, Y., et al. (2024) · Materials Characterization · doi:10.1016/j.matchar.2024.113827 · Cited by 2
Self-citationStudy of YOLOv7 Model Lightweighting Based on Group-Level Pruning
Bai, X., Shi, H., Wang, Z., Hu, H., & Zhang, C. (2024) · IEEE Access · doi:10.1109/access.2024.3423816 · Cited by 2
Self-citationNishimori, H., & Suzuki, T. (2024) · Information Geometry · doi:10.1007/s41884-024-00142-3 · Cited by 2
“However, it has been reported for various models experimentally that over-parametrized models generalize very well even when no regularization is applied [5, 16, 21].”
Degrees of Freedom: Search Cost and Self-Consistency
Wang, L., Zhao, H., & Fan, X. (2024) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2024.2388545 · Cited by 2
“The authors claimed that it could help explain the “double descent” phenomenon in over-parameterized interpolating models (e.g., Zhang et al. 2021; Hastie et al. 2022).”
Wu, Z., Cai, H., Ling, Y., & Pan, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5591-2_34 · Cited by 3
Hierarchical Generalization Bounds for Deep Neural Networks
He, H., Yu, C. L., & Goldfeld, Z. (2024) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit57864.2024.10619279 · Cited by 2
“Despite this wealth of research, the precise factors contributing to the generalization capacity of DNNs remain elusive, as indicated in [18], [19].”
BackgroundLai, K.-W., & Muthukumar, V. (2024) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit57864.2024.10619252 · Cited by 2
“Overparameterized models are ubiquitous in machine learning theory and practice today because of their state-of-the-art generalization (in the sense of low test error) even while perfectly fitting the training data [3], [4].”
BackgroundEnhancing Sharpness-Aware Minimization by Learning Perturbation Radius
Wang, X., Jiang, W., Fu, S., & Zhang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70344-7_22 · Cited by 3
“Deep neural networks have demonstrated remarkable performance across various fields [17,48], but they tend to overfit on the training data with poor ability of generalization due to overparameterization [49].”
BackgroundHSALC: hard sample aware label correction for medical image classification
Wang, Y., Ye, Y., Xie, Y., Tang, M., & Fan, L. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20114-0 · Cited by 3
“Numerous studies [43] indicate that deep neural networks will first tend to learn clean samples and then fit noisy samples.”
Continuous Value Assignment: A Doubly Robust Data Augmentation for Off-Policy Learning
Lin, J., Huang, Z., Wang, K., Liu, L., & Lin, L. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3435406 · Cited by 3
Yuhan, Z., Jiru, C., Wang, B., Shaochun, H., Wang, W., & Jiayi, Z. (2024) · Annals of Nuclear Energy · doi:10.1016/j.anucene.2024.110913 · Cited by 2
Location-Aware and Privacy-Preserving Data Cleaning for Intelligent Transportation
Wang, Y., Zhang, J., Ma, Z., Lu, N., Li, T., & Ma, J. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3453340 · Cited by 2
“A set of experiments [8] have shown that the presence of noise in a dataset for training can greatly reduce the accuracy of models.”
Chen, Z., Wu, X., & Zhang, Y. L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00493 · Cited by 2
“In the first stage, we optimize the color and geometry models to obtain a well-initialized radiance field and SDF. Previous studies [2, 53] have indicated that neural networks tend to memorize clean and easy patterns in the early stages of training.”
MethodLu, L., Ying, S., Xiao, Q., & Hou, C. (2024) · Measurement Science and Technology · doi:10.1088/1361-6501/ad7e40 · Cited by 2
“This strongly validates the conclusions drawn by Bengio et al [40] and underscores the feasibility of our approach to calibrate hyperparameters by comparing model deviations on the temperature generalization dataset.”
InfluentialAdaptive Adversarial Cross-Entropy Loss for Sharpness-Aware Minimization
Ratchatorn, T., & Tanaka, M. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10647582 · Cited by 2
Linear Feature Source Prediction and Recombination Network for Noisy Label Learning
Zheng, R., Han, C., Gao, C., & Sang, N. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3478771 · Cited by 2
“Many methods [7], [8], [9], [10], [12], [13], [14], [16], [17], [18], [19], [38], [39], [40], [41] have focused on training DNN under the condition of noisy label.”
FastSimiFeat: A Fast and Generalized Approach Utilizing k -NN for Noisy Data Handling
Lee, J., Park, H., Kim, M., Yoon, J., Yoo, K., & Byun, S.-J. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679591 · Cited by 2
Ibias, A., Antona, H., Ramirez-Miranda, G., Guinovart, E., & Alarcón, E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33010-9_27 · Cited by 2
Abnormality Monitoring and Recognition of Surveillance Video Based on ResNet Residual Network
Yang, H., Chen, D., & Feng, X. (2024) · International Conference on Artificial Intelligence and Computer Information Technology (AICIT) · doi:10.1109/aicit62434.2024.10730173 · Cited by 2
Deep weighted survival neural networks to survival risk prediction
Hui, Y., Wang, Q., Zhou, X., Gu, L., & Zhao, Z. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01670-2 · Cited by 2
“For example, Zhang et al. [12] demonstrated that although a standard DNNs can accommodate any proportion of noise in the training datasets, this ulti-123 mately leads to poor prediction performance.”
Sawalha, H., Abu-Alrub, M., Abu-Alrub, B., & Kmail, A. (2024) · Journal of Animal Health and Production · doi:10.17582/journal.jahp/2024/12.s1.145.156 · Cited by 2
Liu, Y., Yue, C., Lo, S., Wu, Z., & Deng, S. (2024) · Signal Image and Video Processing · doi:10.1007/s11760-024-03591-6 · Cited by 2
Channel- and Label-Flip Data Augmentation for Motor Imagery-Based Brain–Computer Interfaces
Hoshino, T., Kanoga, S., & Aoyama, A. (2024) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc53108.2024.10782028 · Cited by 2
“T. Hoshino and S. Kanoga are with the National Institute of Advanced Industrial Science and Technology (AIST), 2-4-7 Aomi, Koto-ku, Tokyo 135-0064, Japan. to be as effective as regularization [9], [10].”
Zhu, H., Lu, H., Jiang, Z., Li, H., Yang, C., Ni, Z., Feng, H., & Zhang, S. (2024) · Ironmaking & Steelmaking Processes Products and Applications · doi:10.1177/03019233241306076 · Cited by 2
Labels Generated by Large Language Models Help Measure People’s Empathy in Vitro
Hasan, M. R., Yao, Y., Hossain, M. Z., Krishna, A., Rudas, I. J., Rahman, S., & Gedeon, T. (2026) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2026.3671186 · Cited by 3
“This issue is exacerbated by noisy labels, where traditional regularisation techniques like dropout and weight decay often fall short [18].”
Cyclic Learning Rate-Based Co-Training for Image Classification With Noisy Labels
Zheng, Y., Gu, Y., Bai, P., Yuan, D., Zhou, S., Lyu, X., & Chen, A. (2025) · IEEE Access · doi:10.1109/access.2025.3526332 · Cited by 2
Machine learning models for predicting ultimate bond strength of grouted sleeve connections
Lou, J., Li, Y., Feng, Q., & Xu, R. (2025) · Structures · doi:10.1016/j.istruc.2024.108186 · Cited by 2
Unraveling the complexity of rat object vision requires a full convolutional network and beyond
Muratore, P., Alemi, A., & Zoccolan, D. (2025) · Patterns · doi:10.1016/j.patter.2024.101149 · Cited by 3
“…of deep CNNs to achieve virtually perfect training accuracy in classification tasks where images are randomly associated to arbitrary category labels 99 —an impressive feat that, however, points to a fundamental difference from biological vision in terms of the ability to learn abstract category…”
Efficient deep neural network training via decreasing precision with layer capacity
Shen, A., Lai, Z., Sun, T., Li, S., Ge, K., Liu, W., & Li, D. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-40669-3 · Cited by 7
The Hidden Influence of Latent Feature Magnitude When Learning with Imbalanced Data
Dablain, D. A., & Chawla, N. V. (2025) · Lecture notes in computer science · doi:10.1007/978-981-97-8702-9_6 · Cited by 2
An Empirical Study of Named Entity Recognition for Electronic Health Records
Manoonwong, N., Muresan, A., Raavi, A. R., & Zhu, X. (2024) · IEEE International Conference on Knowledge Graph (ICKG) · doi:10.1109/ickg63256.2024.00036 · Cited by 2
Building the Foundations: Effective Approaches in Mathematics Teacher Preparation
Olawale, B. E. (2025) · Open Books and Proceedings · doi:10.38140/obp3-2025 · Cited by 2
Universal Training of Neural Networks to Achieve Bayes Optimal Classification Accuracy
Naeini, M. T., Bereyhi, A., Noshad, M., Liang, B., & Hero, A. O. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890777 · Cited by 2
Fault-GSA: High Generalization 3-D Fault Detection Method Based on Sparse Annotations
Zhou, G., Li, T., Li, K., Chu, S., & Zhu, X. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3557022 · Cited by 8
“Studies [23] show that despite label noise, CNNs can memorize all training data, potentially impairing generalization and causing overfitting.”
Adversarial Training with Maximal Coding Rate Reduction
Chu, H.-Y., Zhao, H., & Flierl, M. (2024) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/ieeeconf60004.2024.10942802 · Cited by 2
“The second limitation of the cross-entropy loss is that it can fit mislabeled labels [6].”
Adaptive Attention Based on Mixture Distribution for Zero-Shot Non-Line-of-Sight Imaging
Zhang, Q., Liu, J., & Duan, Y. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3558458 · Cited by 2
“The other is the generalization [25], e.g., the network trained on the dataset with a specific distribution may not perform well on datasets with unseen distributions.”
Arasteh, A. (2025) · Finance research letters · doi:10.1016/j.frl.2025.107403 · Cited by 2
DISTR: Detecting multi-stage IoT botnets through contextual traffic and causal analytics
Meng, F., Gui, J., Zou, F., Li, Y., & Wu, Y. (2025) · Computers & Security · doi:10.1016/j.cose.2025.104531 · Cited by 2
Robust Testing for Deep Learning using Human Label Noise
Lim, G., Larson, S., & Leach, K. (2025) · IEEE/ACM International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest) · doi:10.1109/deeptest66595.2025.00009 · Cited by 4
“In deep learning (DL) systems, label noise in training datasets often degrades neural network (NN) performance, as NNs may learn incorrect patterns from mislabeled data [1], [2].”
CNLA: Collaborative noisy label adaptive learning for facial expression recognition
Ye, J., Liu, D., Wang, C., Huang, H., Liang, Y. J., Zhang, L., & Jiang, A. (2025) · Information Sciences · doi:10.1016/j.ins.2025.122436 · Cited by 2
Hong, H., & Kim, H. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111597 · Cited by 3
Deep learning-based bone marrow cytology classification: A solution to class imbalance
Bonab, Z. S., Shamekhi, S., & Talebi, M. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108247 · Cited by 2
Ngwenya, S., & Garidzira, T. C. (2025) · Indonesian Journal of Computer Science · doi:10.33022/ijcs.v14i4.4831 · Cited by 2
FedGuCci: Making Local Models More Connected in Landscape for Federated Learning
Li, Z., Lin, J., Li, Z., Zhu, D., Shen, T., Lin, T., Wu, C., & Lane, N. D. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737037 · Cited by 2
Qiao, B., Kaplan, S., & He, J. (2025) · International Journal of Sustainable Transportation · doi:10.1080/15568318.2025.2545449 · Cited by 2
“Meanwhile, the relatively simple parameters ensuring prediction accuracy feasibility usually reduce the overfitting of samples and keep stable generalization performance compared with the complex parameters (C. Zhang et al., 2021).”
UNet-assisted parameterization for B-spline surface approximation
Tang, W., & Yang, Z. (2025) · Computers & Graphics · doi:10.1016/j.cag.2025.104416 · Cited by 2
Kumar, R., Mukherjee, S., & Choudhary, D. (2025) · Annals of Operations Research · doi:10.1007/s10479-025-06770-x · Cited by 3
“…majority of the machine learning (ML) and deep learning (DL) based task objectives have been prevalent in several applications such as sentiment analysis, recommender systems, fraud detection, fake review detection, product classification and image recogni - tion problems (Zhang et al., 2021 ).”
InfluentialBalancing explainability and privacy in AI systems: A strategic imperative for managers
Keaney, S., & Berthon, P. R. (2025) · Business Horizons · doi:10.1016/j.bushor.2025.10.004 · Cited by 2
NR-GCF: Graph Collaborative Filtering with Improved Noise Resistance
Chen, Y., Li, B., Li, Y., Song, L., Wang, H., Wu, W., Zhuo, J., & Yin, H. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761342 · Cited by 2
“The memorization effect in deep neural networks (DNNs) originated from studies on generalization in the presence of noisy data[36].”
Tetteh, F. K., Degbe, F. D., Attah, J., Awumah, E. K., & Nyamekye, B. (2025) · International Journal of Quality & Reliability Management · doi:10.1108/ijqrm-07-2024-0228 · Cited by 4
Hollister, J. D., Paz‐García, D. A., Beas‐Luna, R., Horton, T., Cai, X., & Fenberg, P. B. (2025) · Scientific Reports · doi:10.1038/s41598-025-30613-1 · Cited by 2
“Systematic experiments demonstrated that convolutional networks can easily fit random labelling of training data, achieving near-perfect training accuracy even when no meaningful relationship exists between images and labels 25 .”
InfluentialKim, H. Y., Yi, T. J., & Lee, J. Y. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3548240 · Cited by 2
“If the training accuracy is high and the testing accuracy is low, the model will be overfitted to the training data, and the overfitted model will be overly adapted to the characteristics and noise of the training data, resulting in poor generalization performance [35], [38].”
BioDeepHash: Generating Consistent Templates for Secure Biometric Recognition
Song, B., Zhao, D., Yan, J., Li, H., & Jiang, H. (2026) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2026.3660294 · Cited by 4
Chatterjee, M. J. V., Chan, A. D. C., & Komeili, M. (2026) · Scientific Reports · doi:10.1038/s41598-026-41549-5 · Cited by 2
“Previous studies have demonstrated that when the number of parameters exceeds the number of samples in the dataset, a statistical model can fit unstructured random noise 30 .”
Litrico, M., Talon, D., Battiato, S., Bue, A. D., Giuffrida, M. V., & Morerio, P. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02683-1 · Cited by 2
High-entropy advantage in neural networks' generalizability
Yang, E., Zhang, X., Shang, Y., & Zhang, G. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-026-00100-7 · Cited by 2
Towards a statistical understanding of neural networks: Beyond the neural tangent kernel theories
Zhang, H., Lai, J., Li, Y., Lin, Q., & Liu, J. S. (2026) · Statistical Learning and Data Science · doi:10.1016/j.slads.2026.100026 · Cited by 4
Reducing global reductions in large-scale distributed training
Cong, G., Yang, C.-C., & Zhou, F. (2019) · International Conference on Parallel Processing Workshops (ICPPW) · doi:10.1145/3339186.3339203 · Cited by 2
An Empirical Study of Neural Networks for Trend Detection in Time Series
Miot, A., & Drigout, G. (2020) · SN Computer Science · doi:10.1007/s42979-020-00362-1 · Cited by 2
Analyzing Deep Neural Networks with Noisy Labels
Lim, C. H., Han, S., & Lee, J. (2020) · IEEE International Conference on Big Data and Smart Computing (BigComp) · doi:10.1109/bigcomp48618.2020.00012 · Cited by 2
Para-functional engineering: cognitive challenges
Vallverdú, J. (2022) · International Journal of Parallel Emergent and Distributed Systems · doi:10.1080/17445760.2022.2047678 · Cited by 2
“Surprisingly, contemporary machine learning approaches, such as deep nets, generalize well, despite interpolating noisy data [35,36].”
BackgroundSynergistic Network Learning and Label Correction for Noise-Robust Image Classification
Gong, C., Bin, K., Seibel, E. J., Wang, X., Yin, Y., & Song, Q. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747470 · Cited by 2
Over-precise Predictions Cannot Identify Good Choice Models
Sifar, A., & Srivastava, N. (2022) · Computational Brain & Behavior · doi:10.1007/s42113-022-00146-1 · Cited by 3
“In fact, as recent work in statistical learning demonstrates, it is possible for sufficiently complex models to empirically generalize even systematic functional mappings from noise to random labels (Zhang et al., 2021).”
BackgroundPredicting Entrepreneurship Learning Factors on Entrepreneurial Intent in Student Universities
Arni, Y., SİSWANDARİ, S., Akhyar, M., & ASROWİ, A. (2022) · Journal of Hunan University Natural Sciences · doi:10.55463/issn.1674-2974.49.6.26 · Cited by 3
“Creativity theory defines four determinants of creativity (4P): process, product, person, and place [39].”
BackgroundLearnability of the Boolean Innerproduct in Deep Neural Networks
Erdal, M., & Schwenker, F. (2022) · Entropy · doi:10.3390/e24081117 · Cited by 3
“However, how many hidden layers and units per hidden layer are needed to solve a particular task is generally unknown [6].”
BackgroundA Survey of Convolutional Neural Network and Its Variants
Song, H., & Wen, Y. (2022) · International Conference on Computer and Communications Management (ICCCM) · doi:10.1145/3556223.3556229 · Cited by 2
“Moreover, based on a study by Zhang et al. [21], sometimes the model can retain a near convergent training loss even when there is no explicit regularizes at all.”
MethodTree-based Text-Vision BERT for Video Search in Baidu Video Advertising
Yu, T., Liu, J., Yang, Y., Li, Y., Fei, H., & Li, P. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020922 · Cited by 2
Cao, Y., He, L., Wu, Z., & Dai, X. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191265 · Cited by 2
Hartmann, C., & Richter, L. (2023) · AI critique · doi:10.1515/9783839457320-004 · Cited by 3
“…of parameters that is independent of the dimension (Barron, 1993, Thm. 3); there are, moreover, results that show that it is possible to express any labelling of N data points in R d by an ANN two layers and in total p = 2 N + d parameters (Zhang et al., 2021, Thm. 1); cf. (DeVore et al., 2021).”
Chasles, S., & Major, F. (2023) · Frontiers in Genetics · doi:10.3389/fgene.2023.1254226 · Cited by 2
Ong, S., Isawasan, P., Gomesh, N., Salleh, K. A., & Yusof, U. K. (2023) · International Conference on Artificial Intelligence and Data Sciences (AiDAS) · doi:10.1109/aidas60501.2023.10284682 · Cited by 2
Evaluating Regression Models with Partial Data: A Sampling Approach
Pourkamali‐Anaraki, F., & Hariri‐Ardebili, M. A. (2023) · International Conference on Control, Decision and Information Technologies (CoDIT) · doi:10.1109/codit58514.2023.10284201 · Cited by 2
Zambare, P., & Liu, Y. (2023) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-45882-8_2 · Cited by 2
SleepLess: personalized sleep monitoring using smartphones and semi-supervised learning
Mammen, P. M., Zakaria, C., & Shenoy, P. J. (2023) · CSI Transactions on ICT · doi:10.1007/s40012-023-00389-8 · Cited by 2
“In understanding self-training, Zhang et al. reported error accumulation as a potential problem [48].”
Predicting peak deviatoric stress of gravels with a novel segmented feature transformation approach
Li, D., Zou, D., Liu, J., Xu, K., Ning, F., Zhan, Z., & Jin, W. (2023) · Computers and Geotechnics · doi:10.1016/j.compgeo.2023.105935 · Cited by 2
Bed Topography Inference from Velocity Field Using Deep Learning
Oshtorjani, M. K., & Ancey, C. (2023) · Water · doi:10.3390/w15234055 · Cited by 2
Nguyen, T. T. H., & Okazaki, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.573 · Cited by 2
“…a challenge for VQA models to achieve Out-of-Distribution (OOD) generalization, which involves performing well on data beyond the training distribution instead of relying on independent and identically distributed (iid) data [Zhang et al., 2021, Goyal and Bengio, 2022, Kawaguchi et al., 2022].”
MethodAIGenC: AI Generalisation via Creativity
Cătărău-Cotuţiu, C., Mondragón, E., & Alonso, E. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49011-8_4 · Cited by 4
Multi-head Siamese Prototype Learning against both Data and Label Corruption
Zhang, P.-F., & Huang, Z. H. (2023) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3595916.3626435 · Cited by 4
Improving deep PROPELLER MRI via synthetic blade augmentation and enhanced generalization
Saju, G., Li, Z., & Chang, Y. (2024) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2024.01.017 · Cited by 2
Dynamic Loss Decay Based Robust Oriented Object Detection on Remote Sensing Images with Noisy Labels
Liu, G., Liu, T., Yuan, M., Pang, T., Yang, G., Fu, H., Wang, T., & Liao, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78110-0_27 · Cited by 2
Early learning of the optimal constant solution in neural networks and humans
Rubruck, J., Bauer, J. M., Saxe, A., & Summerfield, C. (2025) · Conference on Cognitive Computational Neuroscience (CCN) · doi:10.32470/04kqim9g · Cited by 6
Local and Adaptive Mirror Descents in Extensive-Form Games
Fiegel, C., Ménard, P., Kozuno, T., Munos, R., Perchet, V., & Valko, M. (2024) · neural information processing systems · doi:10.52202/079017-1806 · Cited by 2
Decorrelating neurons using persistencePreprint
Ballester, R., Casacuberta, C., & Escalera, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.04870 · Cited by 2
“According to experimental results, explicit regularisation may improve generalisation performance [47].”
Background
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.