Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 2,201–2,300 of 2,667
Exploring the Use of Time Series Foundation Model for Car-Following Behavior Analysis
Zeng, L., & Yan, R. (2025) · ASCE International Conference on Transportation and Development (ICTD) · doi:10.1061/9780784486191.026
Voices from the Lungs: An Innovative Approach to Asthma Diagnosis using Machine Learning
Gezer, M., Uysal, M. A., Alagöz, N., Ortak, C., & Niksarlıoğlu, E. Y. Ö. (2025) · Acta Infologica · doi:10.26650/acin.1676351 · Cited by 1
A multi-model dynamic filtering of label noise for regression
Jiang, G., Lei, F., & Wang, W. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111887 · Cited by 2
Implications From the Analogous Relationship Between Evolutionary and Learning Processes
Leong, J. C. K., Imaizumi, M., Innan, H., & Irie, N. (2025) · BioEssays · doi:10.1002/bies.70027 · Cited by 1
“In more modern machine learning, especially deep learning, this understanding of overlearning is further developed [24, 25].”
Capítulo 1: Reconocimiento de vehículos usando Deep Learning
del Quindío, U., Valbuena, S. J., Triviño, J. I., & Hurtado, J. I. M. (2025) · doi:10.59899/ges-cono-75-c1
Capítulo 18: Reconocimiento de Melanoma y Nevus Melanocítico mediante la aplicación de ResNet50
del Quindío, U., Valbuena, S. J., Triviño, J. I., & Hurtado, J. I. M. (2025) · doi:10.59899/ges-cono-75-c18
Low-rank perturbation adjustment (LoPA): An implicit regularization method in image classification
Mahmoud, N. T. A., Antson, H., Chan, W. T., Sulaiman, M., Choi, J., Shimmi, O., & Roy, K. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-025-20941-9
“The research of regularization is mature and yet active ranging from implicit regularization induced by gradient-based optimization in deep learning [ 51] to norm regularization [ 52, 53].”
AI and the Symbiotic HR Redefining Human-Machine Collaboration in the Future Workplace
Akter, Y., Hamid, K. A., Akter, F., Rahman, M. M., Shakil, M. R. U., & Azizan, N. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-8669-9.ch001
Foster noisy label learning by exploiting noise-induced distortion in foreground localization
Chen, A., Xu, F., Lyu, X., Zeng, T., & Li, X. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107712
Balaram, S., Nguyen, M. C., Yu, Y., Mien, I. H., Yang, H., Vanjavaka, N., Rahim, N. H., Koh, A. G. C., Low, S., Kumar, A. J. S., et al. (2026) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2026.3696868
Ma, Y., Wang, W., Wang, S., Yang, X., & Yan, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6606-5_14
Inducing Neural Collapse via Anticlasses and One-Cold Cross-Entropy Loss
Katsikas, D., Passalis, N., & Tefas, A. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3580892
“Despite these limitations, neural networks can generalize remarkably well [10], and understanding the underlying reasons has been the focus of extensive research [11], [12].”
FedNDA: Enhancing Federated Learning with Noisy Client Detection and Robust Aggregation
Kieu, T. D., Fonbonne, C., Tran, T. K., Le, T., Vu, H., Nguyen, H.-T., & Tran, T.-H. (2025) · EAI Endorsed Transactions on Industrial Networks and Intelligent Systems · doi:10.4108/eetinis.v12i3.8720
“Zhang et al. [15] demonstrated that deep learning models are susceptible to overfitting.”
Outlier-trimmed dual-interval smoothing loss for sample selection in learning with noisy labels
Hou, S., Xu, M., Jiang, G., Guo, Y., & Wang, W. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107827
de Araújo, R. D. N. (2025) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.3.2025.tde-11072025-135111
Learning from Noisy Labels for Long-Tailed Data via Optimal Transport
Li, M., & Zhu, C. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6594-5_17
Cui, S., Zhao, S., Gong, L., Xia, W., Yang, Y., Sang, B., Li, T., & Chen, X. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9875-2_30
Hiebl, B., Alessi, N., Calvia, G., Bricca, A., Bonari, G., Zangari, G., Dorigo, W. A., Zerbe, S., & Rutzinger, M. (2025) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2025.104734 · Cited by 6
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.002.0005
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.002.0007
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0007
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0001
Taxonomies of Cognitive Architectures
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0004
Practical Applications of Cognitive Architectures
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0010
Cognitive Architectures, AI, and Cognitive Science
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0003
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.002.0004
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0008
Challenges of the Past and Opportunities Ahead
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0013
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0006
Cognitive Architectures in the Deep Learning Era
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0012
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0009
What Are Cognitive Architectures?
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0002
Evaluating Cognitive Architectures
Kotseruba, I., & Tsotsos, J. K. (2025) · Oxford University Press eBooks · doi:10.1093/oso/9780192844835.003.0011
Learning from Limited Data via Generating and Fine-Tuning
Zhang, Y., Dong, H., Shi, H., He, J. S., Zhang, X., & Li, M. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9894-3_16
Nazarian, M., Neves, R., Klick, L., Schöfer, F., Lau, R., Seibel, A., & Weigand, F. (2025) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-025-16118-1 · Cited by 1
Weight Pruning to Mitigate Class-Specific Accuracy Degradation for LiDAR-Based 3D Object Detection
Ito, T., Hirakawa, T., Yamashita, T., & Fujiyoshi, H. (2025) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv64158.2025.11097784
“Among them, 3D object detection models based on neural networks generally achieve higher performance as the number of weights increases [5], [6].”
Deep Learning-Based Techniques for Detection of Cancer
Sompura, D. U., & Tripathy, B. K. (2025) · Springer eBooks · doi:10.1007/978-981-96-8176-1_17 · Cited by 1
Sohrfiroozani, M. S., & Ghahdarijani, R. M. (2025) · Journal of the Indian Chemical Society · doi:10.1016/j.jics.2025.102000
Kifle, T., & Sintayehu, G. (2025) · Journal of Water and Climate Change · doi:10.2166/wcc.2025.024
Wormhole Dynamics in Deep Neural Networks
Lai, Y.-L., & Jin, Z. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3591614
“Despite these optimization complexities [9], gradient-based methods frequently succeed in locating global minimizers, achieving zero or near-zero training loss even when data and labels are randomized before training [5].”
Towards Effective and Robust Unlearnable Examples Against Object Detection
Yi, C., Meng, R., Peng, H., Shen, B., & Kot, A. C. (2025) · Proceedings - International Conference on Image Processing · doi:10.1109/icip55913.2025.11084567 · Cited by 1
Unified pipeline for generalized mental state detection using EEG signals
Wang, Y., Elrawas, R., Nguyen, A.-D., Girard, M., Mozharovskyi, P., Tartaglione, E., & Nguyen, V.-T. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129422
Ya, L., Arifin, S. R. M., Pien, L. S., & Zain, S. H. M. (2025) · Journal of Psychosomatic Obstetrics & Gynecology · doi:10.1080/0167482x.2025.2541946
“However, most of the current research on non-directive counselling focuses on the intervention effects of a single component, such as the role of trust-building and emotional support in counselling relationships [16].”
Research on Data Construction Method of Entity Relation Instruction for Biological Invasion
Wu, Q., Liu, J., Fu, M.-Y., Yan, Y., & Wang, L. (2025) · International Symposium on Bioinformatics and Computational Biology (BCB) · doi:10.1145/3747357.3747387
Asymptotic generalization error in the online learning of linearized two-layer neural networks
Worschech, R., & Rosenow, B. (2025) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/adf822
Reconciling functional data regression with excess bases
Wakayama, T., & Matsui, H. (2025) · Behaviormetrika · doi:10.1007/s41237-025-00275-4
“Indeed, Zhang et al. ( 2021) empirically showed that deep neural network models with a large number of parameters that perfectly fit the training data can yield near-optimal accuracy for the test data.”
A Comparative Study of Deep Learning Models for Sentiment Analysis in Online Learning Environments
Ouadad, R., & Mouncif, H. (2025) · International Conference on Circuit, Systems and Communication (ICCSC) · doi:10.1109/iccsc66714.2025.11135038
Bao, Z., Zhang, H., Schwendner, M. J., Schröder, A., Meyer, B., Krieg, S. M., & Ille, S. (2025) · Human Brain Mapping · doi:10.1002/hbm.70344
Time-frequency Domain Joint Precoding based on the LSTM Network and Repetitive HSR Channel
Zhu, M., Duan, Q., Liu, G., & Xue, J. (2025) · IEEE/CIC International Conference on Communications in China (ICCC) · doi:10.1109/iccc65529.2025.11148679
Transformers and State-Space Models: Fine-Tuning Techniques for Solving Differential Equations
Ignatenko, V., Surkov, A., Zakharov, V., & Koltcov, S. (2025) · Sci · doi:10.3390/sci7030130
“However, excessive adaptation during fine-tuning may lead to overfitting , a phenomenon in which the model captures noise or spurious patterns in the training data, thereby reducing its generalization ability to unseen inputs [5,6].”
Ullah, N. G., Al-Thelaya, K., Majeed, F., Lu, Z., Boughorbel, S., Schneider, J., & Agus, M. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00194
“This includes noisy labels and typically leads to poor generatiza-tion [2, 55, 65, 71].”
Streaming View Classification With Noisy Label
Ouyang, X., Fan, R., Tao, H., & Hou, C. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3607610
“A common approach to modeling label noise conditional independence between the corruption process and data features given the true labels [20], [21].”
Adebayo, M. (2025) · International Journal of Latest Technology in Engineering Management & Applied Science · doi:10.51583/ijltemas.2025.1408000109
Regularisation and the least squares problem in the analysis of echo state networks
Lu, Y., & Winkler, J. R. (2025) · Physica Scripta · doi:10.1088/1402-4896/ae082f
al., A. E., Altameemi, Y., Alkhalil, A., Aljaloud, A., Abdelrhman, M., Uliyan, D., Ahmad, A., Alogali, A., & Razzaq, A. (2025) · International Journal of ADVANCED AND APPLIED SCIENCES · doi:10.21833/ijaas.2025.10.001 · Cited by 1
Medical Artificial Intelligence
Grote, T. (2025) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197625835.013.0033
CPL: consistent prompt learning for noisy label facial expression recognition
Zhang, J., Wang, Q., Li, J., Li, Z., Zhang, X., Zhao, P., Yin, Y., & Lu, H. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02802-x
A Neuron Splitting Algorithm with Weight Conservation for Adaptive Neural Networks
Abramova, E. S., & Orlov, A. (2025) · International Russian Automation Conference (RusAutoCon) · doi:10.1109/rusautocon65989.2025.11177446
Faster margin maximization rates for generic and adversarially robust optimization methods
Wang, G., Hu, Z., Gentile, C., Muthukumar, V., & Abernethy, J. (2025) · Mathematical Programming · doi:10.1007/s10107-025-02283-4
“Yet, even unregularized first-order optimization methods are observed to converge to solutions that generalize well to test data, as multiple empirical studies have repeatedly confirmed (e.g., Zhang et al. [39], Neyshabur et al. [22]).”
Seal Your Backdoor with Variational Defense
Sabolić, I., Grcić, M., & Šegvić, S. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00078 · Cited by 3
Noise-Robust Learning via Full Consistency
Wang, Z., Chang, X., Zhao, W., Yu, W., Lei, X., & Dong, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3453-1_19
Hybrid Deep Learning and Lee-Carter Model for Mortality Forecasting: A Study of US Adults Aged 35-80Preprint
Abaitey, C., Abakah, E., Asare, C., Oppong, E., Baba-Sadiq, F. M., & Appiah, E. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.10.13.682203
“Machine learning models, including LSTM, Bi-LSTM, and neural networks, are particularly sensitive to data quality issues such as noise and outliers [34].”
Trustworthy Machine Learning with Noisy Labels
Han, B., & Liu, T. (2025) · Springer eBooks · doi:10.1007/978-981-96-9396-2_2
Mohammadagha, M., Bigdeli, F., Sharifi, S., Deldadehasl, M., & Ataei, S. (2025) · doi:10.31224/5632
Information-Geometric Analysis of the Optimal Error Exponent in Fixed-Length Hypothesis Testing
Zhang, Q., Tong, X., Peng, T., & Huang, S. (2025) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit63088.2025.11195304
“The learning process often relies on computable forms of the observed data, including low-dimensional features, gradients, and empirical risks [13], which takes the form of Motivated by practical requirements, the goal of this work is to study the optimal decision error exponent of the fixed-length…”
Ensemble Mirror Descent: a Generalization of Mirror Descent to Multiple Convex Potentials
Varma, K. N., & Hassibi, B. (2025) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit63088.2025.11195656
The Computational Complexity of Satisfiability in State Space Models
Alsmann, E., & Lange, M. (2025) · Frontiers in artificial intelligence and applications · doi:10.3233/faia251006
Peñalosa, C. P., Adonis, N. R., Albao, E. J. A., Malto, J. L., & Caidoy, R. O. P. (2025) · World Journal of Advanced Research and Reviews · doi:10.30574/wjarr.2025.28.1.3597
Toward Sustainable Continual Learning: Detection and Knowledge Repurposing for Reoccurring Tasks
Wang, S., Choi, Y., Chen, J., El‐Khamy, M., & Henao, R. (2025) · Machine learning for signal processing ... · doi:10.1109/mlsp62443.2025.11204265
“Further, [7] showed that overparameterized neural network classifiers can attain zero training error, regardless of the strength of association between images and labels, and in an extreme case, even for random labels.”
Tzilantonis, G. T., Zafeiriou, E., Stimoniaris, A., Kanapitsas, A., & Tsanaktsidis, C. G. (2025) · Resources · doi:10.3390/resources14110167
“Studies on polyacrylamide (PAM) and related hydrogels have demonstrated their strong capacity for water adsorption through hydrogen bonding and polar interactions, offering a low-cost, low-energy approach to moisture extraction [18,19].”
Nonparametric optimality for large compressible deep neural networks under quadratic loss functions
Shaogao, L., Liu, W., & Xing-cai, Z. (2025) · Scientia Sinica Mathematica · doi:10.1360/scm-2024-0536
Training Robust DNNs with Noisy Labels via Contrastive Re-Calibration Learning
Dong, Y., Wang, J., Wang, Z., Wu, G., & Cheng, H. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209815
Learning from Noisy Data Using Pretrained Vision-Language Representations
Liao, Y., Li, A., Chen, Y., Xu, Q., Xie, J., Li, A., & Xiao, B. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11208949
“But this same expressivity makes them vulnerable to noisy or erroneous data labels, as they tend to memorize the noise, thereby compromising their generalization performance [4].”
Digital Tourism Promotion: Social Media, E-WOM, and Celebrity Endorsement in Jakarta
Widayati, C., Malihah, F. D., & Halim, H. A. (2025) · Jurnal Manajemen · doi:10.24912/jm.v29i3.2706
“Research by Zhang et al. (2021) shows that celebrities can influence consumer perception and intent through their appeal and the credibility they bring.”
On the Efficiency of ERM in Feature Learning
Hanchi, A. E., Maddison, C. J., & Erdogdu, M. A. (2024) · neural information processing systems · doi:10.52202/079017-3129
Beyond Limits: A Mathematical Information-Theoretic Framework Redefining AI Generalization Bounds
Mann, M. M. M. (2025) · International Conference on Engineering Management of Communication and Technology (EMCTECH) · doi:10.1109/emctech65814.2025.11220578
Liu, B., Zhao, J., & Lu, F. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3629399
“(cid:2) Multi-channel Deep Convolutional Neural Network (MCDCNN) [54]: This model processes each signal channel separately through Conv1D layers with ReLU activations and max pooling.”
InfluentialLabeler-Dependent Noise with Adversarial Awareness
Tufo, V. A. D., Dawson, G., & Robi Polikar, A. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228212
Classification of Hope in Textual Data using Transformer-Based ModelsPreprint
Ijezue, C. F., Eneye, T.-A. N. F., & Amjad, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.12874
“The small validation-test performance gap suggests good generalization to unseen data, addressing common concerns about overfitting in deep learning [41].”
FedDFS: Data filtering for stable federated learning
Zha, X., & Shen, D. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114934
Dhar, S., Fong, K. S., & Motani, M. (2026) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3795095.3805148
“An alternative approach is to enforce a global smoothness constraint by regularizing the network’s Lips-chitz constant, often approximated by penalizing the spectral norm of the weight matrices [2, 11, 34].”
Zhang, B., Li, Y., Peng, R., Xue, Y., Wang, Q., & Xu, C. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5804583
Membership Inference Attacks Beyond Overfitting
Khalil, M., Blanco-Justicia, A., Jebreel, N., & Domingo‐Ferrer, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-16089-8_3
Deep Neural Networks for Multivariate Granger-Causal Feature Extraction
Kumar, A., Chivukula, A. S., & Kumar, M. (2025) · IEEE Access · doi:10.1109/access.2025.3638657
“Algorithm 6 outlines the methodology for implementing (14) and (13) using wavelet decomposition, as defined in (19).”
Improving Few-Shot Classification via Feature-Aligned AI-Generated Images
Tung, Y.-W., & Yeh, M.-C. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249102
“Despite advances, traditional deep models often suffer from limited generalization beyond the domains on which they were trained, partly due to overfitting and dataset-specific biases [12].”
Energy-Aware Deep Learning on GPUs through Parameter Sharing and Mixed Precision Training
Tchakoute, R. N., & Tadonki, C. (2025) · International Symposium on Computer Architecture and High Performance Computing Workshops (SBAC-PADW) · doi:10.1109/sbac-padw69789.2025.00024
“Cutting-edge DL models, particularly in Natural Language Processing (NLP), are often heavily overparameterized [6].”
Li, H., & Raghavan, N. (2025) · IEEE International Symposium on the Physical and Failure Analysis of Integrated Circuits (IPFA) · doi:10.1109/ipfa65338.2025.11256700
Murota, K., & Kobori, T. (2025) · IEEE International Conference on Quantum Computing and Engineering (QCE) · doi:10.1109/qce65121.2025.00171
On the Effect of Regularization on Nonparametric Mean-Variance RegressionPreprint
Wong‐Toi, E., Boyd, A., Fortuin, V., & Mandt, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.22004
“Most notably, we show that the unregularized maximum likelihood regime admits no finite stationary solution, providing a mathematical explanation for the extreme overconfidence observed when the mean network interpolates the data [Zhang et al., 2021].”
Larwood, B., Sutton, O. J., & Cockburn, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.03519
“On a broader level, untangling the interplay between the processes of memorisation (when an AI system learns to act like a lookup table) and generalisation beyond the model’s training data in deep learning systems remains a challenging open problem [Zhang et al., 2021].”
Bakhouya, M., & Ramchoun, H. (2025) · International Conference on Intelligent Systems: Theories and Applications · doi:10.1109/sita67914.2025.11273763
Research on UAV swarm effectiveness evaluation method based on deep learning
Chi, H., Wang, L., Li, R., Tang, H., Xu, M., Cao, Z., & You, L. (2025) · International Symposium on Aerospace Engineering and Systems (ISAES) · doi:10.1109/isaes66870.2025.11274275
“By adding the weight square term to the loss function to limit the size of the parameter, prevent a feature from affecting the model too much [7] , the formula is L = L + ∑ w (5) Among them, is the regularization coefficient, control the penalty intensity ; is the square sum of the weight of…”
Why Not Diversify Triggers? APK-Specific Backdoor Attack Against Android Malware Detection
Li, H., Wu, B., Yao, Z., Gao, C., Yuan, W., Xia, B., & Luo, X. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3642903
“To ensure robust generalization [3] and mitigate overfitting [4], such AMD models often necessitate a sufficient volume of training data.”
Robust Image Classification via Centroid-Aware Machine Unlearning of Noisy Annotations
Yamamoto, S., Yoshida, S., & Muneyasu, M. (2025) · IEEE Global Conference on Consumer Electronics (GCCE) · doi:10.1109/gcce65946.2025.11275131
FedDNL: Federated differentiated noisy learning for label noise heterogeneity
Jiao, Y., He, D., Li, X., Su, Y., Wang, Z., & Gu, H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.115085 · Cited by 2
Layer-wise Quantization for Quantized Optimistic Dual AveragingPreprint
Nguyen, A. D., Markov, I., Wu, F., Ramezani-Kebrya, A., Antonakopoulos, K., Alistarh, D., & Cevher, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.14371
“This is formally captured by the notion of relative noise (Polyak, 1987): Relative noise model has been studied in several ML application like over-parameterization (Oymak & Soltanolkotabi, 2020), representation learning (Zhang et al., 2021), and multi-agent learning (Lin et al., 2020).”
Weakly Supervised Nuclei Segmentation with Point Annotations and Noise Robust Loss
Nan, M., Zhang, Y., Li, J., & Liang, G. (2025) · International Conference on Bioinformatics and Biomedical Technology (ICBBT) · doi:10.1109/icbbt65815.2025.11276366
PASS: Peer-agreement based sample selection for training with instance dependent noisy labels
Garg, A. K., Nguyen, C. C., Felix, R., Do, T.-T., & Carneiro, G. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105877
Regularization in Neural Networks: A Survey and Empirical Analysis of Approaches
Opperman, C. P., Bosman, A. S., & Malan, K. M. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3644334
Robustness of Probabilistic Models to Low-Quality Data: A Multi-Perspective AnalysisPreprint
Liu, P., & Jin, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.11912
“…models, this is a well-established field; the surprising resilience of deep classifiers to label noise is well-documented (Rolnick et al., 2018; ZhangChiyuan et al., 2021), leading to an ecosystem of solutions, from noise-robust loss functions (Menon et al., 2019; Chen et al., 2020) to…”
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.