Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 2,001–2,100 of 2,667
Gómez-Ríos, A., Luengo, J., & Herrera, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03748
“In the specific case of deep learning, label noise has been proven to harm generalization when training deep neural networks [10], and thus there has been an increasing amount of studies trying to improve the behaviour of deep neural networks as much as possible in presence of label noise [4, 9, 11–13].”
BackgroundCausally Estimating the Sensitivity of Neural NLP Models to Spurious FeaturesPreprint
Zhang, Y., Pan, L., Tan, S., & Kan, M. (2021) · arXiv (Cornell University)
Conditional Gaussian PAC-Bayes.Preprint
Clerico, E., Deligiannidis, G., & Arnaud Doucet, B. (2021) · arXiv (Cornell University)
The Equilibrium Hypothesis: Rethinking implicit regularization in Deep Neural Networks.Preprint
Lou, Y., Mingard, C., & Hayou, S. (2021) · arXiv (Cornell University)
Salvi, R. (2021) · Nomos Verlagsgesellschaft mbH & Co. KG eBooks · doi:10.5771/9783748924012-285
Over-Parametrized Matrix Factorization in the Presence of Spurious Stationary Points
Eftekhari, A. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3139213 · Cited by 1
Back-propagation neural network assisted photonic real-time Fourier transformation system
Wang, Z., Zhu, D., Hu, X., & Pan, S. (2021) · Asia Communications and Photonics Conference · doi:10.1364/acpc.2021.m4e.3
Guo, J. W., Ding, M., Wang, Q., Zhang, C., & Yao, Y. (2022) · Circuits Systems and Signal Processing · doi:10.1007/s00034-021-01928-x
Self-citationTowards understanding deep learning with the natural clustering priorPreprint
Carbonnelle, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.08174
Learning with noisy class labels for pin defect detection in transmission lines
Liu, X., Li, W., Chang, W., Zhang, X., Han, Z., & Zhong, C. (2022) · International Conference on Image, Signal Processing and Pattern Recognition (ISPP) · doi:10.1117/12.2636841
Knowing When You Don’t Know in Online Fashion: An Uncertainty-Aware Size Recommendation Framework
Bahuleyan, H., Lasserre, J., Lefakis, L., & Shirvany, R. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-3-030-94016-4_3 · Cited by 1
PACMAN: PAC-style bounds accounting for the Mismatch between Accuracy and Negative log-loss
Vera, M., Vega, L. J. R., & Piantanida, P. (2024) · Information and Inference A Journal of the IMA · doi:10.1093/imaiai/iaae002
Certifying Data-Bias Robustness in Linear RegressionPreprint
Meyer, A. P., Albarghouthi, A., & D’Antoni, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.03575 · Cited by 3
“Poisoning attacks, where an attacker can modify a small portion of the training dataset, can be effective at reducing the accuracy of models at test-time [5, 32, 38, 37].”
BackgroundPlaat, A. (2022) · Springer eBooks · doi:10.1007/978-981-19-0638-1_10
A Deep Learning Approach for Robust Detection of Bots in Twitter Using Transformers Model
Pazhanivel, M. K., Ajai Kumar., B., Mageshwaran., M., & Dhivakar., K. (2022) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-4610
A Frobenius Norm Regularization Method for Convolutional Kernel Tensors in Neural Networks
Guo, P.-C. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/3277730 · Cited by 2
“*erefore, when training deep convolutional neural networks, if the singular values of the Jacobian of each layer are close to 1, it can help to avoid the exploding/vanishing gradient problem [4, 8–10] and improve the generalizability of a neural network [7, 11, 12].”
BackgroundDeep Learning Systems Integrated into the Digital Strategy of a Company Involved in e-commerce
Rusu, R., & Avram, C. (2022) · Annals of Dunarea de Jos University of Galati Fascicle I Economics and Applied Informatics · doi:10.35219/eai15840409238
Privacy Leakage in GAN Enabled Load Profile SynthesisPreprint
Huang, J., & Wu, C. (2022) · doi:10.36227/techrxiv.21341562.v1
Privacy Leakage in GAN Enabled Load Profile SynthesisPreprint
Huang, J., & Wu, C. (2022) · doi:10.36227/techrxiv.21341562
Utilizing Excess Resources in Training Neural Networks
Henig, A., & Giryes, R. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897994
Predicting Text-To-Speech Quality using Brain Activity
Rhenaldy, Karenza, L. S., Parmonangan, I. H., & Kurniadi, F. I. (2022) · IEEE International Conference on Internet of Things and Intelligence Systems (IoTaIS) · doi:10.1109/iotais56727.2022.9975857
Epoch-Wise Double Descent Triggered by Learning a Single Sample
Kawaguchi, A., Kera, H., & Yamasaki, T. (2022) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipaasc55919.2022.9980025
Harnessing Uncertainty - Multi-label Dysfluency Classification with Uncertain Labels
Jouaiti, M., & Dautenhahn, K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20980-2_26
Diminishing Empirical Risk Minimization for Unsupervised Anomaly Detection
Wang, S., Liu, Y., Chen, L., & Chengqi Zhang, B. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892076
“One of the obvious reasons is that DNN are often over-parameterized and designed with non-linear activation function between layers, which makes DNN an universal approximator [12] fitting well with not only normal data but also the anomalies [45].”
TOPS: Transition-Based Volatility-Reduced Policy Search
Xu, L., Lyu, D., Pan, Y., Jiang, A., & Liu, B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20179-0_1
Squeeze-Loss: A Utility-Free Defense Against Membership Inference Attacks
Zhang, Y., Yan, H., Lin, G., Peng, S., Zhang, Z., & Wang, Y. (2022) · Communications in computer and information science · doi:10.1007/978-981-19-7242-3_19
The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs
Theunissen, M. W., Mouton, C., & Davel, M. H. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-22321-1_6
“‘Toy problems’ such as these are used extensively to probe generalization [15, 23, 13].”
BackgroundRetroKD : Leveraging Past States for Regularizing Targets in Teacher-Student Learning
Jandial, S., Khasbage, Y., Pal, A., Krishnamurthy, B., & Balasubramanian, V. N. (2023) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3570991.3571014
“15 can be written as; with 𝑢 (·) implying that ∀ 𝑓 ∈ F the 𝑅 ( 𝑓 ) > 0 with equality when 𝑠 ( 𝑥 ) = 0 and the the 𝑐 > 0 [28, 44].”
Sampling matters: SGD smoothing through importance sampling
Zancato, L., & Chiuso, A. (2022) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc51059.2022.9992486
Deep learning with noisy labels: Learning True Labels as Discrete Latent Variable
Elhassouny, A., & Idbraim, S. (2022) · International Conference on Computational Intelligence and Communication Networks (CICN) · doi:10.1109/cicn56167.2022.10008239
Xu, Z., Fu, L., & Wang, X. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020399 · Cited by 1
Robust Learning with Noisy Label Detection and Counterfactual Correction
Qi, W., & Chelmis, C. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020228
Impartial competitive learning in multi-layered neural networks
Kamimura, R. (2023) · Connection Science · doi:10.1080/09540091.2023.2174079
“…point, we think that the impartiality can explain one of the phenomena inherent to neural networks, where even as a network size becomes larger in terms of components, generalisation performance does not necessarily decrease, contrary to our expectation (Arpit et al., 2017; Zhang et al., 2021).”
ResultLearning Category Distribution for Text Classification
Wang, X., & Zong, C. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3585279
“However, generalization of deep neural networks to noisy data is very harmful [43].”
BackgroundPerformance Assessment of Normalization in CNN with Retinal Image Segmentation
Kundalakkaadan, J., Rawat, A., & Kumar, R. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-19-6525-8_13
Adapting to Complexity: Deep Learnable Architecture for Protein-protein Interaction Predictions
Wu, J., Paquet, E., Viktor, H. L., & Michalowski, W. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25599-1_39
A Decoding Model of Upper Limb Movement Intention Based on Data Augmentation
Ke, X., Bi, L., Fei, W., & Feleke, A. G. (2022) · China Automation Congress (CAC) · doi:10.1109/cac57257.2022.10055468
Shen, X.-J., Ma, S., Lu, Z., & Xu, S. (2022) · 2022 4th International Academic Exchange Conference on Science and Technology Innovation (IAECST) · doi:10.1109/iaecst57965.2022.10062290
“Furthermore, inspired by the bootstrapping loss proposed in [39], we construct a complementary loss term based on the derived label correlations to hedge the penalty in the cross-entropy loss, when correlated labels were predicted by the classifier but missed by human annotators.”
MethodInfluentialLearning More with Less Data: Reaching the Power-Law Limits in Steganalysis Using Larger Batch Sizes
Benkhettou, M. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-28540-0_18
Enhancing Robustness for Facial Emotion Recognize
Qin, R., & Duan, Z. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-0416-7_42
A Simple Semi-Supervised Joint Learning Framework for Few-shot Text Classification
Lu, S., Chen, L., Wang, W., Xu, C. X., Zhao, W. Z., Guan, Z., & Lu, G. (2022) · doi:10.1145/3573942.3573945
“Directly using weakly-labeled data to train the deep model, the error caused by noise samples will accumulate in the process of backpropagation [11], which will affect the performance of the model [10].”
MethodSample Reweighting for Label Denoising of Neural Activity Data
Xu, D., & Chen, R. (2023) · International IEEE/EMBS Conference on Neural Engineering · doi:10.1109/ner52421.2023.10123809
“[3]–[6].”
Improving Variational Methods for Representation Learning and Generative Modelling of Speech
Braithwaite, D. T. (2023) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.23462480
A Stable Model for Learningand Memorization in Over-Parameterized Networks
Pandey, E., & Kumar, S. (2023) · Springer proceedings in mathematics & statistics · doi:10.1007/978-3-031-25194-8_3
Unpacking the black box of deep learning for identifying El Niño-Southern oscillation
Sun, Y. X., Habibulla, Y., Hu, G., Meng, J., Lu, Z., Liu, M., & Chen, X. (2023) · Communications in Theoretical Physics · doi:10.1088/1572-9494/ace17d
“When a deep learning model exhibits perfect performance in solving a speci fi c task, it implies that the model ’ s representation encodes the task ’ s intrinsic features [ 53 ] .”
Graph TopoFilter: A Method for Noisy Labels Detection for Graph-Structured Classes
Budniak, A., & Kajdanowicz, T. J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36024-4_51
“Deep neural networks are capable of fitting to the training set when it is tagged with true or entirely reshuffled labels [13], which is an impressive property, but also a drawback, when real data sets are considered.”
InfluentialThe advance of neural networks generalization performance
Chen, H. (2023) · Applied and Computational Engineering · doi:10.54254/2755-2721/5/20230711
Correlation Based Node Partitioning to Sparse Multilayer PerceptronPreprint
Benar, C., & Akansu, A. N. (2023) · doi:10.36227/techrxiv.23713962.v1
Correlation Based Node Partitioning to Sparse Multilayer PerceptronPreprint
Benar, C., & Akansu, A. N. (2023) · doi:10.36227/techrxiv.23713962.v2
Research on hard points height prediction method based on BP neural network
Yang, Q., Zhang, Q., Zhang, Z., Chen, D., Pan, M., Zhou, W., Ren, Y., Ma, H., & Liu, L. (2023) · International Conference on Automation Control, Algorithm and Intelligent Bionics (ACAIB) · doi:10.1117/12.2686683
“Hyper-parameterized deep feedforward neural networks have been used in many tasks [8-10] .”
MethodFerraz, L. C., Zamberlan, C., VASCONCELOS, J., Munhoz, O. L., & Ilha, S. (2023) · doi:10.22533/at.ed.9472315083
Caracterización de eventos microsísmicos: procesamiento y algoritmos
Brunini, G. I. (2023) · El Servicio de Difusión de la Creación Intelectual (National University of La Plata) · doi:10.35537/10915/156467 · Cited by 3
Huang, Z., Li, J., Yang, L., Guo, J., & Yao, Z. (2023) · Research Square · doi:10.21203/rs.3.rs-3302256/v1
Friend-training: Learning from Models of Different but Related Tasks
Zhang, M., Jin, L., Song, L., Mi, H., Zhou, X., & Yu, D. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.18
Combining Self-Supervised and Supervised Learning with Noisy Labels
Zhang, Y., Zhang, H., Yao, Q., & Wan, J. (2023) · Proceedings - International Conference on Image Processing · doi:10.1109/icip49359.2023.10221957
The Oil and Water Separation Phenomenon Inspired Loss for Feature Learning
Wang, Z., Yang, W., Liu, Z.-S., Ni, J., Chen, Q., & Jia, Z. (2023) · Proceedings - International Conference on Image Processing · doi:10.1109/icip49359.2023.10222467
“Moreover, empirical studies indicate that deep networks trained with CE are not robust to label noise [2].”
BackgroundMethodCollaborative Extreme Noise Classifier: A Multi-Network Approach Based Extreme Noise ClassificationPreprint
Guo, Z., Chen, C., Jiang, Z., & Wang, L. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4571141
Label-noise learning via mixture proportion estimation
ZHENG, Q., Cao, S., Ruan, J., Zhao, R., & Dong, B. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0126
Is My Neural Net Driven by the MDL Principle?
Brandão, E., Duffner, S., Emonet, R., Habrard, A., Jacquenet, F., & Sebban, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43415-0_11
“It is thus puzzling that, in spite of their implicit simplicity, NN classifiers trained by SGD are able to perfectly fit pure random noise [53], even while explicitly using regularization.”
Quantifying the Knowledge in Deep Neural Networks: An Overview
Valsamara, I., Mademlis, I., & Pitas, I. (2026) · ACM Computing Surveys · doi:10.1145/3845595
Texton Tri-alley Separable Feature Merging (TTSFM) Capsule Network for Brain Tumor Detection
Akoto-Adjepong, V., Appiah, O., Appiahene, P., & Mensah, P. K. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140951
“Area under the curve (AUC): The model's performance is assessed on datasets where classes are imbalanced or unevenly distributed by creating Receiver Operating Characteristic (ROC) and precision-recall (PR) curves [24]-[25].”
MethodBarros, P. H., Murai, F., & Ramos, H. S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45368-7_29
CTLC: A Co-Training-Based Label Correction Method for Webly-Supervised Learning
Lv, X., Li, M., Zhang, R., & Gao, L. (2023) · Neural Processing Letters · doi:10.1007/s11063-023-11425-8
“Because of the memory effect [8, 22], neural networks will gradually fit the noisy labels, which usually leads to poor model performance [23, 24].”
VRLVMix: Combating Noisy Labels with Sample Selection based on Loss Variation
Wang, H., Tu, Z., Jiang, B., & Ding, Y. (2023) · International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE) · doi:10.1109/icbaie59714.2023.10281331
“Research [2, 3] Neural Networks (DNNs) are susceptible to overfitting caused by noisy labels, which has a detrimental impact on model accuracy and generalization.”
BackgroundHardness of Deceptive Certificate Selection
Wäldchen, S. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44070-0_21
Ghane, P., & Braga-Neto, U. (2023) · Machine learning for signal processing ... · doi:10.1109/mlsp55844.2023.10285901
“Despite a substantial body of research, a thorough analysis of the underlying root factors that drive neural network generalization remains unanswered question [1, 2, 3, 4].”
BackgroundBayesian Deep Learning Detection of Anomalies and Failure: Application To Medical Images
Carannante, G., & Bouaynaya, N. C. (2023) · Machine learning for signal processing ... · doi:10.1109/mlsp55844.2023.10285922 · Cited by 3
TabMentor: Detect Errors on Tabular Data with Noisy Labels
Zhang, Y., Qin, J., Wang, Y., Ali, M. A., Ji, Y., & Mao, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-46671-7_12
Deterioration Level Estimation for Infrastructures Considering Noisy Labels via DivideMix
Kashiwagi, M., Maeda, K., Togo, R., Ogawa, T., & Haseyama, M. (2023) · IEEE Global Conference on Consumer Electronics (GCCE) · doi:10.1109/gcce59613.2023.10315319
da Cunha, A., d’Amore, F., & Natale, E. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.09858
“Much of the success of deep learning techniques relies on extreme over-parameterization [Zhang et al., 2017, 2021, Novak et al., 2018, Brutzkus et al., 2018, Du et al., 2019, Kaplan et al., 2020].”
Dual-Stage OOD Detection Learning with an Unsupervised Start
Kongmanee, J., Thanapattheerakul, T., & Chignell, M. (2023) · doi:10.1145/3628454.3628497
“[2 , 33] studied the learning dynamics of a softmax classifier trained by minimizing CE and found that differences in optimization procedures and the reliability of labels in the training data affected the degree of overfitting and generalizability of the associated models.”
BackgroundAI without networksPreprint
Mitra, P. P., & Sire, C. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.12.09.570922
Optional Hyperparameter Tuning of Convolutional Neural Network for ECG Classification
Akbar, M., Nurmaini, S., & Partan, R. U. (2023) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis60089.2023.10349032
Note on Abbreviated Citations to Historical Works
Buckner, C. J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197653302.002.0008
Buckner, C. J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197653302.003.0008
Buckner, C. J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197653302.003.0005
Buckner, C. J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197653302.003.0004
Buckner, C. J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197653302.002.0007
Buckner, C. J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197653302.003.0006
Buckner, C. J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197653302.003.0003
Buckner, C. J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197653302.002.0003
Buckner, C. J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197653302.003.0007
What Is Deep Learning, and How Should We Evaluate Its Potential?
Buckner, C. J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197653302.003.0002
A Double Regularization Loss Based on Long-tailed Noisy Labels
Wang, L., & Li, S.-Y. (2023) · IEEE International Conference on Civil Aviation Safety and Information Technology (ICCASIT) · doi:10.1109/iccasit58768.2023.10351565
“These labels will affect the accuracy of data analysis and deep learning models, so they need to be dealt with when processing data[14].”
BackgroundBuckner, C. J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197653302.002.0004
Loss Filtering Factor for Crowd Counting
Chen, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8435-0_1
Santos, F., & Zanchettin, C. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00448 · Cited by 3
Liu, Z., Xiao, L., Shen, C., Huang, Y., He, H., & Chen, W. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-7405-4_14 · Cited by 2
Zhang, L., Zhou, X. Y., Yang, C., Ding, X., & Wei, C. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01044-9 · Cited by 1
Deep Learning With DAGsPreprint
Bagil, S., Daoud, A., Pena, J. M., Wodtke, G. T., & Zhou, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4698386
“…neural networks, such that their number of weights exceeds the number of observations available for training, might actually enhance performance, regardless of whether regularization methods are employed (Allen-Zhu et al., 2019; Brutzkus et al., 2017; Yang et al., 2020; Zhang et al., 2021).”
Algorithm for High-Dimensional Data Based on Comprehensible Deep Progressive Fuzzy Network
Qian, J., & Qi, R. (2023) · International Symposium on Computer Science and Intelligent Control (ISCSIC) · doi:10.1109/iscsic60498.2023.00032
Loukil, Z., Mirza, Q. K. A., & Sayers, W. (2023) · International Conference on Future Internet of Things and Cloud (FiCloud) · doi:10.1109/ficloud58648.2023.00047
Deep Learning from Noisy Labels via Robust Nonnegative Matrix Factorization-Based Design
Wolnick, D. G., Ibrahim, S., Marrinan, T., & Fu, X. (2023) · IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing · doi:10.1109/camsap58249.2023.10403492
ContRE: A Complementary Measure for Robustness Evaluation of Deep Networks via Contrastive Examples
Li, X., Wu, X., Kong, L., Zhang, X., Huang, S., Dou, D., & Xiong, H. (2023) · Proceedings · doi:10.1109/icdm58522.2023.00040
A Novel Approach to Synthesize Class Labels in Highly Imbalanced Large Data
Kennedy, R. K. L., & Khoshgoftaar, T. M. (2023) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi58952.2023.00012
“Noisy, or inaccurately labeled data, negatively affects machine learning classification performance [5].”
BackgroundPerception Algorithm of Converter Station Based on Deep Stability Learning
Sun, H., Li, S., Li, H., Qiao, Z., Wang, J., & Huang, J. (2023) · International Conference on Mobile Networks and Wireless Communications (ICMNWC) · doi:10.1109/icmnwc60182.2023.10435879
“As a key node of energy transmission and conversion, accurate perception and anomaly detection of converter stations are crucial to the stable operation of power systems [2].”
BackgroundNoisy-label learning techniques
Carneiro, G. (2024) · Elsevier eBooks · doi:10.1016/b978-0-44-315441-6.00013-7
(2024) · Elsevier eBooks · doi:10.1016/b978-0-44-315441-6.00016-2
Sdelc: An Ensemble Label Noise Correction Method Based on Self-DistillationPreprint
Huang, Y., Huang, Y., Tang, Z., & Liu, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4760942
High Performance and Lightweight Single Semi-Lattice Algebraic Machine Learning
Haidar, I. M., Sliman, L., Damaj, I., & Haidar, A. M. (2024) · IEEE Access · doi:10.1109/access.2024.3376525
“Zhang et al. [30] investigated the generalization behaviors of deep convolutional neural networks.”
Background
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.