Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 1,901–2,000 of 2,667
Harnessing Vision Transformers for LiDAR Point Cloud Segmentation
Inan, B. A., Rondao, D., & Aouf, N. (2023) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio58561.2023.10354753 · Cited by 1
“ViTs have demonstrated impressive results, often outper-forming CNNs in various benchmark tasks, including image classification, object detection, and semantic segmentation [10].”
BackgroundMLRNet: A Meta-Loss Reweighting Network for Biased Data on Text Classification
Yu, H., & Li, X. (2023) · Applied Sciences · doi:10.3390/app14010164 · Cited by 1
Liu, C., Chen, Y. S., & Lou, S. (2023) · Transactions on Economics Business and Management Research · doi:10.62051/6wagcn49 · Cited by 1
“Additionally, C. Zhang et al. (2021) elaborate that a significant number of businesses are projected to adopt AI of Electronic Commerce (EC) by 2030, further reinforcing the prevalence and importance of this technology in various industries.”
BackgroundPrediction of abrasive wear and surface hardness of printed parts by SLA technology
Muñoz, P., Villena-López, O., Ases, L. A. M., Pérez-Salinas, C., & Moya, D. (2024) · Ingenius · doi:10.17163/ings.n31.2024.02 · Cited by 1
“According to research [28–30], determine that too few neurons lead to underfitting, while too many neurons can contribute to overfitting.”
ResultStability of accuracy for the training of DNNs via the uniform doubling condition
Shmalo, Y. (2024) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-023-09919-1 · Cited by 1
“Furthermore, issues like overfitting, where a model excels on the training data but underperforms on unseen data, further complicate this relationship [9].”
Breaking the U: Asymmetric U-Net for Object Recognition in Muon Tomography☆
Cutura, J. A., & Pasch, S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386932 · Cited by 1
“Balancing between model intricacy, computational demands, and overfitting susceptibility is crucial [22].”
BackgroundGUENANA, M., Van Binh, D. I. N. H., Neu, T., Guyomarc’h, D., & Dang, H.-A. (2023) · Asia Meeting on Environment and Electrical Engineering (EEE-AM) · doi:10.1109/eee-am58328.2023.10395410 · Cited by 1
“Early stopping technique [24] was also incorporated, constantly monitoring model performance on a validation set and stopping training as soon as performance began to deteriorate.”
MethodDomain Adaptation for Edge Detectors Using Lightweight Networks
Bommireddy, V. S. R., Raniwala, S., & Kumar, P. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394371 · Cited by 1
Matteo Marsili, A. (2024) · Indian Journal of Physics · doi:10.1007/s12648-024-03068-9 · Cited by 2
Ensemble of Artificial Neural Networks and AutoML for Predicting Steel Properties
Silva, K., Serpa, P., Sgrott, D. M., Cerqueira, F. M., Miranda, F. J. F., Filho, J. F. S., & Parpinelli, R. S. (2023) · Congresso Brasileiro de Inteligência Computacional (CBIC) · doi:10.21528/cbic2023-149 · Cited by 1
Thilagavathi, S., Sridhar, D., & Sundaram, J. (2024) · Advances in medical diagnosis, treatment, and care (AMDTC) book series · doi:10.4018/979-8-3693-1281-0.ch008 · Cited by 1
Exploring the Drivers of Bank Loyalty in a Digitized World
Sharif, T., & Billah, U. I. (2022) · Lahore Journal of Business · doi:10.35536/ljb.2022.v10.i2.a5 · Cited by 1
A Probability Gradient Based Approach for Sampling Boundaries of In-Domain Data
Miao, J., Sethu, V., & Ahmed, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10445872 · Cited by 1
Sequence Models of Artificial Intelligence for Pattern Recognition in Lung Ultrasound Videos
Meschino, G. J., Betti, F. G., Tusman, G., & Acosta, C. M. (2024) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-031-61960-1_59 · Cited by 1
Asymptotic generalization errors in the online learning of random feature models
Worschech, R., & Rosenow, B. (2024) · Physical Review Research · doi:10.1103/physrevresearch.6.l022049 · Cited by 1
Deep Feature Discriminability as a Diagnostic Measure of Overfitting in CNN Models
Sikha, O. K., & Benítez, R. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-62281-6_16 · Cited by 1
Yang, N., Rongione, C., Jacquemart, A., Draye, X., & Vleeschouwer, C. D. (2024) · Applied Sciences · doi:10.3390/app14125156 · Cited by 3
Background suppression for volcano muography with machine learning
Galgóczi, G., Albrecht, G., Hamar, G., & Varga, D. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad64a7 · Cited by 1
Cheng, Y., & Petrides, K. V. (2024) · Educational and Psychological Measurement · doi:10.1177/00131644241262964 · Cited by 1
“NNs can learn both intrinsic and noise-related relationships for a dataset over enough iterations (Zhang et al., 2021).”
Malviya, A., & Maurya, C. K. (2024) · IEEE Access · doi:10.1109/access.2024.3435555 · Cited by 1
“Proposed framework exploits the observation that updates generated by stochastic gradient descent (SGD) reside within the range of input data points [80].”
Ma, Y. (2024) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001424520244 · Cited by 1
Cyber Attack Detection in IoT Using Enhanced Stream Classification Algorithm
Li, F., Ali, M., & Zhang, J. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-67447-1_29 · Cited by 1
Benefits of Stochastic Mirror Descent in High-Dimensional Binary Classification
Varma, K. N., & Hassibi, B. (2024) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit57864.2024.10619386 · Cited by 3
Trojan attribute inference attack on gradient boosting decision trees
Ito, K., Enkhtaivan, B., Teranishi, I., & Sakuma, J. (2024) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp60621.2024.00036 · Cited by 1
“The label manipulation involves rewriting labels to disturb the correlations being learned and degrade model accuracy: the works of Biggio et al. [9], Xiao et al. [57] treat Support Vector Machines (SVM), Zhang et al. [61] deal with NN.”
MethodDOST—Domain Obedient Self-supervision for Trustworthy Multi Label Classification with Noisy Labels
Saha, S., Garain, U., Ukil, A., Pal, A., & Khandelwal, S. (2024) · Studies in computational intelligence · doi:10.1007/978-3-031-63592-2_10 · Cited by 1
Martinetz, J., Linse, C., & Martinetz, T. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650680 · Cited by 1
Cestari, R. G., Maroni, G., Cannelli, L., Piga, D., & Formentin, S. (2024) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2024.08.535 · Cited by 2
Ai, X., Liao, Z., & Xia, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73158-7_18 · Cited by 1
Okyere-Gyamfi, S., Akoto-Adjepong, V., Adu, K., Ayidzoe, M. A., Appiah, O., Appiahene, P., Mensah, P. K., Opoku, M., Bawah, F. U., Awarayi, N. S., et al. (2024) · Scientific African · doi:10.1016/j.sciaf.2024.e02432 · Cited by 3
Feasibility study of data-driven wall interference correction framework for subsonic wind tunnel
Tai, M., Hwang, H., Jeong, S., Bak, J., & Park, D. (2024) · Journal of Wind Engineering and Industrial Aerodynamics · doi:10.1016/j.jweia.2024.105923 · Cited by 1
Towards Formal Design of FDIR Components with AI
Bozzano, M., Cimatti, A., Cristoforetti, M., Griggio, A., Svaizer, P., & Tonetta, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75387-9_19 · Cited by 2
COVID-19 Traumatic Stress, Working Memory Deficits, and Mindfulness
Gabay, G., Cho, H., Rice, K. G., Ashby, J. S., & Kira, I. A. (2024) · Journal of Loss and Trauma · doi:10.1080/15325024.2024.2411683 · Cited by 1
“…interventions in the USA and Europe, where mindfulness alleviated conditions of stress, but the levels of stress in the intervention groups in these studies were much higher than those in the control groups and in our samples (Galante et al., 2021; Kabat-Zinn, 2013; Oman, 2023; Zhang et al., 2021).”
Revisiting Linearization of Spatial Maps in SoTA Face Recognition Backbone
Bhatta, A., Wu, H., Öztürk, K., & Bowyer, K. W. (2024) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb62174.2024.10744486 · Cited by 1
Frequency Ensemble Based on Multi-Band Checkpoint Saving Mechanism for Time Series Forecasting
Yuan, P., & Zhu, C. (2024) · International Conference on Intelligent Computing and Signal Processing (ICSP) · doi:10.1109/icsp62122.2024.10743879 · Cited by 1
“Early stopping [8] is considered effective in avoiding the overfitting.”
BackgroundA weakly supervised method for 3D object detection with partially annotated samples
Lu, B., Li, Q., & Liang, Y. (2024) · Measurement and Control · doi:10.1177/00202940241297568 · Cited by 1
“Even though F 0 img and F lidar are aligned in position, the heterogeneous feature extraction branches exhibit different inductive biases, 28 leading to suboptimal performance when directly merging F 0 img and F lidar with equal weights.”
FreqAT: An Adversarial Training Based on Adaptive Frequency-Domain Transform
Zhang, D., Liang, Y., Huang, Q., Huang, X., Liao, P., Yang, M. D., & Zeng, L. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0850-8_19 · Cited by 2
Lei, C., Singh, S., Hou, F., & Wang, R. (2024) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3696409.3700259 · Cited by 1
“ERM incorrectly allows large neural networks to memorize the training data rather than generalize to it, even with strong regularization in classification problems [32].”
Improving volcanic event recognition through new seismic signals representation
Espinosa-Curilem, C., Novoa, C. S., Saldías, M. C., Huenupán, F., Sáez, D. A., & Alarcón, D. B. (2024) · IEEE Latin American Conference on Computational Intelligence (LA-CCI) · doi:10.1109/la-cci62337.2024.10814838 · Cited by 1
Palit, S., Banerjee, B., & Chaudhuri, S. (2024) · Indian Conference on Computer Vision, Graphics and Image Processing · doi:10.1145/3702250.3702287 · Cited by 2
Three Strategies for Salvaging Explanatory Value in Deep Neural Network Modeling
Verreault-Julien, P. (2025) · Routledge eBooks · doi:10.4324/9781003342816-13 · Cited by 2
Research on weight initialization of CNN student models based on knowledge distillation
Fei, C., Zhong, Z., Jiang, K., Shao, C., Sui, Y., Liu, H., & Sun, R. (2025) · Intelligent Data Analysis · doi:10.1177/1088467x241301381 · Cited by 1
When Less is More: Achieving Faster Convergence in Distributed Edge Machine Learning
Basani, A. R., Vivek, S., Krishna, A., & Paul, A. K. (2024) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.1109/hipc62374.2024.00034 · Cited by 1
Genetic NEAT-Based Method for Multi-Class Classification
Alfaham, A., Raemdonck, S. V., & Mercelis, S. (2024) · International Conference on Algorithms, Computing and Artificial Intelligence (ACAI) · doi:10.1109/acai63924.2024.10899662 · Cited by 1
Machine learning algorithms to optimize the training effect of table tennis players
Han, T. (2025) · Journal of Computational Methods in Sciences and Engineering · doi:10.1177/14727978251321950 · Cited by 1
Multi-label Recognition under Noisy Supervision: A Confusion Mixture Modeling Approach
Gonzalez, D. L., & Ibrahim, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889155 · Cited by 1
Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm
Zhang, Y., Li, B., & Giannakis, G. B. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889586 · Cited by 3
“Owing to the markedly high dimensionality, DNNs can memorize a large gamut of training data [4].”
A Neural Network Optimization Approach for Anemia Prediction Using Image Pixels and Hemoglobin Data
Nababan, A. A., Turnip, M., Dharma, A., Yennimar, Y., Prabowo, A., & Manday, D. R. (2024) · IEEE International Conference on Control & Automation, Electronics, Robotics, Internet of Things, and Artificial Intelligence (CERIA) · doi:10.1109/ceria64726.2024.10915131 · Cited by 1
Kolli, C. S., Venkata Naga Raju, D., Kumar, P., Srinivasa Rao, D., Vinay, P., & Kumar, G. R. (2025) · Proceedings on Engineering Sciences · doi:10.24874/pes07.01b.012 · Cited by 2
“In the detection of credit card fraud, feature engineering techniques commonly revolve around aggregating features Cheng et al. (2021) and Zhang et al. (2021).”
Elevating Vision Transformer Performance: The KAN-VIT Model for Image Recognition
Ye, L., & Guo, Z. (2024) · IEEE Smart World Congress (SWC) · doi:10.1109/swc62898.2024.00353 · Cited by 1
Liu, Y., Li, C., Shen, L.-C., Yan, H., Wei, G., Gasser, R. B., Hu, X. T., Song, J., & Yu, D. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110068 · Cited by 1
SPL-PlaneTR: Lightweight and Generalizable Indoor Plane Segmentation Based on Prompt Learning
Deng, Z., Ge, Y., Qi, X., Sun, K., Wan, R., Zhang, B., Zhang, S., Zhang, X., & Meng, Y. (2025) · Sensors · doi:10.3390/s25092797 · Cited by 1
Zhang, H., Chen, Q., Xue, B., Banzhaf, W., & Zhang, M. (2025) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2025.3581739 · Cited by 1
“However, modern feature learning techniques, such as deep neural networks, possess high VC dimensions yet generalize well, making traditional complexity measures fail to accurately predict the generalization performance of modern machine learning methods [17].”
Analysis of the Content and Efficacy of Limo Rhizome: A Herbal Blend That Is Beneficial for Health
Septia, A. A., Rhani, N. A., & Ridho, I. N. (2024) · Jurnal Inovasi dan Pengembangan Hasil Pengabdian Masyarakat · doi:10.61650/jip-dimas.v2i2.625 · Cited by 1
Deep-learning-aided modifier adaptation: synergies with process intensification
Patrón, G. D., Tsay, C., & Ricardez‐Sandoval, L. (2025) · Chemical Engineering and Processing - Process Intensification · doi:10.1016/j.cep.2025.110581 · Cited by 1
Inferring Communities of Interest in Collaborative Learning-based Recommender Systems
Belal, Y., Maouche, M., Mokhtar, S. B., & Simonet, A. (2025) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs63083.2025.00056 · Cited by 3
Salman, S., Shams, M., Liu, X., & Zhu, L. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229323 · Cited by 3
Klasifikasi Kematangan Buah Kelapa Sawit Menggunakan Model Yolov8 Berbasis Deep Learning
Muna, M. S., Setiyo, Y., Wirawan, I. P. S., Syarovy, M., & Jaya, G. H. (2025) · Journal of Agricultural and Biosystem Engineering Research · doi:10.20884/1.jaber.2025.6.1.15953 · Cited by 1
McGinnity, C., Bamford, C. C. G., Fenney, N., Fleming, A., Forcada, J., Tift, M. S., Hückstädt, L. A., Costa, D. P., & Fretwell, P. T. (2026) · Remote Sensing in Ecology and Conservation · doi:10.1002/rse2.70060 · Cited by 1
“High-capacity models can even memorize noisy labels (Zhang et al., 2021), underscoring why label quality and quantity remain a fundamental constraint for training large models.”
Architecture and independence controller for deep learning in safety critical applications
Bodenhausen, U. (2019) · Proceedings · doi:10.1007/978-3-658-25939-6_90 · Cited by 1
Kuzborskij, I., & Szepesvári, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05341 · Cited by 1
“Yet, the networks trained without regularization still demonstrate a surprising ability to perform well on the entire population even in the presence of label noise (Zhang et al., 2021).”
BackgroundImproving Generalization Bounds for VC Classes Using the Hypergeometric Tail InversionPreprint
Leboeuf, J.-S., LeBlanc, F., & Marchand, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.00062 · Cited by 1
“…research avenues indicate that the VC dimension is not the relevant complexity parameter to characterize the generalization error of overparametrized deep neural networks (see for examples Bartlett et al. (2019) and Zhang et al. (2021)), neurals networks suffer from a lack of interpretability.”
BackgroundSelf-supervised denoising for massive noisy imagesPreprint
Wang, F., Henninen, T. R., Keller, D., & Erni, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.11911 · Cited by 1
“The deep networks that are trained by the standard gradient descent algorithm are memorizing the training data, and use the memorized samples for prediction with a weighted similarity function[18, 19, 20, 21, 22, 23].”
MethodInvestigating Temporal Convolutional Neural Networks For Satellite Image Time Series ClassificationPreprint
Abdallah, Z. S., & Brock, J. (2022) · doi:10.36227/techrxiv.19590580.v1 · Cited by 1
Applications of neural tangent kernel theory in deep learning
Shi, X. (2022) · Digital Repository Service (Northeastern University) · doi:10.17760/d20439229 · Cited by 1
Walking Path Images from Real-Time Location Data Predict Degree of Cognitive ImpairmentPreprint
Faruk, T., Shum, L. C., Iaboni, A., & Khan, S. S. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4192590 · Cited by 1
Generalization Error Bounds for Noisy, Iterative Algorithms via Maximal LeakagePreprint
Issa, I., Esposito, A. R., & Gastpar, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.14518 · Cited by 1
“…in many settings (Bousquet et al., 2003; Shalev-Shwartz and Ben-David., 2014), existing bounds and techniques provide vacuous expressions when employed to show the generalization capabilities of deep neural networks (DNNs) (Bartlett et al., 2017, 2019; Jiang et al., 2020; Zhang et al., 2021).”
BackgroundCorrelation Based Node Partitioning to Sparse Multilayer PerceptronPreprint
Benar, C., & Akansu, A. N. (2023) · doi:10.36227/techrxiv.23713962 · Cited by 1
Integração de Datasets de Vídeo para Tradução Automática da LIBRAS com Aprendizado Profundo
Sarmento, A. H. D. A. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.55.2023.tde-10012024-093541 · Cited by 1
Can ChatGPT code the technical part of a Bachelor's Thesis in Informatics?Preprint
Amin, S., Hellström, J., & Hintze, A. (2024) · Qeios · doi:10.32388/aaaj8n · Cited by 1
Kanwal, N., Khoraminia, F., Kiraz, U., Mosquera‐Zamudio, A., Monteagudo, C. C., Janssen, E. A. M., Zuiverloon, T. C. M., Rong, C., & Engan, K. (2024) · Research Square · doi:10.21203/rs.3.rs-4092609/v1 · Cited by 1
Gonon, A., Brisebarre, N., Riccietti, E., & Gribonval, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.15006 · Cited by 5
Reconstructing training data from document understanding modelsPreprint
Dentan, J., Paran, A., & Shabou, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.03182 · Cited by 1
Correntropy Meets Cross-Entropy: A Robust Loss Against Noisy LabelsPreprint
Deng, Q., Huang, X., Luo, W., Du, Y., Zhou, N., Shi, K., & Chen, B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4875114 · Cited by 1
Automatic segmentation of metal surface defects with noise labelPreprint
Xu, Y., Zhu, J., Xu, S., Liu, W., & Bian, X. (2024) · Research Square · doi:10.21203/rs.3.rs-4787816/v1 · Cited by 1
Convex Distillation: Efficient Compression of Deep Networks via Convex OptimizationPreprint
Varshney, P., & Pilancı, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.06567 · Cited by 3
“Additionally, while DNNs have the capacity to memorize the training dataset, they often end up learning basic solutions that generalize well to test datasets (Zhang et al., 2021).”
BackgroundHidden Activations Are Not Enough: A General Approach to Neural Network PredictionsPreprint
Leblanc, S., Rasolomanana, A., & Armenta, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.13163 · Cited by 1
“Generalization in deep learning is one of the biggest open problems (Zhang et al., 2017; 2021a; 2020).”
BackgroundMartinsson, J., Virtanen, T. I., Sandsten, M., & Mogren, O. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.09363 · Cited by 1
“While class label noise has been extensively studied (Song et al., 2022; Zhang et al., 2021), the effects of segment label noise remain underexplored.”
Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help?Preprint
Rouchouse, D., Gonon, A., Gribonval, R., & Guedj, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.26149 · Cited by 1
“Deep neural networks generalize well despite being massively overparameterized, a fact that remains only partially explained by statistical learning theory [47, 6, 5].”
Mohammadagha, M., Bigdeli, F., Sharifi, S., Deldadehasl, M., & Ataei, S. (2025) · Preprints.org · doi:10.20944/preprints202511.0501.v1 · Cited by 1
Christian Ethical Guidelines to Artificial Intelligence and Technological Singularity
Van Heerden, E. (2024) · Boloka Institutional Repository (North-west University) · doi:10.13140/rg.2.2.36814.88648 · Cited by 2
Capturing the Computational Complexity of Geometric Problems by the First-Order Theory of the Reals
Jungeblut, P. (2024) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000170385 · Cited by 1
Assessment of Data-Driven Model for the Prediction of Ship Performances and Responses
Ferlita, A. L. (2025) · DuEPublico (University of Duisburg-Essen) · doi:10.17185/duepublico/83370 · Cited by 3
Xygkou-Tsiamoulou, A. (2025) · Kent Academic Repository (University of Kent) · doi:10.22024/unikent/01.02.109186 · Cited by 3
Machine Learning for Prognostic Modeling in Head and Neck Cancer using Multimodal Data
Kazmierski, M. (2021) · TSpace (University of Toronto) · Cited by 2
Carlsson, A. (2023) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 3
Liu, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5964738 · Cited by 1
Quantization of Neural Network Equalizers in Optical Fiber Transmission Experiments
Darweesh, J. (2023) · theses.fr (ABES) · doi:10.70675/86dfc56fz712cz44c5zacb3z805ce6ae49d2 · Cited by 1
Complexity Methods in Physics-Guided Machine Learning
Brandão, E. (2023) · theses.fr (ABES) · doi:10.70675/3ea12bd7z6df4z4c24z966fzd6569cf17d06 · Cited by 1
Deep learning with Lipschitz constraints
Béthune, L. (2024) · theses.fr (ABES) · doi:10.70675/0c74dbdfz8185z43baz9026z9472eba47667 · Cited by 8
Estimating Implicit Regularization in Deep LearningPreprint
Rudoler, J. H., Tan, K., Hooker, G., & Kording, K. P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.05436 · Cited by 1
“One of the great challenges of modern machine learning is to understand how massively over-parametrized neural networks which can effectively memorize their training sets still manage to generalize [31, 46].”
Characterizing Learning in Deep Neural Networks using Tractable Algorithmic Complexity AnalysisPreprint
Bakhtiarifard, P., Wilson, S. N., Afifi, M., Wenshøj, J., & Selvan, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.15551 · Cited by 1
Rethinking Federated Unlearning via the Lens of MemorizationPreprint
Wei, J., Zhang, Y., Zhang, H., Zhang, L. Y., Chen, C., Ong, K., Zhang, J., & Xiang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.24545 · Cited by 1
Conservation Laws from Data Symmetry in Neural NetworksPreprint
Galley, J., Shahverdi, V., & Flinth, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.10913 · Cited by 1
Min, H., MacDonald, L. E., & Vidal, R. (2026) · IEEE Signal Processing Magazine · doi:10.1109/msp.2026.3665010 · Cited by 2
Jo, H., Kim, A.-Y., Lee, D., Baek, J.-S., Lee, K., Ku, B., Stern, A. W., & Kim, J. H. (2026) · Research Square · doi:10.21203/rs.3.rs-9458448/v1 · Cited by 1
Hong, Z., Wang, H., Zada, Z., Gazula, H., Turner, D., Aubrey, B., Niekerken, L., Doyle, W., Devore, S., Dugan, P., et al. (2024) · doi:10.1101/2024.06.12.598513 · Cited by 1
“Recent work has argued that the “size” of these models—the number of learnable parameters—is critical, as some linguistic competencies only emerge in larger models with more parameters (Bommasani et al., 2021; Kaplan et al., 2020; Manning et al., 2020; Sutton, 2019; C. Zhang et al., 2021).”
Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models
Xu, H., Lan, P., Yang, E., Guo, G., Zhao, J., Jiang, L., & Wang, X. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.646
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
How and What Deep Learning Learns
Matsubara, T. (2017) · Kagaku tetsugaku · doi:10.4216/jpssj.50.0_51
Compression Implies GeneralizationPreprint
Grønlund, A., Høgsgaard, M. M., Kamma, L., & Larsen, K. G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07989
Contrastive Representations for Label Noise Require Fine-TuningPreprint
Nodet, P., Lemaire, V., Bondu, A., & Cornuéjols, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.09154
“Deep Learning (DL) paradigm has proved very powerful in many tasks, however recent papers [34, 49] have shown that “noisy labels” are a real challenge for end-to-end deep learning architectures.”
Background
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.