Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 1,801–1,900 of 2,667
Theoretically Grounded Adaptive Sampling for Imbalanced Multi-Label Classification in Medical Text
Cho, K. W., Youm, S., & Shin, K.-S. (2025) · IEEE Access · doi:10.1109/access.2025.3642470 · Cited by 2
Yuan, Q., Zhang, B., & Zhang, Z. (2025) · Measurement Science and Technology · doi:10.1088/1361-6501/ae2c0f · Cited by 1
Interpretable surrogate modeling of extrusion flows with deep learning
Stoecklein, D., Destgeer, G., Munger, C., Zhou, Y., & Carlo, D. D. (2025) · Physics of Fluids · doi:10.1063/5.0298999 · Cited by 1
DONUT: physics-aware machine learning for real-time X-ray nanodiffraction analysis
Luo, A., Zhou, T., Du, M., Holt, M. V., Singer, A., & Cherukara, M. J. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01860-7 · Cited by 2
“Byaddingsimulateddatatothe modeltraining,wecanensurethatthemodellearnsasuf fi cient range of features to cover the solution space of all experimental measurements on a sample, which produces more accurate predictions 38,39 .”
Gan, C., Zhou, D., Zhu, Q., Wang, X., Jain, D. K., & Štruc, V. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3645933 · Cited by 1
“There exists a two-layer neural network with ReLU (Rectified Linear Unit) activation functions and 2 n + d parameters that can represent any function over a dataset of n samples in a d -dimensional space [38].”
Yang, Z.-L., Wang, D., Liu, J., Qing, Z., & Liu, X. (2025) · International Symposium on Artificial Intelligence and Computational Social Sciences (AICSS) · doi:10.1145/3776759.3776862 · Cited by 1
Noisy Correspondence Rectification in Multimodal Clustering Space for Cross-Modal Matching
Yang, S., Long, Y., Wei, Y., Xie, Z., Yao, H., Xu, M. G., & Nie, L. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3646184 · Cited by 1
Shen, M., Song, K., Wang, X., Jia, W., Wang, T., & Han, Z. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104788 · Cited by 1
Don’t Miss Out on Novelty: Importance of Novel Features for Deep Anomaly Detection
Sivaprasad, S., & Fritz, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-12840-9_14 · Cited by 1
“Firstly, neural networks show ’unreasonable’ generalize well beyond the training data Zhang et al. (2021), often showing invariance in representation to even OOD samples Jacobsen et al. (2018).”
BackgroundLTMIA: a loss trajectory-based membership inference attack method in federated learning
Song, J., Yuan, J., Chen, G., Liu, Y., & Yang, N. (2026) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2025.104364 · Cited by 3
Overfitting, regularisation and condition estimation in regression
Winkler, J. R. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00906-9 · Cited by 1
“…in [4, §2.3] with reference to physics informed neural networks (neural networks that are used to solve partial differential equations), and in [5, §1.2] with reference to deep neural networks, that regularisation has little or no effect in minimising overfitting, but details are not provided.”
InfluentialHandling mislabeled data in fault diagnosis: A graph-assisted random forest approach
Chen, S., Xi, X., Zhong, M., Yang, R., & Orchard, M. E. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.132669 · Cited by 1
Relative Information Assisted Autonomous Navigation for Flexible Landing on Asteroids
Liu, J., Chen, Z., Ge, D., Zhu, S., & Cui, P. (2026) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2026.3652033 · Cited by 1
“The precision requirements of the neural network pro-foundly influence the scale of the network [32].”
Solar power forecasting with sparse deep learning for fast frequency response ancillary services
Haputhanthri, D., Silva, D. P. D., Kaynak, O., Mills, N., Kahawala, S., Moraliyage, H., & Jennings, A. (2026) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2025.111552 · Cited by 1
Interpretability in Cardiovascular Diagnostics: Classical Machine Learning vs. Deep Learning
Kumari, S., Chaudhary, M., & Mehra, R. (2025) · International Conference on Computing, Communication and Automation · doi:10.1109/iccca66364.2025.11325603 · Cited by 1
Clean-Anchor Alignment: Toward Robust and Stable Graph Neural Networks
Cho, J., Zhang, C., Bae, S., & Lee, S. (2026) · IEEE Access · doi:10.1109/access.2026.3658657 · Cited by 1
Self-citationA noisy label correction framework for apnea-hypopnea index estimation from sleep breathing sounds
Song, Y., Peng, J., Ding, L., Song, L., & Zhang, X. (2026) · Journal of Applied Biomedicine · doi:10.1016/j.bbe.2026.01.003 · Cited by 1
Invariant learning improves out-of-distribution generalization for IP geolocation
Wang, X., Liu, X., Tai, W., Walker, J., Wang, Y. J., Chen, K., & Zhou, F. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104641 · Cited by 2
Sun, W., Liu, H., Wu, Y., Qin, Y., Zhang, Y., Jin, W., Hai, H., Zhang, Y., 聂, 瑾., & Fu, H. (2026) · Food Analytical Methods · doi:10.1007/s12161-026-03015-w · Cited by 1
Data-related Ablation for Reinforcing Deep Learning in Explaining Complex Phenomena
Lanzino, R., Cinque, L., Foresti, G. L., & Placidi, G. (2026) · International Journal of Neural Systems · doi:10.1142/s0129065726500061 · Cited by 1
Stable and Generalizable Neural Network for Spatiotemporal Electromagnetic Modeling
Zheng, X., Zhang, Y., Peng, T. J., Wang, S. S., Hou, J., Chen, Q. C., Zhang, J., Mao, Y., You, J. W., & Cui, T. J. (2026) · IEEE Transactions on Antennas and Propagation · doi:10.1109/tap.2026.3658540 · Cited by 1
Only strict saddles in the energy landscape of predictive coding networks? *
Innocenti, F., Achour, E. M., Singh, R. R., & Buckley, C. L. (2025) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ade2eb · Cited by 1
GAF-MLGNN: An Efficient Meta-Learning Framework for Few-Shot HRRP RATR With GNN
Chen, L., Hu, P., Liu, Q., & Liu, Z. (2026) · IEEE Transactions on Signal and Information Processing over Networks · doi:10.1109/tsipn.2026.3663887 · Cited by 1
Venkatesan, K., Joshi, A., Sattari, F., Lefsrud, L., & Khan, M. A. (2026) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2026.153950 · Cited by 1
Occlusion-robust descriptor and hierarchical filtering-aggregation for 3D registration
Tan, S., Wang, X., & Zhang, Q. (2026) · Pattern Analysis and Applications · doi:10.1007/s10044-026-01624-z · Cited by 1
“These methods seek to leverage the powerful feature learning capabilities of neural networks to replace or optimize key steps in the traditional registration pipeline (such as feature extraction, correspondence matching, and outlier rejection) [15 – 20].”
On ridge estimation in high-dimensional rotationally sparse linear regression
Liang, L., & Tan, Z. (2026) · Electronic Journal of Statistics · doi:10.1214/26-ejs2496 · Cited by 1
García-Ontiyuelo, M., Acuña-Alonso, C., Peco-Costas, C., & Álvarez, X. (2026) · Advances in Space Research · doi:10.1016/j.asr.2026.02.055 · Cited by 1
Application of artificial neural networks in predicting the Mohr-Coulomb shear strength parameters
Fernandes, F. D. A. L., Dantas, S. A., & Mota, D. G. D. A. (2026) · Soils and Rocks · doi:10.28927/sr.2026.009124 · Cited by 1
Dong, X., Gao, Y. M., Liu, M., & Yu, W. (2026) · Algorithms · doi:10.3390/a19030177 · Cited by 1
Algorithm for improving drill core sizing accuracy using image segmentation
Salis, J., Zevallos, J., & Figueroa, J. (2026) · Bulletin of Engineering Geology and the Environment · doi:10.1007/s10064-025-04774-2 · Cited by 1
Few-shot annotation correction for lightweight retinal vessel image segmentation
Li, H., Sun, Y., Organisciak, D., Long, Y., Su, Y., & Wang, F. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1682878 · Cited by 1
“Currently, some research has analyzed the effect of noisy labels on model performance (9).”
InfluentialWAHNet: A Wave‐Attention Hybrid Network for Surface Wave Suppression in Seismic Data
Gao, L., Zheng, S., & Min, F. (2026) · Geophysical Prospecting · doi:10.1111/1365-2478.70151 · Cited by 1
“Moreover, they exhibit strong generalization ability when dealing with complex noise patterns (C. Zhang et al. 2021 ; Du et al. 2022 ).”
Pseudo-Text Guided Robust Learning for Noisy Correspondence in Cross-Modal Retrieval
Shi, D., Li, Z., Zhu, L., & Tang, J. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3675497 · Cited by 1
“It divides the data into clean and noisy subsets based on the memorization e ff ect of Deep Neural Networks (DNNs) [18] and rectifies correspondences by incorporating a soft margin into the triplet loss, thereby improving the model’s robustness.”
Hybrid learning framework to enable recency-aware attention
Chen, M., & Etemadi, A. H. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-11972-w · Cited by 1
Unsupervised Continual Learning: A Review of Challenges, Techniques, and Evaluation Metrics
Ruan, H., Maul, T., Chen, Y., Chandesa, T., Liao, I. Y., Chen, Z., & Zhou, M. (2026) · IEEE Access · doi:10.1109/access.2026.3679348 · Cited by 1
Qadir, A., Arshad, M., Yang, Z. X., & Changwen, X. (2026) · Business Process Management Journal · doi:10.1108/bpmj-09-2025-1418 · Cited by 1
You can’t fight in here! This is BBS!
Futrell, R., & Mahowald, K. (2026) · Behavioral and Brain Sciences · doi:10.1017/s0140525x26105123 · Cited by 1
Deep learning in chemical engineering: Strengths, weaknesses, opportunities, and threats
Prasad, K. S. N. V., Purohit, B. K., Bethi, B., Talari, V. K., & Rao, A. V. R. (2026) · The Canadian Journal of Chemical Engineering · doi:10.1002/cjce.70392 · Cited by 1
Online Learning-Enhanced High Order Adaptive Safety Control
Pan, L., Catellani, M., Silva, T. C., Sabattini, L., & Ayanian, N. (2026) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2026.3686645 · Cited by 1
Yin, Q., & Bao, X. (2026) · BMC Oral Health · doi:10.1186/s12903-026-08295-6 · Cited by 1
ACD-U: Asymmetric co-teaching with machine unlearning for robust learning with noisy labels
Fukunaga, R., Yoshida, S., & Muneyasu, M. (2026) · Image and Vision Computing · doi:10.1016/j.imavis.2026.106020 · Cited by 1
“DNNs tend to overfit to noisy labels [9, 10], causing them to learn incorrect patterns and significantly degrading their generalization performance on unseen data.”
A Data‐Driven Inverse Design Methodology for Magnetic Soft Millirobots Navigating in Confined Spaces
Ren, Z., Wang, H., Tse, C. W., Zheng, Y., Chen, X., & Hu, W. (2026) · Advanced Science · doi:10.1002/advs.75714 · Cited by 1
Ahmed, A. M., Shragge, J., & Tsvankin, I. (2026) · Applied Sciences · doi:10.3390/app16115352 · Cited by 1
“The distributions of target reservoir parameters in the training samples can strongly influence NN convergence properties and generalization performance [52,53].”
Training dynamics and state taxonomy in deep visual recognition networks
Quang, H. L., Ugail, H., Howard, N., Tien, C. T., Hoai, N. V., & Viet, H. N. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01593-6 · Cited by 1
“The fragility of learned solutions to the random relabelling of training data [5] and the existence of sparse subnetworks that alone drive performance [6] further illustrate that the dynamics of learning are structured at a level invisible to output-level signals.”
Risk-Aware Robust Learning: Reducing Clinical Risk under Label Noise in Medical Image Classification
Pereira, M. R. S., & Cordeiro, F. R. (2026) · doi:10.5753/sbcas.2026.20827 · Cited by 1
A Bibliometric Review of Artificial Neural Networks in Construction over the Past Decade (2015–2025)
Ametepey, S. O., Gyadu-Asiedu, O., Aigbavboa, C. O., & Addy, H. (2026) · Buildings · doi:10.3390/buildings16122470 · Cited by 1
Machine-learning modeling of pore pressure variation in geological CO2 storage reservoir
Davoodi, S., Thanh, H. V., Wood, D. A., Mehrad, M., & Burnaev, E. V. (2026) · Earth Science Informatics · doi:10.1007/s12145-026-02176-w · Cited by 1
ERASE - A Real-World Aligned Benchmark for Unlearning in Recommender Systems
Lubitzsch, P., de Rijke, M., & Schelter, S. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808596 · Cited by 1
“Modern machine learning models are known to encode information about their training data, ranging from statistical influence to explicit memorization of individual examples [6, 57].”
Deep learning-based intrusion detection for internet of medical things: a comprehensive review
Shah, S., Ali, G., Asim, M., ELAffendi, M. A., & Jabeen, S. (2026) · International Journal of Computers and Applications · doi:10.1080/1206212x.2026.2697794 · Cited by 1
“A model with better generalization is better even if its training accuracy is slightly low [94].”
A systematic review of deep learning based machine vision for quality control in industry 4.0
Mighouar, Z., Melloui, J., Mansouri, K., Ech-Chhibat, M. E. H., & Zahiri, L. (2026) · Discover Mechanical Engineering · doi:10.1007/s44245-026-00320-w · Cited by 1
“In addition to creating new methodologies for decision-making, deep learning has created an entirely new way of designing the decision support systems, moving away from rigid rule-based systems to adaptive data-driven systems [9, 47–50].”
Bréon, P., Argui, I., & Havard, V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33500-5_12 · Cited by 1
Evidence-guided learning against noisy and sparse labels on graphs
Yi, S., Zhang, W., Mao, Z., Zhou, Y., Qiao, Z., Shen, L., Tao, D., Lv, J., & Ju, W. (2026) · Artificial Intelligence · doi:10.1016/j.artint.2026.104608 · Cited by 1
AI-enhanced and data-driven mechanical analysis and structural topology optimization
Guo, Y., Du, Z., Liu, C., Zhang, W., Tang, S., Hao, W., Zhao, Z., Wu, C., Xu, Z., Bian, T., et al. (2026) · Science China Technological Sciences · doi:10.1007/s11431-026-3336-9 · Cited by 1
“Their application to large-scale and highly complex problems also entails substantial computational expense, which can erode the advantages they are supposed to o ff er over traditional numerical simulation methods in both runtime and cost [135].”
van der Miesen, L. (2021) · Electronic workshops in computing · doi:10.14236/ewic/pom2021.34 · Cited by 1
Enhancing Material Extrusion Additive Manufacturing with Sensor Fusion and Machine Learning
Ahmad, T., Shoaib, M., & Jan, R. U. (2025) · Journal of Industrial Intelligence · doi:10.56578/jii030105 · Cited by 2
Computational Neuroscience Offers Hints for More General Machine Learning
Rawlinson, D., & Kowadlo, G. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-63703-7_12 · Cited by 1
A Synergy of Freezing and Dropout - A New Learning Strategy of Convolutional Networks
Banach, M., & Skubalska-Rafajłowicz, E. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61401-0_1 · Cited by 3
On some similarities and differences between deep neural networks and kernel learning machines
Pei, E., & Fokoué, E. (2022) · Mathematics for Applications · doi:10.13164/ma.2022.07 · Cited by 1
“Even more recently, [52] emphasizes that Understanding Deep Learning (Still) Requires Rethinking Generalization .”
BackgroundTrustworthy Multimedia Analysis
Huang, X., Zhang, J., Zhang, Y., Zhao, X., & Sang, J. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3478871 · Cited by 1
“The notorious issues like dataset bias and Out-of-Distribution degradation, as well as the recently discussed phenomenon of random noise [8], shortcut learning [2], and reward hacking [4] can also be attributed to the utilization of this type of spurious correlation.”
BackgroundVOVU: A Method for Predicting Generalization in Deep Neural Networks
Wang, J., Ge, L., Liu, G., & Li, G. (2021) · Mathematical Problems in Engineering · doi:10.1155/2021/6170662 · Cited by 1
Robust Classification with Noisy Labels by Scaled Cross Entropy for Manufacturing Applications
Zhao, S., Li, X., & Chen, Y. (2021) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa45728.2021.9613459 · Cited by 1
Interactive Segmentation Using Prior Knowledge-Based Distance Map
Chung, Y., Lu, W., & Tian, X. (2021) · International Conference on Information, Cybernetics, and Computational Social Systems (ICCSS) · doi:10.1109/iccss53909.2021.9721959 · Cited by 2
A Survey on Implicit Bias of Gradient Descent
Li, P., Xu, L., & Song, H. (2022) · International Conference on Computer Research and Development · doi:10.1109/iccrd54409.2022.9730384 · Cited by 5
“On the other hand, even if we remove all of the explicit regularizations, the generalization performance of some models is still excellent [6].”
BackgroundGermi, S. B., & Rahtu, E. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-06427-2_52 · Cited by 2
Instance Discrimination Based Robust Training for Facial Expression Recognition Under Noisy Labels
Vikas, G., Gera, D., & Balasubramanian, S. (2022) · SN Computer Science · doi:10.1007/s42979-022-01410-8 · Cited by 1
“It has been observed in [12] and later shown in [13] that during the early part of training, DNNs fit on clean data, and then, eventually, start over-fitting on noisy labels due to their memorization ability.”
BackgroundEstablishing the Parameters of a Decentralized Neural Machine Learning Model
Ioste, A. R., & Finger, M. (2022) · doi:10.5753/eniac.2022.227342 · Cited by 1
“…decentralized models are those covered in the federated learning literature, with several alternative variations [McMahan et al. 2016, Koneˇcn`y et al. 2016, McMahan et al. 2017, Li et al. 2018, Zhao et al. 2018, Mohri et al. 2019, Yurochkin et al. 2019, Wang et al. 2020, Zhang et al. 2021].”
BackgroundLearning from Crowds with Mutual Correction-Based Co-Training
Tian, N., Wu, M., Jiang, J., & Zhang, J. (2022) · IEEE International Conference on Knowledge Graph (ICKG) · doi:10.1109/ickg55886.2022.00040 · Cited by 1
Lu, K.-L., & Chapman, A. (2025) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661825700415 · Cited by 1
Tan, Z., & Teng, Z. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14397-y · Cited by 1
“Domain generalization does not need to access any target domain samples in the training phase, but only by learning samples from a single or multiple domain, transplanting the model to unknown target domain through strategies such as data augmentation [7, 37] or feature alignment [8, 35].”
BackgroundLearning Texture Enhancement Prior with Deep Unfolding Network for Snapshot Compressive Imaging
Jin, M., Wei, Z., & Xiao, L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26313-2_22 · Cited by 1
Sample Size Estimation for Effective Modelling of Classification Problems in Machine Learning
Vinayak, N., & Ahmad, S. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-28183-9_26 · Cited by 1
Label-Free Model Evaluation with Out-of-Distribution Detection
Zhu, F., Zhao, Y., Liu, Z., & Liu, X. (2023) · Applied Sciences · doi:10.3390/app13085056 · Cited by 1
A Nested Ensemble Method to Bilevel Machine Learning
Chen, L., Abbas, M., & Chen, T. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097096 · Cited by 1
“It has been an important subject in past decades, especially in single-level ERM problems [9, 10].”
BackgroundInverse-Reference Priors for Fisher Regularization of Bayesian Neural Networks
Kim, K., Ma, E.-Y., Choi, J., & Kim, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.25997 · Cited by 3
“The generalization of deep neural networks (DNNs) with a large number of parameters has received considerable attention in recent studies for complex data analysis (Neyshabur 2017; Jiang et al. 2019; Zhang et al. 2021).”
BackgroundPMR-CNN: Prototype Mixture R-CNN for Few-Shot Object Detection
Zhou, J., Mei, J., Li, H., & Hu, Y. H. (2023) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv55152.2023.10186683 · Cited by 1
Robust Learning with Noisy Ship Trajectories by Adaptive Noise Rate Estimation
Yang, H., Wang, M., Chen, Z., Xiao, K., Li, X., & Huang, H. (2023) · Sensors · doi:10.3390/s23156723 · Cited by 1
“Deep learning models have powerful learning capabilities and can fit training sets with arbitrary label noise proportions [19].”
BackgroundHandling noisy annotations in deep supervised learning
Lemghari, I., Hégarat‐Mascle, S. L., Aldea, E., & Vandoni, J. (2023) · International Conference on Quality Control by Artificial Vision (QCAV) · doi:10.1117/12.2692547 · Cited by 1
“4 In this context, our main objective is to find a way to effectively train deep neural networks with modern architectures under label noise in a supervised learning classification setting.”
BackgroundLooking Closer to the Transferability Between Natural and Medical Images in Deep Learning
Rufaida, S. I., Putra, T. A., Leu, J., Song, T., & Katayama, T. (2023) · IEEE Access · doi:10.1109/access.2023.3299819 · Cited by 2
“Given that numerous training iterations can result in network memorization [16], particularly when dealing with small datasets, and since medical datasets are typically small, it is important to exercise caution when training deep neural networks on medical data.”
BackgroundDiscriminative Domain Adaptation Network for Fine-grained Disease Severity Classification
Wen, S., Chen, Y., Guo, S., Ma, Y., Gu, Y., & Chan, P. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191720 · Cited by 1
“However, when faced with FUDA, the model fitting of these bias-samples with large label bias will lead to overfitting [32], which will significantly increase the value of ε(h)s.”
BackgroundThe Impact of Synthetic Data on Membership Inference Attacks
Khan, M. S. N., & Buchegger, S. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-5177-2_6 · Cited by 2
Forecasting of the Reservoir in Flow with Recurrent Neural Network Architecture
Sharma, P. (2023) · International Conference on Edge Computing and Applications (ICECAA) · doi:10.1109/icecaa58104.2023.10212288 · Cited by 1
“In contrast, the forward layer yield request, ht, is over and over registered involving inputs in a +ve request from time T n to time T 1 [25].”
BackgroundLower Bounds for Multiclass Classification with Overparameterized Linear Models
Wu, D. X., & Sahai, A. (2023) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit54713.2023.10206652 · Cited by 2
Overparameterized Maximum Likelihood Tests for Detection of Sparse Vectors
Golubev, G. K. (2023) · Problems of Information Transmission · doi:10.1134/s0032946023010040 · Cited by 1
“It turned out that in many cases training of an overparametrized neural network using gradient descent type methods gives good results as compared to training of a nonoverparametrized network [1].”
ResultLearning from Few Examples with Nonlinear Feature Maps
Tyukin, I., Sutton, O., & Gorban, A. N. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-37717-4_15 · Cited by 1
New regularization methods for convolutional kernel tensors
Guo, P.-C. (2023) · AIMS Mathematics · doi:10.3934/math.20231335 · Cited by 1
The gap between theoretical computational theories and application models
Yang, C. (2023) · Journal of Physics Conference Series · doi:10.1088/1742-6596/2580/1/012027 · Cited by 1
“Hopefully, by finding proper assumptions and ideas for future research, the researchers can reconstruct their work to improve the understanding of the generalization both theoretically and practically [10].”
BackgroundKucharavy, A., Guerraoui, R., & Dolamic, L. (2023) · Conference on Artificial Life (ALIFE) · doi:10.1162/isal_a_00695 · Cited by 1
Find Important Training Dataset by Observing the Training Sequence Similarity
Liu, Z., Diao, H., Zhang, F., & Khan, S. U. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44213-1_34 · Cited by 1
Gradient and Feature Conformity-Steered Medical Image Classification with Noisy Labels
Xing, X., Chen, Z., Gao, Z., & Yuan, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43987-2_8 · Cited by 1
Co-assistant Networks for Label Correction
Chen, X., Fu, W., Tian, L., Shi, X., Shen, H. T., & Zhu, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43898-1_16 · Cited by 1
Integrated Machine Learning in Satellite Hydrology
Ndehedehe, C. E. (2023) · Springer climate · doi:10.1007/978-3-031-37727-3_9 · Cited by 1
Ichiki, A. (2023) · Physical review. E · doi:10.1103/physreve.108.044134 · Cited by 1
Training Noise Robust Deep Neural Networks with Self-supervised Learning
Wang, Z., Jin, S., Du, J., Li, L., & Dong, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-46674-8_21 · Cited by 1
Texture-Based Data Augmentation for Small Datasets
Dash, A., & Albu, A. B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45382-3_29 · Cited by 1
Le, V. T., Hao, P., Uwizeyemungu, S., Dinh, T. L., & Le, T. D. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-8296-7_16 · Cited by 1
Nardi, E. E. D. S., Padilha, B., Kamaura, L. T., & Ferreira, J. E. (2023) · Journal of Information and Data Management · doi:10.5753/jidm.2023.3179 · Cited by 1
“This technique is one of the most successful in object detection, mainly due to its [ Zhang et al ., 2021] generalization property.”
MethodDiCarlo, J. J., Yamins, D. L. K., Ferguson, M. E., Fedorenko, E. G., Bethge, M., Bonnen, T., & Schrimpf, M. (2023) · Behavioral and Brain Sciences · doi:10.1017/s0140525x23001607 · Cited by 4
Robust meta gradient learning for high-dimensional data with noisy-label ignorance
Liu, B., & Lin, Y. (2023) · PLoS ONE · doi:10.1371/journal.pone.0295678 · Cited by 2
“This, in turn, reduces the robustness of models and increases learning costs [12], negatively impacting model generalization and accuracy [13].”
Towards Robust Learning using Diametrical Risk Minimization for Network Intrusion Detection
McCollum, K. J., Bastian, N. D., & Røyset, J. Ø. (2023) · IEEE Conference on Dependable and Secure Computing (DSC) · doi:10.1109/dsc61021.2023.10354173 · Cited by 1
“However, modern DNN architectures tend to overfit to the training data [3] and therefore do not generalize well in the presence of mislabeled training data.”
BackgroundChannel-Agnostic Training of Transmitter and Receiver for Wireless Communications
Davey, C. P., Shakeel, I., Deo, R. C., & Salcedo‐Sanz, S. (2023) · Sensors · doi:10.3390/s23249848 · Cited by 1
“It achieves this through reducing the complexity of the model during training [27].”
Background
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.