Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Deep learning for time series classification: a review
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00619-1 · Cited by 3,342
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595
Shen, C. (2018) · Water Resources Research · doi:10.1029/2018wr022643 · Cited by 1,133
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023) · arXiv · doi:10.48550/arxiv.2307.06435 · Cited by 1,259
“This paper will serve as a valuable resource for researchers, offering insights into the recent advancements in LLMs and providing fundamental concepts and details to develop improved LLMs.”
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
“This paper will serve as a valuable resource for researchers, offering insights into the recent advancements in LLMs and providing fundamental concepts and details to develop improved LLMs.”
Deep learning modelling techniques: Current progress, applications, advantages, and challenges
Ahmed, S. F., Alam, M. S. B., Hassan, M., Rozbu, M. R., Ishtiak, T., Rafa, N., Mofijur, M., Shawkat Ali, A. B. M., & Gandomi, A. H. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10466-8 · Cited by 1,079
Robust Aggregation for Federated Learning
Pillutla, K., Kakade, S. M., & Harchaoui, Z. (2022) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3153135 · Cited by 801
Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users
Jospin, L. V., Laga, H., Boussaid, F., Buntine, W., & Bennamoun, M. (2022) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2022.3155327 · Cited by 879
Membership Inference Attacks From First Principles
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., & Tramèr, F. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833649 · Cited by 502
Benign overfitting in linear regression
Bartlett, P. L., Long, P. M., Lugosi, G., & Tsigler, A. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907378117 · Cited by 547
Membership Inference Attacks on Machine Learning: A Survey
Hu, H., Salcic, Z., Sun, L., Dobbie, G., Yu, P. S., & Zhang, X. (2022) · ACM Computing Surveys · doi:10.1145/3523273 · Cited by 564
“Machine learning (ML) models such as DNNs are often overparameterized, which means that they have sufficient capacity to memorize information about their training dataset [16, 144, 183, 234].”
BackgroundBrain Tumor Detection Based on Deep Learning Approaches and Magnetic Resonance Imaging
Abdusalomov, A. B., Mukhiddinov, M., & Whangbo, T. K. (2023) · Cancers · doi:10.3390/cancers15164172 · Cited by 494
“The trained models show improved generalizability and accuracy when applied to distributions outside the training data when this information is added to the set [68].”
BackgroundNauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01756-8 · Cited by 497
“Most of them investigate the connections between the memorization capacity of deep models [187] and the learning process, in order to know the contributions of training examples, including long-tailed ones and outliers.”
BackgroundPrivacy and Robustness in Federated Learning: Attacks and Defenses
Lyu, L., Yu, H., Ma, X., Chen, C., Sun, L., Zhao, J., Yang, Q., & Yu, P. S. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3216981 · Cited by 548
“Hence, GAN attack is not targeted to reconstruct the exact training inputs, but only the class representatives.”
BackgroundA Systematic Literature Review of User Trust in AI-Enabled Systems: An HCI Perspective
Bach, T. A., Khan, A., Hallock, H., Beltrão, G., & Sousa, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2138826 · Cited by 377
“For example, the importance of explanations of AI-enabled systems have been consistently mentioned (Barda et al., 2020; Glikson & Woolley, 2020; O ’ Sullivan et al., 2019; Zhang, Bengio, et al., 2021), highlighting their potential role to improve user experiences and attitudes.”
BackgroundMetaReg: towards domain generalization using meta-regularization
Balaji, Y., Sankaranarayanan, S., & Chellappa, R. (2018) · neural information processing systems · Cited by 805
The Implicit Bias of Gradient Descent on Separable DataPreprint
Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., & Srebro, N. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.10345 · Cited by 829
Model complexity of deep learning: a survey
Hu, X., Chu, L., Pei, J., Liu, W., & Bian, J. (2021) · Knowledge and Information Systems · doi:10.1007/s10115-021-01605-0 · Cited by 402
Tiu, E., Talius, E., Patel, P., Langlotz, C. P., Ng, A. Y., & Rajpurkar, P. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00936-9 · Cited by 442
Shifting machine learning for healthcare from development to deployment and from models to data
Zhang, A., Xing, L., Zou, J., & Wu, J. C. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00898-y · Cited by 420
“In certain instances, data leakage can occur, such as when ML models ‘memorize’ datasets 80–82 and when access to model parameters and updates can be used to infer the original dataset 83 .”
BackgroundShah, H. A., Saeed, F., Yun, S., Park, J.-H., Paul, A., & Kang, J.-M. (2022) · IEEE Access · doi:10.1109/access.2022.3184113 · Cited by 337
“Models trained in this manner are more generalizable and generate better predictions from distributions other than the training data [50].”
BackgroundErasing Concepts from Diffusion Models
Gandikota, R., Materzyńska, J., Fiotto-Kaufman, J., & Bau, D. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00230 · Cited by 227
“While the traditional goal of machine learning is to generalize without memo-rization, large models are capable of exact memorization if specifically trained to do so [48], and unintentional memo-rization has also been observed in large-scale settings [6, 5], including diffusion models [45].”
BackgroundA comprehensive survey on regularization strategies in machine learning
Tian, Y., & Zhang, Y. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.005 · Cited by 298
Computing receptive fields of convolutional neural networks
Araujo, A., Norris, W., & Sim, J. (2019) · Distill · doi:10.23915/distill.00021 · Cited by 312
A Comprehensive Survey on Poisoning Attacks and Countermeasures in Machine Learning
Tian, Z., Cui, L., Liang, J., & Yu, S. (2023) · ACM Computing Surveys · doi:10.1145/3551636 · Cited by 271
“In the literature [151], Zhang et al. conducted a detailed experimental analysis on the generalization ability of deep learning models.”
BackgroundDoes learning require memorization? a short tale about a long tail
Feldman, V. (2020) · Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing · doi:10.1145/3357713.3384290 · Cited by 287
Var-CNN: A Data-Efficient Website Fingerprinting Attack Based on Deep Learning
Bhat, S., Lu, D., Kwon, A., & Devadas, S. (2019) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2019-0070 · Cited by 244
SimMatch: Semi-supervised Learning with Similarity Matching
Zheng, M., You, S., Huang, L., Wang, F., Qian, C., & Xu, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01407 · Cited by 248
“However, when there are only very limited annotated data, the semantic classifier is no longer reliable; applying the pseudo-label method will cause the “overconfidence” issue [13,63], which means the model will fit on the confident but wrong pseudo-labels, resulting in poor performance.”
BackgroundNemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X., & Hu, C. (2023) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2023.110796 · Cited by 243
Is Quantum Advantage the Right Goal for Quantum Machine Learning?
Schuld, M., & Killoran, N. (2022) · PRX Quantum · doi:10.1103/prxquantum.3.030101 · Cited by 207
“Furthermore, it is by now largely uncontested that the algorithm with which neural networks are trained, as well as the data itself, plays a crucial role in the phenomena we observe in deep learning [41, 43, 44].”
BackgroundTheory of overparametrization in quantum neural networks
Larocca, M., Ju, N., García-Martín, D., Coles, P. J., & Cerezo, M. (2023) · Nature Computational Science · doi:10.1038/s43588-023-00467-6 · Cited by 176
“Usually, this implies having a number of parameters in the NN ∗ The first two authors contributed equally to this work. that is much larger than the number of training points [9].”
BackgroundAdversarial machine learning : a taxonomy and terminology of attacks and mitigations
Vassilev, A., Oprea, A., Fordyce, A. J., & Anderson, H. S. (2024) · doi:10.6028/nist.ai.100-2e2023 · Cited by 317
Ma, C., Wang, K., Chi, Y., & Chen, Y. (2020) · Foundations of Computational Mathematics · doi:10.1007/s10208-019-09429-9 · Cited by 204
Javed, H., El-Sappagh, S., & Abuhmed, T. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-11005-9 · Cited by 180
Transforming sentiment analysis in the financial domain with ChatGPT
Fatouros, G., Soldatos, J., Kouroumali, K., Makridis, G., & Kyriazis, D. (2023) · Machine Learning with Applications · doi:10.1016/j.mlwa.2023.100508 · Cited by 181
Accurate global machine learning force fields for molecules with hundreds of atoms
Chmiela, S., Vassilev-Galindo, V., Unke, O. T., Kabylda, A., Sauceda, H. E., Tkatchenko, A., & Müller, K.-R. (2023) · Science Advances · doi:10.1126/sciadv.adf0873 · Cited by 177
“Al-beit, the exceptional expressive power of global MLFFs goes along with a stark increase in parametric complexity [4–6] over classical FFs.”
BackgroundArtificial intelligence for modelling infectious disease epidemics
Kraemer, M. U. G., Tsui, J. L.-H., Chang, S. Y., Lytras, S., Khurana, M. P., Vanderslott, S., Bajaj, S., Scheidwasser, N., Curran-Sebastian, J. L., Semenova, E., et al. (2025) · Nature · doi:10.1038/s41586-024-08564-w · Cited by 169
“We join the AI community in recommending that the fundamental theory underpinning current AI models is studied further 149 .”
The loss surface of deep and wide neural networksPreprint
Nguyen, Q., & Hein, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.08045 · Cited by 282
CAFE: Learning to Condense Dataset by Aligning Features
Wang, K., Zhao, B., Peng, X., Zhu, Z., Yang, S., Wang, S., Huang, G., Bilen, H., Wang, X., & You, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01188 · Cited by 170
Ahmad, S., Shakeel, I., Mehfuz, S., & Ahmad, J. (2023) · Computer Science Review · doi:10.1016/j.cosrev.2023.100568 · Cited by 197
Learning with Noisy Labels Revisited: A Study Using Real-World Human AnnotationsPreprint
Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., & Liu, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12088 · Cited by 242
“There are some techniques for synthesizing the instance-dependent label noise, such as the polynomial margin diminishing label noise [Zhang et al., 2021b] where instances near decision boundary are easier to be mislabeled, the part-dependent label noise [Xia et al., 2020b] where different parts of…”
BackgroundMethodBiological constraints on neural network models of cognitive function
Pulvermüller, F., Tomasello, R., Henningsen-Schomers, M. R., & Wennekers, T. (2021) · Nature Reviews Neuroscience · doi:10.1038/s41583-021-00473-5 · Cited by 173
Concepts and applications of digital twins in healthcare and medicine
Zhang, K., Zhou, H.-Y., Baptista‐Hon, D. T., Gao, Y., Liu, X., Oermann, E. K., Xu, S. H., Jin, S., Zhang, J., Sun, Z., et al. (2024) · Patterns · doi:10.1016/j.patter.2024.101028 · Cited by 220
“83 Data augmentation using generative adversarial networks can generate a large amount of training data to solve the problems of insufficient data and uneven distribution.”
BackgroundMethodRandom Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
Liu, F., Huang, X., Chen, Y., & Suykens, J. A. K. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3097011 · Cited by 268
Meta-learning approaches for learning-to-learn in deep learning: A survey
Tian, Y., Zhao, X., & Huang, W. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.078 · Cited by 161
Quantum machine learning for chemistry and physics
Sajjan, M., Li, J., Selvarajan, R., Sureshbabu, S. H., Kale, S. S., Gupta, R., Singh, V., & Kais, S. (2022) · Chemical Society Reviews · doi:10.1039/d2cs00203e · Cited by 211
Guiding Pseudo-labels with Uncertainty Estimation for Source-free Unsupervised Domain Adaptation
Litrico, M., Bue, A. D., & Morerio, P. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00738 · Cited by 244
“[67] demonstrated that a deep neural network can easily overfit an entire dataset with any ratio of corrupted labels which results in poor generalisation on test data.”
BackgroundGeneralization Error in Deep Learning
Jakubovitz, D., Giryes, R., & Rodrigues, M. R. D. (2019) · Applied and numerical harmonic analysis · doi:10.1007/978-3-319-73074-5_5 · Cited by 202
Tabular and latent space synthetic data generation: a literature review
Fonseca, J., & Bação, F. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00792-7 · Cited by 221
“A common problem in the training of ML classifiers is their capacity to generalize [40] (i.e., reduce the difference in classification performance between known and unseen observations).”
BackgroundExplainable Software Analytics
Dam, H., Tran, T., & Ghose, A. (2018) · 2018 IEEE/ACM 40th International Conference on Software Engineering: New Ideas and Emerging Technologies Results (ICSE-NIER) · doi:10.1145/3183399.3183424 · Cited by 153
Co-learning: Learning from Noisy Labels with Self-supervision
Tan, C., Xia, J., Wu, L., & Li, S. Z. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475622 · Cited by 203
“However, deep neural networks can easily overfit to noisy labels, as shown in recent research [18, 22, 34, 40], and this can dramatically degrade the generalization performance.”
BackgroundFenwick, A., Molnár, G., & Frangos, P. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1272823 · Cited by 153
Hard Sample Aware Noise Robust Learning for Histopathology Image Classification
Zhu, C., Wen-kai, C., Peng, T., Wang, Y., & Jin, M. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3125459 · Cited by 183
A Review of Hydrodynamic and Machine Learning Approaches for Flood Inundation Modeling
Karim, F., Armin, M. A., Ahmedt‐Aristizabal, D., Tychsen-Smith, L., & Petersson, L. (2023) · Water · doi:10.3390/w15030566 · Cited by 150
“There is even evidence to suggest that SGD is important for generalization performance [37].”
BackgroundHoloDetect: Few-Shot Learning for Error Detection
Heidari, A., McGrath, J., Ilyas, I. F., & Rekatsinas, T. (2019) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3299869.3319888 · Cited by 197
Uncertainty Inspired RGB-D Saliency Detection
Zhang, J., Fan, D.-P., Dai, Y., Anwar, S., Saleh, F. S., Aliakbarian, S., & Barnes, N. M. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3073564 · Cited by 176
RoFL: Robustness of Secure Federated Learning
Lycklama, H., Burkhalter, L., Viand, A., Küchler, N., & Hithnawi, A. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179400 · Cited by 195
Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
Vassilev, A. (2025) · doi:10.6028/nist.ai.100-2e2025 · Cited by 182
Generative Low-Bitwidth Data Free Quantization
Xu, S., Li, H., Zhuang, B., Liu, J., Cao, J., Liang, C., & Tan, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58610-2_1 · Cited by 187
Rather, I. H., Kumar, S., & Gandomi, A. H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10859-3 · Cited by 168
Theoretical guarantees for permutation-equivariant quantum neural networks
Schatzki, L., Larocca, M., Nguyen, Q. T., Sauvage, F., & Cerezo, M. (2024) · npj Quantum Information · doi:10.1038/s41534-024-00804-1 · Cited by 194
“Namely, in classical machine learning researchers have studied overparametrization through the optics of generalization [109, 144–147], while others have investigated the effect of overparametrization on the training processes.”
BackgroundPoisoning Web-Scale Training Datasets is Practical
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2024) · 2024 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp54263.2024.00179 · Cited by 138
“This quantity-over-quality tradeoff has so far been deemed acceptable, both because modern neural networks are extremely resilient to large amounts of label noise [79], [114], and because training on noisy data can even improve model utility on out-of-distribution data [74], [75].”
BackgroundShen, C., Gonzalez, Y., Klages, P., Qin, N., Jung, H., Chen, L., Nguyen, D. T. D., Jiang, S. B., & Jia, X. (2019) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ab18bf · Cited by 156
Sun, Y., Deng, K., Ren, K., Liu, J., Deng, C., & Jin, Y. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2023.12.011 · Cited by 159
Training data influence analysis and estimation: a survey
Hammoudeh, Z., & Lowd, D. (2024) · Machine Learning · doi:10.1007/s10994-023-06495-7 · Cited by 228
Poisoning Web-Scale Training Datasets is Practical
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2023) · arXiv · doi:10.48550/arxiv.2302.10149 · Cited by 138
“This quantity-over-quality tradeoff has so far been deemed acceptable, both because modern neural networks are extremely resilient to large amounts of label noise [79], [114], and because training on noisy data can even improve model utility on out-of-distribution data [74], [75].”
BackgroundDeep learning technologies for shield tunneling: Challenges and opportunities
Zhou, C., Gao, Y., Chen, E. J., Ding, L., & Qin, W. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.104982 · Cited by 139
UAVs meet LLMs: Overviews and perspectives towards agentic low-altitude mobility
Tian, Y., Lin, F., Li, Y., Zhang, T., Zhang, Q., Fu, X., Huang, J., Dai, X., Wang, Y., Tian, C., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103158 · Cited by 174
“Furthermore, these models exhibit notable limitations in human-machine interaction and task collaboration [317, 318, 319].”
(2019) · doi:10.1145/3474085.3475591 · Cited by 134
DISC: Learning from Noisy Labels via Dynamic Instance-Specific Selection and Correction
Li, Y., Han, H., Shan, S., & Chen, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02305 · Cited by 198
“[55] observe that the capacity of DNNs is sufficient for memorizing the entire data set.”
BackgroundRobust Federated Learning With Noisy Labels
Yang, S., Park, H., Byun, J., & Kim, C. (2022) · IEEE Intelligent Systems · doi:10.1109/mis.2022.3151466 · Cited by 156
Efficient Decentralized Deep Learning by Dynamic Model Averaging
Kamp, M., Adilova, L., Sicking, J., Hüger, F., Schlicht, P., Wirtz, T., & Wrobel, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-10925-7_24 · Cited by 125
VeriFi: Towards Verifiable Federated Unlearning
Gao, X., Ma, X., Wang, J., Sun, Y., Li, B., Ji, S., Cheng, P., & Chen, J. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3382321 · Cited by 163
EXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications
Ortigossa, E. S., Gonçalves, T., & Nonato, L. G. (2024) · IEEE Access · doi:10.1109/access.2024.3409843 · Cited by 124
“Zhang et al. [76] indicated modern learning systems tend to fit the training data perfectly while still performing well on the test data.”
BackgroundHassan, S. U., Abdulkadir, S. J., Zahid, M. S. M., & Al-Selwi, S. M. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109569 · Cited by 138
Xiong, P., Lee, S. M., & Chan, G. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.860032 · Cited by 140
“(50), researchers challenged this conventional view by adopting the randomization test to compare how the learning algorithm performs on the natural data vs.”
BackgroundFedIPR: Ownership Verification for Federated Deep Neural Network Models
Li, B.-W., Fan, L.-X., Gu, H.-L., Li, J., & Yang, Q. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3195956 · Cited by 88
“We ascribe the stable detection rate hT to the generalization capability of over-parameterized networks as demonstrated in [49], [50].”
MethodJenni, S., & Favaro, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01249-6_38 · Cited by 119
Understanding quantum machine learning also requires rethinking generalization
Gil-Fuster, E., Eisert, J., & Bravo-Prieto, C. (2024) · Nature Communications · doi:10.1038/s41467-024-45882-z · Cited by 164
“For each i ∈ [ N ] , ˆ ρ i is efficiently preparable with a PQC.”
InfluentialA Dataset and Benchmark for Multimodal Biometric Recognition Based on Fingerprint and Finger Vein
Ren, H., Sun, L., Guo, J. W., & Han, C. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3175599 · Cited by 108
Recent Theoretical Advances in Non-Convex Optimization
Danilova, M., Dvurechensky, P. E., Gasnikov, A. V., Gorbunov, E., Guminov, S. V., Kamzolov, D., & Shibaev, I. (2022) · Springer optimization and its applications · doi:10.1007/978-3-031-00832-0_3 · Cited by 159
Beyond generalization: a theory of robustness in machine learning
Freiesleben, T., & Grote, T. (2023) · Synthese · doi:10.1007/s11229-023-04334-9 · Cited by 106
“It is debatable whether deep neural nets can exhibit such a strong inductive bias, as it has been shown that they can even fit random noise perfectly (Zhang et al., 2021).”
BackgroundNuSeT: A deep learning tool for reliably separating and analyzing crowded cells
Yang, L., Ghosh, R. P., Franklin, J. M., Chen, S., You, C., Narayan, R. R., Melcher, M. L., & Liphardt, J. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008193 · Cited by 103
Yang, Z., Sinnott, R., Bailey, J., & Ke, Q. (2023) · Knowledge and Information Systems · doi:10.1007/s10115-023-01853-2 · Cited by 121
“Formulti-class classification tasks, the overfitting issue may deteriorate significantly [95], especially when there exists noise in the original data labels [96].”
BackgroundThe Cost-Accuracy Trade-Off in Operator Learning with Neural Networks
de Hoop, M. V., Huang, D. Z., Qian, E., & Stuart, A. M. (2022) · Journal of Machine Learning · doi:10.4208/jml.220509 · Cited by 136
Bakrania, A. K., Joshi, N., Zhao, X., Zheng, G., & Bhat, M. (2023) · Pharmacological Research · doi:10.1016/j.phrs.2023.106706 · Cited by 105
Neural Networks-Based Equalizers for Coherent Optical Transmission: Caveats and Pitfalls
Freire, P. J., Napoli, A., Spinnler, B., Costa, N., Turitsyn, S., & Prilepsky, J. (2021) · IEEE Journal of Selected Topics in Quantum Electronics · doi:10.1109/jstqe.2022.3174268 · Cited by 68
“There has been a plethora of empirical evidence that deep enough NNs can memorize random labels, even when using considerably large datasets [40].”
A graph convolutional autoencoder approach to model order reduction for parametrized PDEs
Pichi, F., Moya, B., & Hesthaven, J. S. (2024) · Journal of Computational Physics · doi:10.1016/j.jcp.2024.112762 · Cited by 98
Explainable ensemble deep learning-based model for brain tumor detection and classification
Hosny, K. M., Mohammed, M., Salama, R. A., & Elshewey, A. M. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10401-0 · Cited by 111
“To avoid overfitting, adopt a dropout layer [35] and L2 regularization [36].”
Lie-algebraic classical simulations for quantum computing
Goh, M. L., Larocca, M., Cincio, Ł., Cerezo, M., & Sauvage, F. (2025) · Physical Review Research · doi:10.1103/3y65-f5w6 · Cited by 143
“…where training a neural network with a capacity larger than that which is necessary to represent the training data distribution may lead to improved performance [86–89] and even provable convergence results [90, 91], rather than to the overfitting and training difficulties one may naïvely expect.”
Reliable Mutual Distillation for Medical Image Segmentation Under Imperfect Annotations
Fang, C., Wang, Q., Cheng, L., Gao, Z., Pan, C., Cao, Z., Zheng, Z., & Zhang, D. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3237183 · Cited by 102
“Theexistence of incorrectly labeled samples in imperfect coarse annotations confuses the learning of CNN-based segmentation models since CNNs are capable of memorizing samples with wrong labels [12].”
BackgroundDar, Y., Muthukumar, V., & Baraniuk, R. G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.02355 · Cited by 116
“, 2017) now appears in a 2021 version of Communications of the ACM (Zhang et al., 2021).”
Bashyam, V., Doshi, J. J., Erus, G., Srinivasan, D., Abdulkadir, A., Singh, A., Habes, M., Yong Fan, A., Masters, C. L., Maruff, P. T., et al. (2021) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.27908 · Cited by 99
“5,6 Restrictions in generalizability are a common characteristic of modeling high dimensional data with many degrees of freedom, which can be overcome partially if suf fi ciently large and diverse training sets are available.”
BackgroundRethinking Graph Auto-Encoder Models for Attributed Graph Clustering
Mrabah, N., Bouguessa, M., Touati, M. F., & Ksantini, R. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3220948 · Cited by 113
Sequence to Sequence ECG Cardiac Rhythm Classification Using Convolutional Recurrent Neural Networks
Pokaprakarn, T., Kitzmiller, R. R., Moorman, J. R., Lake, D., Krishnamurthy, A. K., & Kosorok, M. R. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3098662 · Cited by 92
Learn to Forget: Machine Unlearning via Neuron Masking
Ma, Z., Liu, Y., Liu, X., Liu, J., Ma, J., & Ren, K. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3194884 · Cited by 130
On Learning Contrastive Representations for Learning with Noisy Labels
Li, Y., Liu, S., She, Q., McLeod, A. I., & Wang, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01618 · Cited by 107
PiCO+: Contrastive Label Disambiguation for Robust Partial Label Learning
Wang, H., Xiao, R., Li, Y., Feng, L., Niu, G., Chen, G., & Zhao, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3342650 · Cited by 136
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.