Understanding deep learning (still) requires rethinking generalization
Open accessThe DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.
Overview
CoChat Summary
TL;DR
Deep neural networks can fit random labels and random pixels with zero training error, so traditional complexity and regularization explanations do not account for their generalization. The paper argues that explicit regularization is neither necessary nor sufficient, and offers a finite-sample expressivity construction plus linear-model implicit-regularization analysis.
Key findings
- On CIFAR10, Inception, AlexNet, and MLPs reach 100% (or 99.82%/99.34%) training accuracy on random labels while test accuracy is near chance: Inception 9.78%, Inception w/o BatchNorm 10.12%, AlexNet 9.86%, MLP 3×512 10.48%, MLP 1×512 10.61%.
- On ImageNet, Inception reaches 95.20% top-1 accuracy on 1.2 million random labels from 1000 categories without hyperparameter tuning; with explicit regularizers on it still reaches ∼90% top-1.
- Explicit regularization may improve generalization but is neither necessary nor sufficient: on CIFAR10 all models still generalize well with regularizers off; on ImageNet top-1 accuracy without regularization is 59.80% vs 0.1% random guessing, and with data augmentation only Inception reaches 72.95% top-1.
- Theoretically, a two-layer ReLU network with p = 2n + d parameters can represent any labeling of any sample of size n in d dimensions; depth-k networks can use O(n/k) parameters per layer. In linear models, SGD converges to a minimum-norm solution, but minimum norm is not predictive: on MNIST the minimum norm with no preprocessing is ∼220 and with wavelet preprocessing ∼390, yet test error drops by a factor of 2.
Methods
- Randomization tests on CIFAR10 and ImageNet ILSVRC 2012: train Inception V3 on ImageNet and smaller Inception, AlexNet, and MLPs on CIFAR10 under true labels, partially corrupted labels, random labels, shuffled pixels, random pixels, and Gaussian pixels, toggling data augmentation, weight decay, and dropout.
- Compare training/test accuracy and convergence time; supplement with a finite-sample expressivity theorem for two-layer ReLU networks and with linear/kernel experiments on MNIST and CIFAR10 (Gaussian kernels, random convolutional filters).
Limitations
- Stated: the paper offers no new theory of generalization; it says a full mathematical characterization remains challenging and that it cannot fully survey recent progress. It also states that minimum-norm intuition is only a small piece of the generalization story and that it does not explain why certain architectures generalize better.
- Evident: the empirical scope is limited to image classification benchmarks (CIFAR10, ImageNet) and a small set of architectures/regularizers, so the findings may not extend to other domains or model families. The theoretical construction uses a depth-2 network with large width, and the paper notes depth-k alternatives with O(n/k) parameters per layer.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 7 Not valid · 12 Not checked · 85 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, by randomizing labels alone we can force the generalization error of a model to jump up considerably without changing the model, its size, hyperparameters, or the optimizer.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The networks fit the corrupted training set perfectly for all the cases.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Partially corrupted labels.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.”
- Also stated in: § 1.3. Finite sample expressivity · plain
“More formally, we exhibit a very simple two-layer ReLU network with p = 2n + d parameters that can express any labeling of any sample of size n in d dimensions.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.3. Finite sample expressivity · plain
70 more claims
Create a free account to see all 104 claims and every source verdict.
- It may state two separate claims as one. 17
- Probably not a substantial claim. 15
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Based on existing research(Saha et al., 2021; Zhang et al., 2021), the following conclusion can be drawn: the gradient updates of a linear layer always lie within the space spanned by its input.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Zhang et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“If a model is successfully trained on a dataset with random labels, which has been shown to be possible with deep neural networks [294], then it has learned the data generation process by memorizing the random labels.”
From From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2023) · cites this paper as [294]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works in privacy-preserving federated learning are mostly developed based on the wellknown privacy-preserving techniques, including: (1) homomorphic encryption (HE), such as Paillier [90], Elgamal [91] and Brakerski-Gentry-Vaikuntanathan cryptosystems [92]; (2) Secure Multiparty Computation (SMC), such as garbled circuits [93] and secret sharing [94]; and (3) differential privacy (DP) [95”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [95]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2.1. Fitting random labels and pixels
“On the CIFAR10 dataset, Alexnet and MLPs all converge to zero loss on the training set.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > Rademacher complexity and VC-dimension.
“(fitting random labels) No No 99.82 9.86 MLP 3 × 512 1,735,178 No Yes 100.0 53.35No No 100.0 52.39 (fitting random labels) No No 100.0 10.48 MLP 1 × 512 1,209,866 No Yes 99.80 50.39No No 100.0 50.51 (fitting random labels) No No 99.34 10.61”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 1.1. The randomization test
“More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error.”
§ Understanding Deep Learning (Still) Requires Rethinking Generalization > 2. EFFECTIVE CAPACITY OF NEURAL NETWORKS
“although it does not reach the perfect 100% top-1 accuracy, 95.20% accuracy is still very surprising for 1.2 million random labels from 1000 categories.”
Citations
Cites
(2014) · Cited by 33,419
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Self-citationReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 1×
“23, 7, 6, 16 Despite significant progress on theoretical understanding of deep learning in the past few years, a full mathematical characterization of the whole story remains challenging.”
BackgroundA Generalized Representer Theorem
Schölkopf, B., Herbrich, R., & Smola, A. J. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_27 · Cited by 1,589 · cited 1×
“We have thus derived the “kernel trick” 34 —albeit in a round-about fashion.”
MethodA Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Bartlett, P. L. (1998) · IEEE Transactions on Information Theory · doi:10.1109/18.661502 · Cited by 1,166
Research works printed but not linked: 8
- Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963.
- Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.
- Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. Sébastien Bubeck, V. Perchet, eds., 2018, 297--299.
- Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334.
- Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736.
- Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Grünwald and E. Hazan, eds. 2015, 1376--1401.
- Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
- Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019.
Cited by
Items 801–900 of 2,667
High-dimensional asymptotics of Langevin dynamics in spiked matrix models
Liang, T., Sen, S., & Sur, P. (2023) · Information and Inference A Journal of the IMA · doi:10.1093/imaiai/iaad042 · Cited by 12
Yezzi-Woodley, K., Terwilliger, A., Li, J., Chen, E., Tappen, M., Calder, J., & Olver, P. J. (2024) · Journal of Human Evolution · doi:10.1016/j.jhevol.2024.103495 · Cited by 11
TC-Net: Detecting Noisy Labels Via Transform Consistency
Yi, R., & Huang, Y. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3115635 · Cited by 11
Empirical risk minimization in the interpolating regime with application to neural network learning
Mücke, N., & Steinwart, I. (2025) · Machine Learning · doi:10.1007/s10994-025-06738-9 · Cited by 13
Wang, H., Chen, J., & Wei, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.07899 · Cited by 13
There is no Double-Descent in Random ForestsPreprint
Buschjäger, S., & Morik, K. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.04409 · Cited by 11
On Margins and Derandomisation in PAC-BayesPreprint
Biggs, F., & Guedj, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03955 · Cited by 13
“This is necessary in cases where our broad class can easily overfit—as for example with many neural networks architectures, which were shown by Zhang et al. (2021) to be able to fit random training labels—since any guarantee must then selectively favour predictors which are reasonable given real data.”
BackgroundCherryChèvre: A fine-grained dataset for goat detection in natural environments
Vayssade, J.-A., Arquet, R., Troupe, W., & Bonneau, M. (2023) · Scientific Data · doi:10.1038/s41597-023-02555-8 · Cited by 11
“On the other hand, a low quality dataset with incorrect or inconsistent annotations, limited or biased images, and noisy or erroneous data can negatively affect the accuracy and generalization ability of the trained detector 12 .”
BackgroundNeural networks memorise personal information from one sample
Hartley, J., Sanchez, P. P., Haider, F., & Tsaftaris, S. A. (2023) · Scientific Reports · doi:10.1038/s41598-023-48034-3 · Cited by 10
Wang, Q., Wu, J., Wang, B., Dou, H., Zhang, B., & Xu, C. (2024) · Journal of Quantitative Spectroscopy and Radiative Transfer · doi:10.1016/j.jqsrt.2024.108994 · Cited by 10
TBC-MI : Suppressing noise labels by maximizing cleaning samples for robust image classification
Li, Y., Guo, Z., Wang, L., & Xu, L. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103801 · Cited by 11
Machine Learning-Based Etiologic Subtyping of Ischemic Stroke Using Circulating Exosomal microRNAs
Bang, J. H., Kim, E. H., Kim, H. J., Chung, J., Seo, W., Kim, G., Lee, D., Kim, H., & Bang, O. Y. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms25126761 · Cited by 10
“Average daily alcohol consumption was divided into three categories: no drink, light–moderate (1–4 drinks per day, 1 drink = 10 g ethanol), and heavy (5 or more drinks per day).”
BackgroundEfficient Model Stealing Defense with Noise Transition Matrix
Wu, D., Fu, C., Wu, W., Xia, W., Zhang, X., Zhou, J., & Zhang, M.-L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02294 · Cited by 10
“Motivated by memorization effects [11, 63] that neural networks can easily overfit noisy data, thus hampering model generalization, we revisit the utilization of noise transition matrix (NTM) in noisy labels learning [43, 58], where the matrix models the noise process.”
MethodNoise-resistant sharpness-aware minimization in deep learning
Su, D., Jin, L., & Wang, J. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106829 · Cited by 10
Data-driven analysis of climate impact on tomato and apple prices using machine learning
Yoon, S.-H., Kim, T., & Kim, D. S. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e41478 · Cited by 10
Deep Networks Are Reproducing Kernel Chains
Heeringa, T. J., Spek, L., & Brüne, C. (2026) · SIAM Journal on Mathematics of Data Science · doi:10.1137/25m1747853 · Cited by 10
Song, Y., Zhang, H., Man, J., Jin, X., & Li, Q. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3540970 · Cited by 12
“This characteristic implies that the network fundamentally does not deduce features but instead memorizes them using weights [47].”
RED: Residual estimation diffusion for low-dose PET sinogram reconstruction
Ai, X., Bin, H., Chen, F., Shi, L., Li, B., Wang, S., & Liu, Q. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103558 · Cited by 10
Hui, Y., Wang, Q., & Zhou, X. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103147 · Cited by 10
QMix: Quality-Aware Learning With Mixed Noise for Robust Retinal Disease Diagnosis
Hou, J., Xu, J., Feng, R., & Chen, H. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3562614 · Cited by 16
“Most existing LNL methods aim to address this by filtering correctly labeled samples from the entire noisy dataset [5], [6], [7].”
Enhancing lettuce classification: Optimizing spectral wavelength selection via CCARS and PLS-DA
Dilillo, N., Sanna, A., Belcore, E., Smith, K. N., Piras, M., Montrucchio, B., & Ferrero, R. (2025) · Smart Agricultural Technology · doi:10.1016/j.atech.2025.100962 · Cited by 10
A hybrid deep learning framework for improving short-term precipitation forecasts
Salehi, S., & Neyshabouri, S. A. A. S. (2025) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2025.106635 · Cited by 11
Human Heterogeneity Invariant Stress Sensing
Xiao, Y., Sharma, H., Kaur, S., Bergen‐Cico, D., & Salekin, A. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3749465 · Cited by 11
“The HHISS framework relies on IRM-regularized over-parameterized networks in its initial step since it provides the capacity to capture a wide range of potential patterns and relationships in the data [143].”
Interplay between Bayesian neural networks and deep learning: A survey
Chen, Y., Yu, S. S., Li, Z., Eshraghian, J. K., & Lim, C. P. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114438 · Cited by 12
Deep learning, transformers and graph neural networks: a linear algebra perspective
Baggag, A., & Saad, Y. (2025) · Numerical Algorithms · doi:10.1007/s11075-025-02218-2 · Cited by 11
“However, the paper [34] shows by means of experiments that looking at DL from the angle of minimizing the loss function fails to explain these nice generalization properties.”
Rahman, T., Deb, N., Larguech, S., Moniruzzaman, M., Ghosh, A. K., Sami, A. A., Abd.Aziz, A. B., & Al‐Bawri, S. S. (2026) · Scientific Reports · doi:10.1038/s41598-025-34262-2 · Cited by 10
$\ell _1$ Regularization in Two-Layer Neural Networks
Li, G., Gu, Y., & Ding, J. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3129698 · Cited by 10
Skipout: An Adaptive Layer-Level Regularization Framework for Deep Neural Networks
Moayed, H., & Mansoori, E. G. (2022) · IEEE Access · doi:10.1109/access.2022.3178091 · Cited by 10
Efficient Meta label correction based on Meta Learning and bi-level optimization
Mallem, S., Hasnat, A., & Nakib, A. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105517 · Cited by 10
A convolutional neural network framework for classifying inappropriate online video contents
Tanantong, T., & Yongwattana, P. (2022) · IAES International Journal of Artificial Intelligence · doi:10.11591/ijai.v12.i1.pp124-136 · Cited by 10
To Overfit, or Not to Overfit: Improving the Performance of Deep Learning-Based SCA
Rezaeezade, A., Perin, G., & Picek, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17433-9_17 · Cited by 10
“This simplifies the optimization problem so that we can use very complex models for simple problems and still reach extremely good generalization [30].”
MethodBrudner, S. N., Pearson, J., & MOONEY, R. D. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011051 · Cited by 12
“We were unsure of the complexity of this problem, and initially deployed an overparameterized model (Fig 4A; 141089 to 143402 trained parameters depending on k; see methods) because these often generalize well [39,40].”
MethodTimon, C., Hussey, P., Lee, H., Murphy, C., Rai, H. V., & Smeaton, A. F. (2023) · Digital Health · doi:10.1177/20552076231184084 · Cited by 14
“deep learning approaches are regarded as best in terms of accuracy but their downside is that they need much training data in order to be reliable.(22) In addition, once the models have been created they cannot offer any explanation for recommendations or outputs that they generate.”
BackgroundSubject-Level Membership Inference Attack via Data Augmentation and Model Discrepancy
Liu, Y., Jiang, P., & Zhu, L. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3318950 · Cited by 10
“This improvement can be attributed to the reduction of losses by the GAN and the contribution that the global model is able to bring by better capturing information about the subject’s data at a later stage [29].”
BackgroundImproving Medical Image Classification in Noisy Labels Using only Self-supervised Pretraining
Khanal, B., Bhattarai, B., Khanal, B., & Linte, C. A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44992-5_8 · Cited by 17
Multi-teacher Self-training for Semi-supervised Node Classification with Noisy Labels
Liu, Y., Wu, Z., Lu, Z., Wen, G., Ma, J., Lu, G., & Zhu, X. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3613117 · Cited by 19
Studying the Impact of Augmentations on Medical Confidence Calibration
Rao, A., Lee, J., & Aalami, O. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00260 · Cited by 14
“Increasing the network depth and width of CNNs typically increases accuracy [61] however this has negative effects on calibration.”
BackgroundRethinking Cost-Sensitive Classification in Deep Learning via Adversarial Data Augmentation
Chen, Q., Kontar, R. A., Nouiehed, M., Yang, X. J., & Lester, C. A. (2024) · INFORMS Journal on Data Science · doi:10.1287/ijds.2022.0033 · Cited by 14
Predicting mortality after transcatheter aortic valve replacement using preprocedural CT
Brüggemann, D., Cener, D., Kuzo, N., Anwer, S., Kebernik, J., Eberhard, M., Alkadhi, H., Tanner, F. C., & Konukoğlu, E. (2024) · Scientific Reports · doi:10.1038/s41598-024-63022-x · Cited by 11
“We believe this is due to large neural networks such as the one used in this study being prone to overfitting 13 .”
Crespo-Otero, A., Esteve, P., & Zanin, M. (2024) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2024.115359 · Cited by 10
Peng, L., Wei, T., Chen, X., Chen, X., Sun, R., Wan, L., Chen, J., & Zhu, X. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3502461 · Cited by 12
“…of growth in prediction errors caused by increased simulated label noise compared to the rate of growth in simulated label noise itself, suggesting a considerable resilience or robustness of ConvNets to such simulated label noise [17], [18], [19], [20], [21], [22], [23], [24], [25] but see [26].”
Learning With Noisy Low-Cost MOS for Image Quality Assessment via Dual-Bias Calibration
Wang, L., Wu, Q., Yuan, D., Ngan, K. N., Li, H., Meng, F., & Xu, L. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3543014 · Cited by 10
“Zhang et al. [38] demonstrated that DNNs can easily fit the entire training dataset with any corrupted label ratio, ultimately leading to less generality on the test dataset.”
Trustworthy Machine Learning: From Data to Models
Han, B., Yao, J., Liu, T., Li, B., Koyejo, S., & Liu, F. (2025) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000043 · Cited by 28
Information-Theoretic Generalization Bounds for Deep Neural Networks
He, H., & Goldfeld, Z. (2025) · IEEE Transactions on Information Theory · doi:10.1109/tit.2025.3568697 · Cited by 17
“The seminal works [18], [44] analyzed the finite-sample expressivity of DNNs and proved that any two-layer NN with ReLU activation can generalize, given su ffi ciently many parameters.”
InfluentialKhagi, B., & Kwon, G. (2021) · IEEE Access · doi:10.1109/access.2021.3093455 · Cited by 10
“Inan experiment [21] to understand the effect of randomness in NN, the Gaussian function was used to add noise to the input and bring non-uniformity in the signal by adding random pixels to the source image.”
MethodLANCET: labeling complex data at scale
Zhang, H., Cao, L., Madden, S. R., & Rundensteiner, E. A. (2021) · Proceedings of the VLDB Endowment · doi:10.14778/3476249.3476269 · Cited by 15
Data Cleansing for Salt Dome Dataset With Noise Robust Network on Segmentation Task
Chung, Y., Lu, W., & Tian, X. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2022.3170497 · Cited by 11
When Speaker Recognition Meets Noisy Labels: Optimizations for Front-Ends and Back-Ends
Li, L., Tong, F., & Hong, Q. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3169977 · Cited by 14
Forecasting the future clinical events of a patient through contrastive learning
Zhang, Z., Yan, C., Zhang, X., Nyemba, S., & Malin, B. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac086 · Cited by 14
Pereira, E. J. S., Carneiro, G., & Cordeiro, F. R. (2022) · SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) · doi:10.1109/sibgrapi55357.2022.9991791 · Cited by 10
Preserving Differential Privacy in Deep Learning Based on Feature Relevance Region Segmentation
Wang, F., Xie, M., Tan, Z., Li, Q., & Wang, C. (2023) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2023.3244174 · Cited by 10
ENInst: Enhancing weakly-supervised low-shot instance segmentation
Ye-Bin, M., Choi, D., Kwon, Y., Kim, J., & Oh, T.-H. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109888 · Cited by 10
Private Image Generation with Dual-Purpose Auxiliary Classifier
Chen, C., Liu, D., Ma, S., Nepal, ., & Xu, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01950 · Cited by 15
“Unfortunately, recent works have shown that GANs are not safe from leaking sensitive information about training sample [3, 29, 40] as GANs are subject to model inversion attacks and membership inference attacks in both whitebox and black-box settings [15, 23, 35, 42].”
BackgroundSigCo: Eliminate the inter-class competition via sigmoid for learning with noisy labels
Chen, A., Xu, F., Zeng, T., Lyu, X., & Li, X. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111651 · Cited by 10
Progressive Feedforward Collapse of ResNet Training
Wang, S., Gai, K., & Zhang, S. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3620798 · Cited by 10
“Zhang et al. illustrate this phenomenon by a theoretical construction of a two-layer network [4].”
On the Equivalence between Neural Network and Support Vector MachinePreprint
Chen, Y., Huang, W., Nguyen, L. M., & Weng, T.-W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.06063 · Cited by 21
“This differs from traditional complexity measures that cannot explain the random label phenomenon [54].”
BackgroundTight bounds for minimum l1-norm interpolation of noisy dataPreprint
Wang, G., Donhauser, K., & Yang, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05987 · Cited by 12
“Recent experimental studies (Belkin et al., 2019; Zhang et al., 2021) reveal that in the modern highdimensional regime, models that perfectly fit noisy training data can still generalize well.”
BackgroundDLC: Dynamic Loss Correction for Cross-Domain Remotely Sensed Segmentation
He, Q., Yan, Z., Diao, W., & Sun, X. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3402127 · Cited by 10
“The existence of noisy pseudo-labels can easily lead to overfitting of neural networks, which can seriously hinder the generalization of adaptive models [16], [21].”
BackgroundJeon, M., Choi, I.-H., Seo, S., & Kim, S.-W. (2024) · IEEE Transactions on Components Packaging and Manufacturing Technology · doi:10.1109/tcpmt.2024.3454991 · Cited by 10
“Therefore, unsupervised methods [13], including the use of an autoencoder [14], have been primarily considered for high value-added manufacturing, such as semiconductor manufacturing.”
MethodDeep learning can detect elbow disease in dogs screened for elbow dysplasia
Hauback, M. N., Huynh, B. N., Steiro, S. E. D., Groendahl, A. R., Bredal, W. P., Tomić, O., Futsæther, C. M., & Skogmo, H. K. (2024) · Veterinary Radiology & Ultrasound · doi:10.1111/vru.13465 · Cited by 9
Khanal, B., Shrestha, P., Amgain, S., Khanal, B., Bhattarai, B., & Linte, C. A. (2024) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc53108.2024.10782929 · Cited by 9
“Supervised training with inaccurately annotated or noisy labels can impair a model’s generalizability, resulting in subpar test performance [13], [27], [11], [10].”
Song, Y., Ding, L., Peng, J., Song, L., & Zhang, X. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107472 · Cited by 9
Practical multi-fidelity machine learning: fusion of deterministic and Bayesian models
Yi, J., Cheng, J., & Bessa, M. A. (2026) · Computational Mechanics · doi:10.1007/s00466-026-02799-2 · Cited by 10
“However, they can be prone to over-fitting, especially in the presence of noisy data [2], and are not able to predict uncertainty.”
Siswara, D., Soleh, A. M., & Wigena, A. H. (2024) · Scientific Journal of Informatics · doi:10.15294/sji.v11i3.4067 · Cited by 9
SAR Images Change Detection Based on Attention Mechanism-Convolutional Wavelet Neural Network
Jiahui, E., Wang, L., Zhao, C., Mathiopoulos, P. T., Ohtsuki, T. O., & Adachi, F. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3557434 · Cited by 9
Contrastive learning for robust representations of neutrino data
Wilkinson, A., Radev, R., & Monsalve, S. A. (2025) · Physical review. D/Physical review. D. · doi:10.1103/physrevd.111.092011 · Cited by 10
“However, while explicit regularisation proves beneficial, it may not suffice, leaving models prone to poor generalisation when confronted with target distributions divergent from the training one [17, 18].”
Malekjani, N., Kharaghani, A., & Tsotsas, E. (2025) · Chemical Engineering Science · doi:10.1016/j.ces.2025.121911 · Cited by 9
Jahan, S., Shah, M. B., Mahbub, P., & Rahman, M. M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00224 · Cited by 10
“Furthermore, there exists a direct mapping between static features to specific DNN code sections, which could be faulty [50], [51].”
L-VAE: variational auto-encoder with learnable beta for disentangled representation
Ozcan, H. M., Kalkan, S., & Yarman-Vural, F. T. (2025) · Machine Vision and Applications · doi:10.1007/s00138-025-01728-2 · Cited by 11
“Deep learning architectures inherently possess limitations concerning their capacity for generalization, explainability, and interpretability [2, 14, 33, 43].”
Deep Learning-Based Shape Classification for Hyperspectral-Imaged Microplastics
Liu, Y., Zhao, G., & Liu, F. (2025) · Analytical Chemistry · doi:10.1021/acs.analchem.5c02683 · Cited by 11
DT-UFC: Universal Large Model Feature Coding via Peaky-to-Balanced Distribution Transformation
Gao, C., Liu, Z., Li, L., Liu, D., Sun, X., & Lin, W. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755814 · Cited by 15
“Such inputs tend to encourage neural networks to memorize dominant patterns while neglecting rare but semantically meaningful variations [4, 45].”
Michaud, F., Sueur, J., Sèbe, F., Cesne, M. L., & Haupert, S. (2025) · Ecological Indicators · doi:10.1016/j.ecolind.2025.114475 · Cited by 10
Ting, Y.-S. (2026) · Annual Review of Astronomy and Astrophysics · doi:10.1146/annurev-astro-051024-021708 · Cited by 20
Amyar, A., Fahmy, A. S., Guo, R., Nakata, K., Sai, E., Rodriguez, J., Cirillo, J., Pareek, K., Kim, J. J., Judd, R. M., et al. (2023) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.28739 · Cited by 9
“37 The basic idea of domain adaptation is to learn a good representation from the training data set (source domain) while approximating the distribution of the test data set (target domain).”
BackgroundKim, M. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.04.014 · Cited by 9
USDNL: Uncertainty-Based Single Dropout in Noisy Label Learning
Xu, Y., Niu, X., Yang, J., Drew, S., Zhou, J., & Chen, R. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26264 · Cited by 16
“General regularization methods (Zhang et al. 2021) fail in noisy label learning due to the strong fitting ability of the model.”
BackgroundRathore, P. W. S., Hussain, M., Malik, M. B., & Amin, Y. (2023) · Journal of Applied Geophysics · doi:10.1016/j.jappgeo.2023.105199 · Cited by 11
Liu, B., He, L., Xie, Y., Xiang, Y., Zhu, L., & Ding, W. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102071 · Cited by 15
Liu, Z., Jin, S., Liu, X., Yang, Q., Li, Q., Zang, J., Li, Z., Hu, T., Guo, Z., Wu, J., et al. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3333344 · Cited by 9
Enhanced Meta Label Correction for Coping with Label Corruption
Taraday, M. K., & Baskin, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01493 · Cited by 9
Sample-wise Label Confidence Incorporation for Learning with Noisy Labels
Ahn, C., Kim, K., Baek, J.-W., Lim, J., & Han, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00175 · Cited by 16
“This issue is particularly problematic given the capacity of deep networks to overfit data samples [35].”
BackgroundThe Blame Problem in Evaluating Local Explanations and How to Tackle It
Rahnama, A. H. A. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-50396-2_4 · Cited by 9
Even small correlation and diversity shifts pose dataset-bias issues
Bissoto, A. E., Barata, C., Junior, E. A. D. V., & Avila, S. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.01.026 · Cited by 13
Zhou, R., Gan, W., Wang, F., Yang, Z. X., Huang, Z., & Gan, H. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.105981 · Cited by 9
GloNets: Globally Connected Neural Networks
Cecco, A. D., Metta, C., Fantozzi, M., Morandin, F., & Parton, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-58547-0_5 · Cited by 9
Chudobiński, C., Świderski, B., Antoniuk, I., & Kurek, J. (2024) · Cancers · doi:10.3390/cancers16081564 · Cited by 9
“In [47], norm-based control, sharpness, and robustness are considered as a generalization measure, while the authors of [48] explore the reasons behind good generalization in large neural networks.”
BackgroundRobust consistency learning for facial expression recognition under label noise
Tan, Y., Xia, H., & Song, S. (2024) · The Visual Computer · doi:10.1007/s00371-024-03558-1 · Cited by 9
“Previous studies [16, 17, 39, 57, 59] have found that the cross-entropy (CE) loss, which is prevalent in deep classi-fiers, is prone to fitting noisy labels.”
Tuning parameters of deep neural network training algorithms pays off: a computational study
Coppola, C., Papa, L., Boresta, M., Amerini, I., & Palagi, L. (2024) · Top · doi:10.1007/s11750-024-00683-x · Cited by 12
“Advanced studies in this direction have been carried out over the past years, starting from Zhang et al. (2016) and Zhang et al. (2021), who reconsidered the classical bias–variance trade-off, remarking that most of the state-of-the-art neural models, especially in the field of image classification,…”
Data Augmentation's Effect on Machine Learning Models when Learning with Imbalanced Data
Dablain, D. A., & Chawla, N. V. (2024) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa61799.2024.10722815 · Cited by 10
“These empirical results align with Zhang et al.’s observation that a neural network is capable of arbitrarily memorizing random data and associating it with a label during training [61].”
ResultLi, Y., Maharana, S. K., & Guo, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73650-6_16 · Cited by 9
Golowich, N., Moitra, A., & Rohatgi, D. (2024) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs61266.2024.00117 · Cited by 12
“We know that the ability of deep neural networks to generalize beyond their training data revolves around properties of natural data [3].”
Mintoo, A. A., Nabil, A. R., Alam, M. A., & Ahmad, I. (2024) · Innovatech Engineering Journal · doi:10.70937/itej.v1i01.9 · Cited by 12
Inverse Evolution Layers: Physics-Informed Regularizers for Image Segmentation
Liu, C., Qiao, Z., Li, C., & Schönlieb, C. (2025) · SIAM Journal on Mathematics of Data Science · doi:10.1137/24m1633662 · Cited by 9
Zhu, H., Zhao, Y., Zhao, L., Yang, R., & Han, Z. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.110071 · Cited by 9
GLiRA: Closed-Box Membership Inference Attack via Knowledge Distillation
Galichin, A. V., Pautov, M., Zhavoronkin, A., Rogov, O. Y., & Oseledets, I. V. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3550068 · Cited by 15
“R ECENT studies [1], [2], [3] have shown that machine learning models are used to memorize the training data and, hence, have a vast spectrum of data-driven vulnerabilities, such as membership inference attacks [4], [5], [6], [7], [8], [9], [10], [11], [12].”
Zhu, R., Niu, G., Wang, P., He, C., Qu, Z., & Fang, D. (2025) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2025.117923 · Cited by 9
Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding
Kong, L., Xu, X., Cen, J., Zhang, W., Pan, L., Chen, K., & Liu, Z. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00198 · Cited by 19
“…prevailing research topic, numerous calibration methods have been proposed across various tasks, including image classification [36,62,63,75,84, 88,132], semantic segmentation [8,25,27,52,80,114], object detection [53,64,81,85,92], depth estimation [49,54,91, 112], remote sensing [33,98], medical…”
Fine-grained forecasting of COVID-19 trends at the county level in the United States
Song, T., Clemente, L., Pan, X., Jang, J., Santillana, M., & Lee, K. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01606-1 · Cited by 10
“For each forecasting horizon h , we computed FIGI-Net’s RMSE and RRMSE scores across all counties, over the whole time period.”
Facial surgery preview based on the orthognathic treatment prediction
Han, H., Zhang, C., Zhu, L., Singh, P., Hsung, T.-C., Leung, Y. Y., Komura, T., Wang, W., & Gu, M. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108781 · Cited by 9
Self-citationAdaptive Cross Entropy for ultrasmall object detection in Computed Tomography with noisy labels
Chen, H., Tan, W., Li, J., Guan, P., Wu, L., Yan, B., Li, J., & Wang, Y. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105763 · Cited by 10
Metadata
- Authors
- Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O.
- Affiliations
- Google (United States) (US)
- Google Brain, Mountain View, CA
- University of California, Berkeley (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- Communications of the ACM
- Published
- 2021-02-22
- Volume
- 64
- Issue
- 3
- Pages
- 107-115
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2566079294
- Cited by
- 2,400
- References
- 47
- Citations by year
- 2026: 295 · 2025: 558 · 2024: 579 · 2023: 500 · 2022: 316 · 2021: 108
- FWCI
- 132.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-nc-sa)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- generalization, deep neural network, overfitting, overparameterization, expressivity, stochastic gradient descent, regularization, image classification, convolutional neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.