Generalization with Deep Learning: For Improvement on Sensing Capability
The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
With a direct analysis of neural networks, this paper presents a mathematically tight generalization theory to partially address an open problem regarding the generalization of deep learning.Unlike previous bound-based theory, our main theory is quantitatively as tight as possible for every dataset individually, while producing qualitative insights competitively.Our results give insight into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, answering to an open question in the literature.We also discuss limitations of our results and propose additional open problems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665
Understanding Machine Learning: From Theory to Algorithms
Shalev-Shwartz, S., & Ben-David, S. (2014) · doi:10.1017/cbo9781107298019 · Cited by 3,115
Leshno, M., Lin, V. Y., Pinkus, A., & Schocken, S. (1993) · Neural Networks · doi:10.1016/s0893-6080(05)80131-5 · Cited by 2,121
A Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307
Foundations of Machine Learning
Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2018) · doi:10.7551/mitpress/10290.001.0001 · Cited by 1,122
On the Number of Linear Regions of Deep Neural Networks
Montúfar, G., Pascanu, R., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1402.1869 · Cited by 840
Unsupervised Learning for Physical Interaction through Video Prediction
Finn, C., Goodfellow, I., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1605.07157 · Cited by 759
Cited by
Understanding deep learning (still) requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021) · Communications of the ACM · doi:10.1145/3446776 · Cited by 2,400
Super-convergence: very fast training of neural networks using large learning rates
Smith, L. N., & Topin, N. (2019) · Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications · doi:10.1117/12.2520589 · Cited by 1,335
High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks
Wang, H., Wu, X., Huang, Z., & Xing, E. P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00871 · Cited by 638
Deep learning models for solar irradiance forecasting: A comprehensive review
Kumari, P., & Toshniwal, D. (2021) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2021.128566 · Cited by 445
Machine learning in energy economics and finance: A review
Ghoddusi, H., Creamer, G. G., & Rafizadeh, N. (2019) · Energy Economics · doi:10.1016/j.eneco.2019.05.006 · Cited by 436
Deep Learning Scaling is Predictable, Empirically
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., & Zhou, Y. (2017) · arXiv · doi:10.48550/arxiv.1712.00409 · Cited by 427
Why ResNet Works? Residuals Generalize
He, F., Liu, T., & Tao, D. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2966319 · Cited by 406
Generalisation in humans and deep neural networks
Geirhos, R., Temme, C. R. M., Rauber, J., Schütt, H. H., Bethge, M., & Wichmann, F. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.08750 · Cited by 399
Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting
Shu, J., Xie, Q., Yi, L., Zhao, Q., Zhou, S., Xu, Z., & Meng, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07379 · Cited by 385
Accelerating Climate Resilient Plant Breeding by Applying Next-Generation Artificial Intelligence
Harfouche, A. L., Jacobson, D. A., Kainer, D., Romero, J. C., Harfouche, A. H., Scarascia Mugnozza, G., Moshelion, M., Tuskan, G. A., Keurentjes, J. J. B., & Altman, A. (2019) · Trends in Biotechnology · doi:10.1016/j.tibtech.2019.05.007 · Cited by 292
Deep learning generalizes because the parameter-function map is biased towards simple functions
Valle-Pérez, G., Camargo, C. Q., & Louis, A. A. (2018) · arXiv · doi:10.48550/arxiv.1805.08522 · Cited by 105
Deep Learning Under Privileged Information Using Heteroscedastic Dropout
Lambert, J., Sener, O., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00926 · Cited by 93
On the Decision Boundary of Deep Neural Networks
Li, Y., Ding, L., & Gao, X. (2018) · arXiv · doi:10.48550/arxiv.1808.05385 · Cited by 31
Metadata
- Authors
- Chen, Z., Wu, M., & Li, X.
- Venue
- WORLD SCIENTIFIC eBooks
- Publisher
- WORLD SCIENTIFIC
- Published
- 2020-02-23
- OpenAlex
- W2766196653
- Cited by
- 152
- References
- 17
- FWCI
- 9.58
- Type
- book
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.