The DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index.
Model complexity of deep learning: a survey
Abstract
Model complexity is a fundamental problem in deep learning. In this paper, we conduct a systematic overview of the latest studies on model complexity in deep learning. Model complexity of deep learning can be categorized into expressive capacity and effective model complexity. We review the existing studies on those two categories along four important factors, including model framework, model size, optimization process, and data complexity. We also discuss the applications of deep learning model complexity including understanding model generalization, model optimization, and model selection and design. We conclude by proposing several interesting future directions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
Multilayer feedforward networks are universal approximators
Hornik, K., Stinchcombe, M., & White, H. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90020-8 · Cited by 21,728
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · International Conference on Machine Learning · Cited by 13,995
Research works printed but not linked: 28
- Adams RA, Fournier JJ (2003) Sobolev spaces. Elsevier, Amsterdam
- Allen-Zhu Z, Li Y, Liang Y (2019) Learning and generalization in overparameterized neural networks, going beyond two layers. In: Advances in neural information processing systems, pp 6155–6166
- Arora R, Basu A, Mianjy P, Mukherjee A (2018) Understanding deep neural networks with rectified linear units. In: International conference on learning representations
- Bartlett PL, Foster DJ, Telgarsky MJ (2017) Spectrally-normalized margin bounds for neural networks. In: Advances in neural information processing systems, pp 6240–6249
- Bredon GE (2013) Topology and geometry, vol 139. Springer, Berlin
- Bulso N, Marsili M, Roudi Y (2019) On the complexity of logistic regression models. Neural Comput 31(8):1592–1623 · doi:10.1162/neco_a_01207
- Carothers NL (2000) Real analysis. Cambridge University Press, Cambridge · doi:10.1017/cbo9780511814228
- Cohen N, Shashua A (2016) Convolutional rectifier networks as generalized tensor decompositions. In: International conference on machine learning, pp 955–963
- Cook S, Dwork C, Reischuk R (1986) Upper and lower time bounds for parallel random access machines without simultaneous writes. SIAM J Comput 15(1):87–97 · doi:10.1137/0215006
- Delalleau O, Bengio Y (2011) Shallow vs. deep sum-product networks. In: Advances in neural information processing systems, pp 666–674
- Goodfellow I, Warde-Farley D, Mirza M, Courville A, Bengio Y (2013) Maxout networks. In: International conference on machine learning, PMLR, pp 1319–1327
- Hanin B, Sellke M (2017) Approximating continuous functions by relu nets of minimal width. arXiv preprint arXiv:1710.11278 · arXiv
- Kalimeris D, Kaplun G, Nakkiran P, Edelman B, Yang T, Barak B, Zhang H (2019) Sgd on neural networks learns functions of increasing complexity. In: Advances in neural information processing systems, pp 3491–3501
- Kawaguchi K, Kaelbling LP, Bengio Y (2017) Generalization in deep learning. arXiv preprint arXiv:1710.05468 · arXiv
- Keskar NS, Mudigere D, Nocedal J, Smelyanskiy M, Tang PTP (2017) On large-batch training for deep learning: generalization gap and sharp minima. In: International conference on learning representations
- Khrulkov V, Novikov A, Oseledets I (2018) Expressive power of recurrent neural networks. In: International conference on learning representations
- Kileel J, Trager M, Bruna J (2019) On the expressive power of deep polynomial neural networks. In: Advances in neural information processing systems, pp 10310–10319
- Liang T, Poggio T, Rakhlin A, Stokes J (2019) Fisher-rao metric, geometry, and complexity of neural networks. In: The 22nd international conference on artificial intelligence and statistics, PMLR, pp 888–896
- Lu Z, Pu H, Wang F, Hu Z, Wang L (2017) The expressive power of neural networks: a view from the width. In: Advances in neural information processing systems, pp 6231–6239
- Lundqvist S, Oneto A, Reznick B, Shapiro B (2019) On generic and maximal k-ranks of binary forms. J Pure Appl Algebra 223(5):2062–2079 · doi:10.1016/j.jpaa.2018.08.015
- Michel B, Nouy A (2020) Learning with tree tensor networks: complexity estimates and model selection. arXiv preprint arXiv:2007.01165 · arXiv
- Neyshabur B (2017) Implicit regularization in deep learning. arXiv preprint arXiv:1709.01953 · arXiv
- Neyshabur B, Tomioka R, Srebro N (2015) In search of the real inductive bias: on the role of implicit regularization in deep learning. In: International conference on learning representations (workshop)
- Novak R, Bahri Y, Abolafia DA, Pennington J, Sohl-Dickstein J (2018) Sensitivity and generalization in neural networks: an empirical study. In: International conference on learning representations
- Raghu M, Poole B, Kleinberg J, Ganguli S, Dickstein JS (2017) On the expressive power of deep neural networks. In: Proceedings of the 34th international conference on machine learning, vol. 70, JMLR, pp 2847–2854
- Rasmussen CE, Ghahramani Z (2001) Occams razor. In: Advances in neural information processing systems, pp 294–300
- Serra T, Tjandraatmadja C, Ramalingam S (2018) Bounding and counting linear regions of deep neural networks. In: International conference on machine learning, PMLR, pp 4558–4566
- Yao ACC (1997) Decision tree complexity and Betti numbers. J Comput Syst Sci 55(1):36–43 · doi:10.1006/jcss.1997.1495
Cited by
Deep learning modelling techniques: Current progress, applications, advantages, and challenges
Ahmed, S. F., Alam, M. S. B., Hassan, M., Rozbu, M. R., Ishtiak, T., Rafa, N., Mofijur, M., Shawkat Ali, A. B. M., & Gandomi, A. H. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10466-8 · Cited by 1,079
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“Recent research on the complexity of ML models [162] and their cognitive functional complexity [347] may inspire future research on a unified quantitative evaluation metric.”
BackgroundSelf-citationEvaluation of artificial intelligence techniques in disease diagnosis and prediction
Ghaffar Nia, N., Kaplanoglu, E., & Nasab, A. (2023) · Discover Artificial Intelligence · doi:10.1007/s44163-023-00049-5 · Cited by 426
Deep learning: systematic review, models, challenges, and research directions
Talaei Khoei, T., Ould Slimane, H., & Kaabouch, N. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08957-4 · Cited by 411
Luo, K., Chen, X., Zheng, H., & Shi, Z. (2022) · Journal of Energy Chemistry · doi:10.1016/j.jechem.2022.06.049 · Cited by 292
Metadata
- Authors
- Hu, X., Chu, L., Pei, J., Liu, W., & Bian, J.
- Venue
- Knowledge and Information Systems
- Published
- 2021-08-22
- Volume
- 63
- Issue
- 10
- Pages
- 2585-2619
- ISSN
- 0219-1377 (ISSN-L)
- 0219-3116
- OpenAlex
- W3195438473
- Cited by
- 402
- References
- 83
- FWCI
- 21.08
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Funders
- Natural Sciences and Engineering Research Council of Canada
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Model complexity of deep learning: a survey. https://app.cochat.ai/papers/10.1007/s10115-021-01605-0
- The paper
- Hu, X., Chu, L., Pei, J., Liu, W., & Bian, J. (2021). Model complexity of deep learning: a survey. Knowledge and Information Systems. https://doi.org/10.1007/s10115-021-01605-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.