Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 402

The DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index.

Model complexity of deep learning: a survey

Hu, Xia, Chu, Lingyang, Pei, Jian, Liu, Weiqing, Bian, Jiang·2021·Knowledge and Information Systems

Abstract

Model complexity is a fundamental problem in deep learning. In this paper, we conduct a systematic overview of the latest studies on model complexity in deep learning. Model complexity of deep learning can be categorized into expressive capacity and effective model complexity. We review the existing studies on those two categories along four important factors, including model framework, model size, optimization process, and data complexity. We also discuss the applications of deep learning model complexity including understanding model generalization, model optimization, and model selection and design. We conclude by proposing several interesting future directions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10115-021-01605-0

Cites

83 of 83 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 83 references

Research works printed but not linked: 28

  1. Adams RA, Fournier JJ (2003) Sobolev spaces. Elsevier, Amsterdam
  2. Allen-Zhu Z, Li Y, Liang Y (2019) Learning and generalization in overparameterized neural networks, going beyond two layers. In: Advances in neural information processing systems, pp 6155–6166
  3. Arora R, Basu A, Mianjy P, Mukherjee A (2018) Understanding deep neural networks with rectified linear units. In: International conference on learning representations
  4. Bartlett PL, Foster DJ, Telgarsky MJ (2017) Spectrally-normalized margin bounds for neural networks. In: Advances in neural information processing systems, pp 6240–6249
  5. Bredon GE (2013) Topology and geometry, vol 139. Springer, Berlin
  6. Bulso N, Marsili M, Roudi Y (2019) On the complexity of logistic regression models. Neural Comput 31(8):1592–1623 · doi:10.1162/neco_a_01207
  7. Carothers NL (2000) Real analysis. Cambridge University Press, Cambridge · doi:10.1017/cbo9780511814228
  8. Cohen N, Shashua A (2016) Convolutional rectifier networks as generalized tensor decompositions. In: International conference on machine learning, pp 955–963
  9. Cook S, Dwork C, Reischuk R (1986) Upper and lower time bounds for parallel random access machines without simultaneous writes. SIAM J Comput 15(1):87–97 · doi:10.1137/0215006
  10. Delalleau O, Bengio Y (2011) Shallow vs. deep sum-product networks. In: Advances in neural information processing systems, pp 666–674
  11. Goodfellow I, Warde-Farley D, Mirza M, Courville A, Bengio Y (2013) Maxout networks. In: International conference on machine learning, PMLR, pp 1319–1327
  12. Hanin B, Sellke M (2017) Approximating continuous functions by relu nets of minimal width. arXiv preprint arXiv:1710.11278 · arXiv
  13. Kalimeris D, Kaplun G, Nakkiran P, Edelman B, Yang T, Barak B, Zhang H (2019) Sgd on neural networks learns functions of increasing complexity. In: Advances in neural information processing systems, pp 3491–3501
  14. Kawaguchi K, Kaelbling LP, Bengio Y (2017) Generalization in deep learning. arXiv preprint arXiv:1710.05468 · arXiv
  15. Keskar NS, Mudigere D, Nocedal J, Smelyanskiy M, Tang PTP (2017) On large-batch training for deep learning: generalization gap and sharp minima. In: International conference on learning representations
  16. Khrulkov V, Novikov A, Oseledets I (2018) Expressive power of recurrent neural networks. In: International conference on learning representations
  17. Kileel J, Trager M, Bruna J (2019) On the expressive power of deep polynomial neural networks. In: Advances in neural information processing systems, pp 10310–10319
  18. Liang T, Poggio T, Rakhlin A, Stokes J (2019) Fisher-rao metric, geometry, and complexity of neural networks. In: The 22nd international conference on artificial intelligence and statistics, PMLR, pp 888–896
  19. Lu Z, Pu H, Wang F, Hu Z, Wang L (2017) The expressive power of neural networks: a view from the width. In: Advances in neural information processing systems, pp 6231–6239
  20. Lundqvist S, Oneto A, Reznick B, Shapiro B (2019) On generic and maximal k-ranks of binary forms. J Pure Appl Algebra 223(5):2062–2079 · doi:10.1016/j.jpaa.2018.08.015
  21. Michel B, Nouy A (2020) Learning with tree tensor networks: complexity estimates and model selection. arXiv preprint arXiv:2007.01165 · arXiv
  22. Neyshabur B (2017) Implicit regularization in deep learning. arXiv preprint arXiv:1709.01953 · arXiv
  23. Neyshabur B, Tomioka R, Srebro N (2015) In search of the real inductive bias: on the role of implicit regularization in deep learning. In: International conference on learning representations (workshop)
  24. Novak R, Bahri Y, Abolafia DA, Pennington J, Sohl-Dickstein J (2018) Sensitivity and generalization in neural networks: an empirical study. In: International conference on learning representations
  25. Raghu M, Poole B, Kleinberg J, Ganguli S, Dickstein JS (2017) On the expressive power of deep neural networks. In: Proceedings of the 34th international conference on machine learning, vol. 70, JMLR, pp 2847–2854
  26. Rasmussen CE, Ghahramani Z (2001) Occams razor. In: Advances in neural information processing systems, pp 294–300
  27. Serra T, Tjandraatmadja C, Ramalingam S (2018) Bounding and counting linear regions of deep neural networks. In: International conference on machine learning, PMLR, pp 4558–4566
  28. Yao ACC (1997) Decision tree complexity and Betti numbers. J Comput Syst Sci 55(1):36–43 · doi:10.1006/jcss.1997.1495

Cited by

See all 402

Metadata

Authors
Hu, X., Chu, L., Pei, J., Liu, W., & Bian, J.
Venue
Knowledge and Information Systems
Published
2021-08-22
Volume
63
Issue
10
Pages
2585-2619
ISSN
  • 0219-1377 (ISSN-L)
  • 0219-3116
OpenAlex
W3195438473
Cited by
402
References
83
FWCI
21.08
Type
journal article
Language
en
Open access
closed
Licence
Funders
Natural Sciences and Engineering Research Council of Canada

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Model complexity of deep learning: a survey. https://app.cochat.ai/papers/10.1007/s10115-021-01605-0
The paper
Hu, X., Chu, L., Pei, J., Liu, W., & Bian, J. (2021). Model complexity of deep learning: a survey. Knowledge and Information Systems. https://doi.org/10.1007/s10115-021-01605-0

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.