CoChat
Sign Up
PreprintCited by 853Advanced Neural Network ApplicationsNeural Networks and ApplicationsDomain Adaptation and Few-Shot Learning

Predicting Parameters in Deep Learning

Open access
Denil, Misha, Shakibi, Babak, Dinh, Laurent, Ranzato, Marc'Aurelio, de Freitas, Nando·2013·arXiv
DOI 10.48550/arxiv.1306.0543arXiv 1306.0543
Open this paper in CoChat — free accountOpen PDF

Abstract

We demonstrate that there is significant redundancy in the parameterization of several deep learning models. Given only a few weight values for each feature it is possible to accurately predict the remaining values. Moreover, we show that not only can the parameter values be predicted, but many of them need not be learned at all. We train several different architectures by learning only a small number of weights and predicting the rest. In the best case we are able to predict more than 95% of the weights of a network without any drop in accuracy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1306.0543

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Large networks seem necessary for training but learn redundant representations.

    “Researchers have particularly noted that large networks seem to be necessary for training, but learn redundant representations in the process (Denil et al., 2013).”

    From Sequence-Level Knowledge Distillation (2016) · cites this paper as Denil et al., 2013

    The Abstract (p. 1) reports significant redundancy in model parameters, and the Introduction (p. 2) explains that individual feature weights can be redundant. The paper does not say that large networks are necessary for training; its finding concerns redundant parameterization, not necessarily redundant representations. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 23 · Cited by 853

Research works printed but not linked: 25

  1. [1][1] Y. Bengio. Deep learning of representations: Looking forward. Technical Report arXiv:1305.0445, Universit´ e de Montr´ eal, 2013. - · arXiv
  2. [2][2] D. Cires ¸an, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. In IEEE Computer Vision and Pattern Recognition , pages 3642-3649, 2012. -
  3. [3][3] D. Cires ¸an, U. Meier, and J. Masci. High-performance neural networks for visual object classification. arXiv:1102.0183 , 2011. - · arXiv
  4. [4][4] A. Coates, A. Karpathy, and A. Ng. Emergence of object-selective features in unsupervised feature learning. In Advances in Neural Information Processing Systems , pages 2690-2698, 2012. -
  5. [5][5] A. Coates and A. Y. Ng. Selecting receptive fields in deep networks. In Advances in Neural Information Processing Systems , pages 2528-2536, 2011. -
  6. [7][7] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems , pages 1232-1240, 2012. -
  7. [8][8] L. Deng, D. Yu, and J. Platt. Scalable stacking and learning for building deep architectures. In International Conference on Acoustics, Speech, and Signal Processing , pages 2133-2136, 2012. -
  8. [9][9] I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. In International Conference on Machine Learning , 2013. -
  9. [10][10] K. Gregor and Y. LeCun. Emergence of complex-like cells in a temporal product network with local receptive fields. arXiv preprint arXiv:1006.0448 , 2010. - · arXiv
  10. [11][11] C. G¨ ulc ¸ehre and Y. Bengio. Knowledge matters: Importance of prior information for optimization. In International Conference on Learning Representations , 2013. -
  11. [12][12] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. CoRR , abs/1207.0580, 2012. - · arXiv
  12. [13][13] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems , pages 1106-1114, 2012. -
  13. [14][14] K. Lang and G. Hinton. Dimensionality reduction and prior knowledge in e-set recognition. In Advances in Neural Information Processing Systems , 1990. -
  14. [15][15] Q. V. Le, A. Karpenko, J. Ngiam, and A. Y. Ng. ICA with reconstruction cost for efficient overcomplete feature learning. Advances in Neural Information Processing Systems , 24:1017-1025, 2011. -
  15. [16][16] Q. V. Le, M. Ranzato, R. Monga, M. Devin, K. Chen, G. Corrado, J. Dean, and A. Ng. Building highlevel features using large scale unsupervised learning. In International Conference on Machine Learning , 2012. -
  16. [17][17] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. -
  17. [18][18] Y. LeCun, J. S. Denker, S. Solla, R. E. Howard, and L. D. Jackel. Optimal brain damage. In Advances in Neural Information Processing Systems , pages 598-605, 1990. -
  18. [19][19] K.-F. Lee and H.-W. Hon. Speaker-independent phone recognition using hidden markov models. Acoustics, Speech and Signal Processing, IEEE Transactions on , 37(11):1641-1648, 1989. -
  19. [20][20] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proc. 27th International Conference on Machine Learning , pages 807-814. Omnipress Madison, WI, 2010. -
  20. [22][22] R. Rigamonti, A. Sironi, V. Lepetit, and P. Fua. Learning separable filters. In IEEE Computer Vision and Pattern Recognition , 2013. -
  21. [23][23] R. Rubinstein, M. Zibulevsky, and M. Elad. Double sparsity: learning sparse dictionaries for sparse signal approximation. IEEE Transactions on Signal Processing , 58:1553-1564, 2010. -
  22. [24][24] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. Nature , 323(6088):533-536, 1986. -
  23. [25][25] J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis . Cambridge University Press, New York, NY, USA, 2004. -
  24. [26][26] K. Swersky, M. Ranzato, D. Buchman, B. Marlin, and N. Freitas. On autoencoders and score matching for energy based models. In International Conference on Machine Learning , pages 1201-1208, 2011. -
  25. [27][27] P. Vincent and Y. Bengio. A neural support vector network architecture with adaptive kernels. In International Joint Conference on Neural Networks , pages 187-192, 2000.

Authors
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., & de Freitas, N.
Affiliations
  • University of Oxford (GB)
  • University of British Columbia (CA)
  • Université de Montréal (CA)
  • Meta (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2013
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1306.0543
arXiv
1306.0543
OpenAlex
W2952899695
Cited by
853
References
23
Citations by year
2026: 1 · 2025: 28 · 2024: 56 · 2023: 81 · 2022: 79 · 2021: 121 · 2020: 117 · 2019: 115 · 2018: 121 · 2017: 76 · 2016: 30 · 2015: 20 · 2014: 8
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
parameter redundancy, weight prediction, deep neural network, parameterization, weight initialization, model compression

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review