On the Expressive Power of Deep Architectures
The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 10×
“…deep architectures were unsuccessful (Bengio and LeCun, 2007a), a major breakthrough in Deep Learning methods came about with the use of layer-wise unsupervised learning (Hinton et al., 2006; Bengio et al., 2007; Ranzato et al., 2007), as a way to initialize a deep supervised neural network.”
BackgroundMethodInfluentialParallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citationProceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691
Dimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336 · cited 2×
“This idea was used in unsupervised and semi-supervised contexts (Chopra et al., 2005; Hadsell et al., 2006; Weston et al., 2008), and originates in the much older idea of siamese networks (Bromley et al., 1993).”
MethodA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 3×
“The recent surge in experimental work in the field seems to support this notion, accumulating evidence that in challenging AI-related tasks – such as computer vision (Bengio et al., 2007; Ranzato et al., 2007; Larochelle et al., 2007; Ranzato et al., 2008; Lee et al., 2009; Mobahi et al., 2009; Osindero and Hinton, 20…”
BackgroundGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841 · cited 8×
“In the same year that RBMs were successfully proposed for unsupervised pre-training of deep neural networks, auto-encoders were also shown to help initialize deep neural networks much better than random initialization (Bengio et al., 2007).”
BackgroundMethodInfluentialSelf-citation(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051
Learning a Similarity Metric Discriminatively, with Application to Face Verification
Chopra, S., Hadsell, R., & LeCun, Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.202 · Cited by 4,036
Sparse coding with an overcomplete basis set: A strategy employed by V1?
Olshausen, B. A., & Field, D. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00169-7 · Cited by 3,642 · cited 2×
“Sparse coding was introduced in computational neuroscience (Olshausen and Field, 1997) and produced filters very similar to those observed in cortex visual area V1 (before similar filters were achieved with RBMs, sparse predictive decomposition, and denoising auto-encoders, below).”
BackgroundMethodAn Energy Budget for Signaling in the Grey Matter of the Brain
Attwell, D., & Laughlin, S. B. (2001) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1097/00004647-200110000-00001 · Cited by 3,543 · cited 1×
“Neurons in the cortex are believed to have a distributed and sparse representation (Olshausen and Field, 1997), with around 1-4% of the neurons active at any one time (Attwell and Laughlin, 2001; Lennie, 2003).”
BackgroundNoise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. U., & Hyvärinen, A. (2010) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 3,018
Universal approximation bounds for superpositions of a sigmoidal function
Barron, A. R. (1993) · IEEE Transactions on Information Theory · doi:10.1109/18.256500 · Cited by 2,804
Why Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Bengio, S. (2010) · Journal of Machine Learning Research · Cited by 2,641
Self-citationSalakhutdinov, R., & Hinton, G. E. (2009) · International Conference on Artificial Intelligence and Statistics · Cited by 2,605
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412 · cited 5×
“Many unsupervised learning algorithms are being explored for the first phase, including various methods to train Restricted Boltzmann Machines (RBMs) (Freund and Haussler, 1994; Hinton et al., 2006; Tieleman, 2008; Salakhutdinov and Hinton, 2009; Desjardins et al., 2010) or Deep Boltzmann Machines (Salakhutdinov and H…”
BackgroundInfluentialSIGNATURE VERIFICATION USING A “SIAMESE” TIME DELAY NEURAL NETWORK
BROMLEY, J., BENTZ, J. W., BOTTOU, L., GUYON, I., LECUN, Y., MOORE, C., SÄCKINGER, E., & SHAH, R. (1994) · Series in Machine Perception and Artificial Intelligence · doi:10.1142/9789812797926_0003 · Cited by 2,350
Information processing in dynamical systems: foundations of harmony theory
Smolensky, P. (1986) · The MIT Press eBooks · Cited by 2,253
What is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · cited 1×
“Such models based on approximate inference were the first successful examples of stacking a sparse encoding (Ranzato et al., 2007; Jarrett et al., 2009) into a deep architecture (fine-tuned for supervised classification afterwards, as per the above greedy-layerwise recipe).”
MethodDomain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach
Glorot, X., Bordes, A., & Bengio, Y. (2011) · HAL · Cited by 2,020
Self-citationRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885 · cited 2×
“, 2008), or information retrieval (Salakhutdinov and Hinton, 2007; Salakhutdinov et al., 2007) – deep learning methods significantly out-perform comparable but shallow competitors (e.”
BackgroundDeep Learning of Representations for Unsupervised and Transfer Learning.
Bengio, Y. (2011) · Cited by 1,725
Self-citationAdvances in Neural Information Processing Systems 19 (NIPS 2006)
Simonotto, E., Whalley, H. C., Lawrie, S. M., Murray, L. M., McGonigle, D. J., & Storkey, A. (2006) · Cited by 1,651
The Lack of A Priori Distinctions Between Learning Algorithms
Wolpert, D. H. (1996) · Neural Computation · doi:10.1162/neco.1996.8.7.1341 · Cited by 1,647
Contractive Auto-Encoders: Explicit Invariance During Feature Extraction
Rifai, S., Vincent, P., Muller, X., Glorot, X., & Bengio, Y. (2011) · International Conference on Machine Learning · Cited by 1,603
Self-citationConnectionist learning procedures
Hinton, G. E. (1989) · Artificial Intelligence · doi:10.1016/0004-3702(89)90049-0 · Cited by 1,430
Auto-association by multilayer perceptrons and singular value decomposition
Bourlard, H., & Kamp, Y. (1988) · Biological Cybernetics · doi:10.1007/bf00332918 · Cited by 1,345
(1993) · Cited by 1,173
The cost of cortical computation.
Lennie, P. (2003) · Current Biology · doi:10.1016/s0960-9822(03)00135-0 · Cited by 1,134 · cited 1×
“Neurons in the cortex are believed to have a distributed and sparse representation (Olshausen and Field, 1997), with around 1-4% of the neurons active at any one time (Attwell and Laughlin, 2001; Lennie, 2003).”
BackgroundEfficient Learning of Sparse Representations with an Energy-Based Model
Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0147 · Cited by 1,106
A Connection Between Score Matching and Denoising Autoencoders
Vincent, P. (2011) · Neural Computation · doi:10.1162/neco_a_00142 · Cited by 1,011 · cited 4×
“…generative model of the distribution of the uncorrupted examples (Vincent et al., 2008; Vincent, 2011), and in particular through the Score Matching proxy for log-likelihood (Vincent, 2011): the denoising error corresponds to a form of regularized score matching criterion (Kingma and LeCun, 2010).”
BackgroundInfluentialAn empirical evaluation of deep architectures on problems with many factors of variation
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., & Bengio, Y. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273556 · Cited by 992
Self-citationProceedings of the 26th Conference on Uncertainty in Artificial Intelligence (UAI 2010)
Adams, R. P., Dahl, G. E., & Murray, I. (2010) · Cited by 966
Sparse Feature Learning for Deep Belief Networks
Ranzato, M., Boureau, Y.-L., & LeCun, Y. (2007) · neural information processing systems · Cited by 945
(2007) · Cited by 937
Training restricted Boltzmann machines using approximations to the likelihood gradient
Tieleman, T. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390290 · Cited by 837
Learning distributed representations of concepts.
Hinton, G. E. (1989) · eScholarship (California Digital Library) · Cited by 831
Representational Power of Restricted Boltzmann Machines and Deep Belief Networks
Le Roux, N., & Bengio, Y. (2008) · Neural Computation · doi:10.1162/neco.2008.04-07-510 · Cited by 821
Self-citation(2005) · doi:10.5555/1046920.1088696 · Cited by 756
Almost optimal lower bounds for small depth circuits
Hastad, J. (1986) · Proceedings of the eighteenth annual ACM symposium on Theory of computing - STOC '86 · doi:10.1145/12130.12132 · Cited by 620
Proceedings of the 55th Annual ACM Symposium on Theory of Computing
(2023) · doi:10.1145/3564246 · Cited by 587
Separating the polynomial-time hierarchy by oracles
Yao, A. C.-C. (1985) · 26th Annual Symposium on Foundations of Computer Science (sfcs 1985) · doi:10.1109/sfcs.1985.49 · Cited by 548
(2007) · The MIT Press eBooks · doi:10.7551/mitpress/7496.001.0001 · Cited by 536
An Efficient Learning Procedure for Deep Boltzmann Machines
Salakhutdinov, R., & Hinton, G. (2012) · Neural Computation · doi:10.1162/neco_a_00311 · Cited by 484
Measuring Invariances in Deep Networks
Goodfellow, I., Lee, H., Le, Q. V., Saxe, A., & Ng, A. Y. (2009) · neural information processing systems · Cited by 461
Higher Order Contractive Auto-Encoder
Rifai, S., Mesnil, G., Vincent, P., Muller, X., Bengio, Y., Dauphin, Y., & Glorot, X. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-23783-6_41 · Cited by 397
Self-citationDeep learning via semi-supervised embedding
Weston, J., Ratle, F., & Collobert, R. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390303 · Cited by 380
Sum-Product Networks: A New Deep Architecture
Poon, H., & Domingos, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1202.3732 · Cited by 352
Hinton, G. E. (1999) · 9th International Conference on Artificial Neural Networks: ICANN '99 · doi:10.1049/cp:19991075 · Cited by 341
Deep learning from temporal coherence in video
Mobahi, H., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553469 · Cited by 328 · cited 2×
“The recent surge in experimental work in the field seems to support this notion, accumulating evidence that in challenging AI-related tasks – such as computer vision (Bengio et al., 2007; Ranzato et al., 2007; Larochelle et al., 2007; Ranzato et al., 2008; Lee et al., 2009; Mobahi et al., 2009; Osindero and Hinton, 20…”
BackgroundSum-product networks: A new deep architecture
Poon, H., & Domingos, P. (2011) · 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) · doi:10.1109/iccvw.2011.6130310 · Cited by 315
Unsupervised learning of distributions on binary vectors using two layer networks
Freund, Y., & Haussler, D. H. (1991) · neural information processing systems · Cited by 297
Learning invariant features through topographic filter maps
Kavukcuoglu, K., Ranzato, M., Fergus, R., & LeCun, Y. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206545 · Cited by 274
Using fast weights to improve persistent contrastive divergence
Tieleman, T., & Hinton, G. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553506 · Cited by 274 · cited 1×
“Theory suggests (Younes, 1999) this is a good estimator if the parameter changes are small, but practice revealed (Tieleman, 2008) that it worked even for large updates, in fact giving rise to faster mixing (Tieleman and Hinton, 2009; Breuleux et al., 2011).”
BackgroundThe Curse of Highly Variable Functions for Local Kernel Machines
Bengio, Y., Delalleau, O., & Roux, N. L. (2005) · neural information processing systems · Cited by 255
Self-citationUnsupervised and Transfer Learning Challenge: a Deep Learning Approach
Mesnil, G., Dauphin, Y., Glorot, X., Rifai, S., Bengio, Y., Goodfellow, I., Lavoie, E., Muller, X., Desjardins, G., Warde-Farley, D., et al. (2011) · International Conference on Machine Learning · Cited by 255
Self-citationOn the power of small-depth threshold circuits
H�stad, J., & Goldmann, M. (1991) · Computational Complexity · doi:10.1007/bf01272517 · Cited by 247
Learning Many Related Tasks at the Same Time with Backpropagation
Caruana, R. (1994) · Cited by 242
Advances in Neural Information Processing Systems 19 (NIPS 2006)
Girolami, M., & Zhong, M. (2007) · The MIT Press eBooks · Cited by 219
Justifying and Generalizing Contrastive Divergence
Bengio, Y., & Delalleau, O. (2009) · Neural Computation · doi:10.1162/neco.2008.11-07-647 · Cited by 206
Self-citationAdvances in Neural Information Processing Systems 15 (NIPS 2002)
Bishop, C., Svensén, M., & Williams, C. K. I. (1996) · The MIT Press eBooks · Cited by 205
Deep Learners Benefit More from Out-of-Distribution Examples
Bengio, Y., Bastien, F., Bergeron, A., Boulanger-Lewandowski, N., Breuel, T. M., Chherawala, Y., Cissé, M., Côté, M., Erhan, D., Eustache, J., et al. (2011) · Cited by 193
Self-citationFast Inference in Sparse Coding Algorithms with Applications to Object Recognition
Kavukcuoglu, K., Ranzato, M., & LeCun, Y. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1010.3467 · Cited by 183
Tempered Markov Chain Monte Carlo for training of Restricted Boltzmann Machines
Desjardins, G., Courville, A. C., Bengio, Y., Vincent, P., & Delalleau, O. (2010) · International Conference on Artificial Intelligence and Statistics · Cited by 153
Self-citationOn the convergence of markovian stochastic algorithms with rapidly decreasing ergodicity rates
Younes, L. (1999) · Stochastics and Stochastic Reports · doi:10.1080/17442509908834179 · Cited by 136
Modeling image patches with a directed hierarchy of Markov random fields
Osindero, S., & Hinton, G. E. (2007) · neural information processing systems · Cited by 134
Sutton, C. A., & Jordan, M. I. (2010) · Cited by 100
Non-Local Manifold Tangent Learning
Bengio, Y., & Monperrus, M. (2004) · HAL · Cited by 95
Self-citationDecision trees do not generalize to new variations
Bengio, Y., Delalleau, O., & Simard, C. (2010) · Computational Intelligence · doi:10.1111/j.1467-8640.2010.00366.x · Cited by 89 · cited 2×
“A fixed-kernel Support Vector Machine is considered to have depth 2 (Bengio and LeCun, 2007a) and boosted decision trees to have depth 3 (Bengio et al., 2010).”
BackgroundDeep belief net learning in a long-range vision system for autonomous off-road driving
Hadsell, R., Erkan, A., Sermanet, P., Scoffier, M., Muller, U., & LeCun, Y. (2008) · 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2008.4651217 · Cited by 84
Regularized estimation of image statistics by Score Matching
Kingma, D. P., & LeCun, Y. (2010) · neural information processing systems · Cited by 79
Quickly Generating Representative Samples from an RBM-Derived Process
Breuleux, O., Bengio, Y., & Vincent, P. (2011) · Neural Computation · doi:10.1162/neco_a_00158 · Cited by 67
Self-citationDeep Learning using Robust Interdependent Codes
Larochelle, H., Erhan, D., & Vincent, P. (2009) · International Conference on Artificial Intelligence and Statistics · Cited by 35
Poly-logarithmic independence fools bounded-depth boolean circuits
Braverman, M. (2011) · Communications of the ACM · doi:10.1145/1924421.1924446 · Cited by 30
Herding Dynamic Weights for Partially Observed Random Field ModelsPreprint
Welling, M. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.2605 · Cited by 27
Advances in Pattern Recognition Systems Using Neural Network Technologies
Guyon, I., & Wang, P. S. P. (1994) · Series in machine perception and artificial intelligence · doi:10.1142/2068 · Cited by 19
Research works printed but not linked: 3
- Bengio, Y., Delalleau, O.: Shallow versus deep sum-product networks. In: The Learning Workshop, Fort Lauderdale, Florida (2011)
- Poon, H., Domingos, P.: Sum-product networks for deep learning. In: Learning Workshop. FL, Fort Lauderdale (2011)
- Salakhutdinov, R., Hinton, G.E.: Semantic hashing. In: Proceedings of the 2007 Workshop on Information Retrieval and Applications of Graphical Models (SIGIR 2007). Elsevier, Amsterdam (2007)
Cited by
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Self-citationPractical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259
“Theoretical arguments support a potentially exponential gain in expressive power of deeper architectures (Bengio, 2009; Bengio and Delalleau, 2011) and it would be plausible that with this added expressive power coming from the combinatorics of composed reuse of sub-26 functions could come a…”
BackgroundInfluentialSelf-citationEvolving Deep Convolutional Neural Networks for Image Classification
Sun, Y., Xue, B., Zhang, M., & Yen, G. G. (2020) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2916183 · Cited by 775
On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures
Bianchini, M., & Scarselli, F. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2293637 · Cited by 500
Model complexity of deep learning: a survey
Hu, X., Chu, L., Pei, J., Liu, W., & Bian, J. (2021) · Knowledge and Information Systems · doi:10.1007/s10115-021-01605-0 · Cited by 402
Metadata
- Authors
- Bengio, Y., & Delalleau, O.
- Affiliations
- Université de Montréal (CA)
- ORCID
- Bengio, Yoshua
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer Berlin Heidelberg
- Published
- 2011
- Pages
- 18-36
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W116068320
- Cited by
- 182
- References
- 79
- Citations by year
- 2026: 2 · 2025: 9 · 2024: 8 · 2023: 27 · 2022: 19 · 2021: 19 · 2020: 15 · 2019: 19 · 2018: 15 · 2017: 16 · 2016: 12 · 2015: 10 · 2014: 6 · 2013: 2 · 2012: 3
- FWCI
- 2.75
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Lecture notes in computer science (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- deep learning architectures, expressiveness, function families, deep learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.