Deep Learning of Representations: Looking Forward
The DOI matches its Crossref record; 157 of the 157 references it cites were found in a registry or an index.
Abstract
Deep learning research aims at discovering learning algorithms that discover multiple levels of distributed representations, with higher levels representing more abstract concepts. Although the study of deep learning has already led to impressive theoretical results, learning algorithms and breakthrough experiments, several challenges lie ahead. This paper proposes to examine some of these challenges, centering on the questions of scaling deep learning algorithms to much larger models and datasets, reducing optimization difficulties due to ill-conditioning or local minima, designing more efficient and powerful inference and sampling procedures, and learning to disentangle the factors of variation underlying the observed data. It also proposes a few forward-looking research directions aimed at overcoming these challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Self-citationOptimization by Simulated Annealing
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983) · Science · doi:10.1126/science.220.4598.671 · Cited by 45,134
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
(2010) · Cited by 14,134
(2010) · International Conference on Machine Learning · Cited by 13,995
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Self-citationHinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citation(2012) · Cited by 9,062
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542 · cited 1×
“…the Hessian, and second, they often require line searches that may be computationally inefficient if the optimal trajectory is highly curved Bengio et al., 1994) arises because the effect of a long series of non-linear composition tends to produce gradients that can either be very small…”
BackgroundSelf-citation(2006) · Springer eBooks · Cited by 8,123
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Self-citation(2011) · Cited by 6,993
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Emergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694 · cited 3×
“Sparse coding (Olshausen and Field, 1996) is a particular kind of directed graphical model with a linear relationship between visible and latent variables (like in PCA), but in which the latent variables have a prior (e.g., Laplace density) that encourages sparsity (many zeros) in the MAP posterior.”
BackgroundInfluentialBengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 2×
“Another option, following the idea of curriculum learning (Bengio et al., 2009), is to provide guidance ourselves to learning machines (as exemplified in the toy example of Gulcehre and Bengio (2013)), by “teaching them” gradually more complex concepts to help them understand the world around us (keeping in mind that…”
BackgroundSelf-citationA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 2×
“Sampling for the negative phase is usually done by MCMC, although some learning algorithms (Collobert and Weston, 2008; Gutmann and Hyvarinen, 2010; Bordes et al., 2013) involve “negative examples” that are sampled through simpler procedures (like perturbations of the observed input).”
BackgroundNatural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841 · cited 7×
“…of research was kick-started in 2006 by a few research groups, starting with Geoff Hinton’s group, who initially focused on stacking unsupervised representation learning algorithms to obtain deeper representations (Hinton et al., 2006; Bengio et al., 2007; Ranzato et al., 2007; Lee et al., 2008).”
BackgroundMethodInfluentialSelf-citationAn analysis of single-layer networks in unsupervised feature learning
Coates, A., Ng, A. Y., & Lee, H. (2011) · International Conference on Artificial Intelligence and Statistics · Cited by 4,675
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Self-citationBayesian Learning for Neural Networks
Neal, R. M. (1996) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-0745-0 · Cited by 4,276
Multi-column deep neural networks for image classification
Ciresan, D., Meier, U., & Schmidhuber, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248110 · Cited by 3,720
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. U., & Hyvärinen, A. (2010) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 2,992
Salakhutdinov, R., & Hinton, G. E. (2009) · International Conference on Artificial Intelligence and Statistics · Cited by 2,605
Learning Fast Approximations of Sparse Coding
Gregor, K., & LeCun, Y. (2010) · International Conference on Machine Learning · Cited by 2,401
Efficient sparse coding algorithms
Lee, H., Battle, A., Raina, R., & Ng, A. Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0105 · Cited by 2,394
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259 · cited 2×
“A more generic way to avoid gradient vanishing is to incorporate a training penalty that encourages the propagated gradient vectors to maintain their magnitude (Pascanu and Bengio, 2012).”
BackgroundSelf-citationLarge Margin Methods for Structured and Interdependent Output Variables
Tsochantaridis, I., Joachims, T., Hofmann, T., & Altün, Y. (2005) · Journal of Machine Learning Research · doi:10.5555/1046920.1088722 · Cited by 2,225
What is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · cited 2×
“Before covering the unsupervised representation learning algorithms, we quickly review the basic principles behind supervised representation learning algorithms such as the good old multi-layer neural networks.”
BackgroundMethodDeep Neural Networks for Acoustic Modeling in Speech Recognition
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012) · IEEE Signal Processing Magazine · Cited by 2,115
Online dictionary learning for sparse coding
Mairal, J., Bach, F., Ponce, J., & Sapiro, G. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553463 · Cited by 2,114
Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach
Glorot, X., Bordes, A., & Bengio, Y. (2011) · HAL · Cited by 2,020
Self-citationA Tutorial on Energy-Based Learning
LeCun, Y., Chopra, S., Hadsell, R. T., Ranzato, A., & Huang, F. J. (2006) · Cited by 1,931
Restricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Bayesian Learning via Stochastic Gradient Langevin Dynamics
Welling, M., & Teh, Y. W. (2012) · Cited by 1,860
Deep Learning of Representations for Unsupervised and Transfer Learning.
Bengio, Y. (2011) · Cited by 1,702
Self-citationBuilding high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695
Bayesian Reasoning and Machine Learning
Barber, D. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9780511804779 · Cited by 1,631
Contractive Auto-Encoders: Explicit Invariance During Feature Extraction
Rifai, S., Vincent, P., Muller, X., Glorot, X., & Bengio, Y. (2011) · International Conference on Machine Learning · Cited by 1,603
Self-citationSelf-taught learning: transfer learning from unlabeled data
Raina, R., Battle, A., Lee, H., Packer, B., & Ng, A. Y. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273592 · Cited by 1,510
Hinton, G. E., Krizhevsky, A., & Wang, S. D. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-21735-7_6 · Cited by 1,360 · cited 2×
“…discrete factors (e.g., detecting a shape) from associated continuous-valued factors (e.g., pose parameters), as in transforming auto-encoders (Hinton et al., 2011), spike-and-slab RBMs with pooled slab variables (Courville et al., 2011) and other pooling-based models that learn a feature…”
BackgroundSlow Feature Analysis: Unsupervised Learning of Invariances
Wiskott, L., & Sejnowski, T. J. (2002) · Neural Computation · doi:10.1162/089976602317318938 · Cited by 1,350
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (1998) · neural information processing systems · Cited by 1,333
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Niu, F., Recht, B., Re, C., & Wright, S. J. (2011) · arXiv · doi:10.48550/arxiv.1106.5730 · Cited by 1,211 · cited 1×
“would be too inefficient because most nodes would spend their time waiting for the sharing to be completed and would be waiting for the slowest of the nodes. This idea has been analyzed theoretically (Recht et al., 2011) and successfully engineered on a grand scale recently at Google (Le et al., 2012; Dean et al., 2012)…”
BackgroundEfficient Learning of Sparse Representations with an Energy-Based Model
Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0147 · Cited by 1,106
The "Wake-Sleep" Algorithm for Unsupervised Neural Networks
Hinton, G. E., Dayan, P., Frey, B. J., & Neal, R. M. (1995) · Science · doi:10.1126/science.7761831 · Cited by 1,070
Theano: A CPU and GPU Math Compiler in Python
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., & Bengio, Y. (2010) · Proceedings of the Python in Science Conference · doi:10.25080/majora-92bf1922-003 · Cited by 1,052
Self-citationA Connection Between Score Matching and Denoising Autoencoders
Vincent, P. (2011) · Neural Computation · doi:10.1162/neco_a_00142 · Cited by 1,011 · cited 2×
“They only differ in the type of prior put on h, and the corresponding inference procedures to recover h (its posterior P (h | x) or expected value E[h | x]) when x is observed.”
BackgroundLeCun, Y., Bottou, L., Orr, G. B., & Müller, K. R. (1998) · Lecture Notes in Computer Science · doi:10.1007/3-540-49430-8_2 · Cited by 919
Conversational speech transcription using context-dependent deep neural networks
Seide, F., Li, G., & Yu, D. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-169 · Cited by 852
(2007) · Cited by 849
Separating Style and Content with Bilinear Models
Tenenbaum, J. B., & Freeman, W. T. (2000) · Neural Computation · doi:10.1162/089976600300015349 · Cited by 840 · cited 1×
“…priors on the form of the model can be used to enhance disentangling, such as multiplicative interactions between the factors (Tenenbaum and Freeman, 2000; Desjardins et al., 2012) or orthogonality of the features derivative with respect to the input (Rifai et al., 2011b, 2012a; Sohn et…”
MethodA Scalable Hierarchical Distributed Language Model
Mnih, A., & Hinton, G. E. (2008) · Cited by 838
(2005) · Cited by 823
Building high-level features using large scale unsupervised learning
Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G. S., Dean, J., Le, Q. V., & Ng, A. Y. (2012) · International Conference on Machine Learning · Cited by 823
Recent advances in deep learning for speech research at Microsoft
Deng, L., Li, J., Huang, J.-T., Yao, K., Yu, D., Seide, F., Seltzer, M., Zweig, G., He, X., Williams, J., et al. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639345 · Cited by 801
(2005) · doi:10.5555/1046920.1088696 · Cited by 756
Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
Zeiler, M. D., & Fergus, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3557 · Cited by 755
Sparse Feature Learning for Deep Belief Networks
Ranzato, M., Boureau, Y.-L., & LeCun, Y. (2007) · neural information processing systems · Cited by 744
Large-scale deep unsupervised learning using graphics processors
Raina, R., Madhavan, A., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553486 · Cited by 742 · cited 1×
“They have also taken advantage of GPU technology to speed-up computation by one or two orders of magnitude (Raina et al., 2009; Bergstra et al., 2010, 2011; Krizhevsky et al., 2012).”
BackgroundDeep learning via Hessian-free optimization
Martens, J. (2010) · International Conference on Machine Learning · Cited by 732
Modeling Human Motion Using Binary Latent Variables
Taylor, G. W., Hinton, G. E., & Roweis, S. T. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0173 · Cited by 689
Bordes, A., Glorot, X., Weston, J., & Bengio, Y. (2014) · Machine Learning · doi:10.1007/s10994-013-5363-6 · Cited by 675
Self-citationClassification using discriminative restricted Boltzmann machines
Larochelle, H., & Bengio, Y. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390224 · Cited by 655
Self-citationSeide, F., Li, G., Chen, X., & Yu, D. (2011) · 2011 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2011.6163899 · Cited by 632
The Neural Autoregressive Distribution Estimator
Larochelle, H., & Murray, I. (2011) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 626
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives
Bengio, Y., Courville, A. C., & Vincent, P. (2012) · Cited by 581
Self-citationHyvärinen, A., & Hoyer, P. (2000) · Neural Computation · doi:10.1162/089976600300015312 · Cited by 533
Quantum Annealing of a Disordered Magnet
Brooke, J., Bitko, D., F., T., Rosenbaum, & Aeppli, G. (1999) · Science · doi:10.1126/science.284.5415.779 · Cited by 518
(2011) · Cited by 514
Efficient Learning of Deep Boltzmann Machines
Salakhutdinov, R., & Larochelle, H. (2010) · International Conference on Artificial Intelligence and Statistics · Cited by 501
Self-organizing neural network that discovers surfaces in random-dot stereograms
Becker, S., & Hinton, G. E. (1992) · Nature · doi:10.1038/355161a0 · Cited by 495
Advances in optimizing recurrent networks
Bengio, Y., Boulanger-Lewandowski, N., & Pascanu, R. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639349 · Cited by 469
Self-citationMeasuring Invariances in Deep Networks
Goodfellow, I., Lee, H., Le, Q. V., Saxe, A., & Ng, A. Y. (2009) · neural information processing systems · Cited by 461
Boulanger-Lewandowski, N., Bengio, Y., & Vincent, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6392 · Cited by 459
Self-citationJaeger, H. (2007) · Scholarpedia · doi:10.4249/scholarpedia.2330 · Cited by 459
Graphical models for machine learning and digital communication
(1999) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(99)90253-5 · Cited by 454
Structured Variable Selection with Sparsity-Inducing Norms
Jenatton, R., Audibert, J.-Y., & Bach, F. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0904.3523 · Cited by 431
Higher Order Contractive Auto-Encoder
Rifai, S., Mesnil, G., Vincent, P., Muller, X., Bengio, Y., Dauphin, Y., & Glorot, X. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-23783-6_41 · Cited by 397
Self-citationTraining recurrent neural networks
Hinton, G. E., & Sutskever, I. (2013) · Cited by 383
Deep learning via semi-supervised embedding
Weston, J., Ratle, F., & Collobert, R. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390303 · Cited by 380
Binary coding of speech spectrograms using a deep auto-encoder
Deng, L., Seltzer, M. L., Yu, D., Acero, A., Mohamed, A.-R., & Hinton, G. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-487 · Cited by 375
Factored conditional restricted Boltzmann Machines for modeling motion style
Taylor, G. W., & Hinton, G. E. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553505 · Cited by 358
Revisiting Natural Gradient for Deep NetworksPreprint
Pascanu, R., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3584 · Cited by 355
Self-citationHinton, G. E. (1999) · 9th International Conference on Artificial Neural Networks: ICANN '99 · doi:10.1049/cp:19991075 · Cited by 341
Generalized Denoising Auto-Encoders as Generative Models
Bengio, Y., Yao, L., Alain, G., & Vincent, P. (2013) · arXiv · doi:10.48550/arxiv.1305.6663 · Cited by 324
Self-citationStructured Sparsity through Convex Optimization
Bach, F., Jenatton, R., Mairal, J., & Obozinski, G. (2012) · Statistical Science · doi:10.1214/12-sts394 · Cited by 315
Learning algorithms for the classification restricted Boltzmann machine
Larochelle, H., MandelMichael, PascanuRazvan, & BengioYoshua (2012) · Journal of Machine Learning Research · doi:10.5555/2188385.2188407 · Cited by 300
The Manifold Tangent Classifier
Rifai, S., Dauphin, Y., Vincent, P., Bengio, Y., & Muller, X. (2011) · neural information processing systems · Cited by 270
Self-citationPhone Recognition with the Mean-Covariance Restricted Boltzmann Machine
Dahl, G. E., Ranzato, M., Mohamed, A., & Hinton, G. E. (2010) · Cited by 266
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., & Huang, F. J. (2007) · Predicting Structured Data · doi:10.7551/mitpress/7443.003.0014 · Cited by 256
Unsupervised and Transfer Learning Challenge: a Deep Learning Approach
Mesnil, G., Dauphin, Y., Glorot, X., Rifai, S., Bengio, Y., Goodfellow, I., Lavoie, E., Muller, X., Desjardins, G., Warde-Farley, D., et al. (2011) · International Conference on Machine Learning · Cited by 254
Self-citationBetter Mixing via Deep Representations
Bengio, Y., Mesnil, G., Dauphin, Y., & Rifai, S. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4404 · Cited by 237
Self-citationIBA, Y. (2001) · International Journal of Modern Physics C · doi:10.1142/s0129183101001912 · Cited by 233
Deep Learning Made Easier by Linear Transformations in Perceptrons
Raiko, T., Valpola, H., & LeCun, Y. (2012) · International Conference on Artificial Intelligence and Statistics · Cited by 229
Theano: Deep Learning on GPUs with Python
Bergstra, J., Bastien, F., Breuleux, O., Lamblin, P., Pascanu, R., Delalleau, O., Desjardins, G., Warde-Farley, D., Goodfellow, I., Bergeron, A., et al. (2012) · Cited by 223
Self-citationDisentangling Factors of Variation for Facial Expression Recognition
Rifai, S., Bengio, Y., Courville, A., Vincent, P., & Mirza, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_58 · Cited by 215 · cited 3×
“This can be used to endow regularized auto-encoders with a probabilistic interpretation and to sample from the implicitly learned density models (Rifai et al., 2012b; Bengio et al., 2012; Alain and Bengio, 2013) through some variant of Langevin or Metropolis-Hastings Monte-Carlo Markov chains…”
BackgroundMethodSelf-citationLearning image representations from the pixel level via hierarchical sparse coding
Yu, K., Lin, Y., & Lafferty, J. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995732 · Cited by 212
Exploiting Tractable Substructures in Intractable Networks
Saul, L. K., & Jordan, M. I. (1995) · Cited by 203
Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition
Kavukcuoglu, K., Ranzato, M., & LeCun, Y. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1010.3467 · Cited by 183
Tempered Markov Chain Monte Carlo for training of Restricted Boltzmann Machines
Desjardins, G., Courville, A. C., Bengio, Y., Vincent, P., & Delalleau, O. (2010) · International Conference on Artificial Intelligence and Statistics · Cited by 153
Self-citationSample Complexity of Testing the Manifold Hypothesis
Narayanan, H., & Mitter, S. K. (2010) · Cited by 148
Conditional Restricted Boltzmann Machines for Structured Output PredictionPreprint
Mnih, V., Larochelle, H., & Hinton, G. E. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1202.3748 · Cited by 147
Knowledge Matters: Importance of Prior Information for OptimizationPreprint
Gülçehre, Ç., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.4083 · Cited by 142
Self-citationEstimating or Propagating Gradients Through Stochastic NeuronsPreprint
Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1305.2982 · Cited by 128
Self-citationStoyanov, V., Ropson, A., & Eisner, J. M. (2011) · International Conference on Artificial Intelligence and Statistics · Cited by 124
Emergence of invariant-feature detectors in the adaptive-subspace self-organizing map
Kohonen, T. (1996) · Biological Cybernetics · doi:10.1007/s004220050295 · Cited by 116
Taking on the curse of dimensionality in joint distributions using neural networks
Bengio, S., & Bengio, Y. (2000) · IEEE Transactions on Neural Networks · doi:10.1109/72.846725 · Cited by 115
Self-citationShift-Invariance Sparse Coding for Audio ClassificationPreprint
Grosse, R., Raina, R., Kwong, H., & Ng, A. Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.5241 · Cited by 114
Bagnell, J. A., & Bradley, D. M. (2008) · neural information processing systems · Cited by 109
On Autoencoders and Score Matching for Energy Based Models
Swersky, K., Ranzato, M., Buchman, D., de Freitas, N., & Marlin, B. M. (2011) · International Conference on Machine Learning · Cited by 109
Bengio, Y. (2008) · Scholarpedia · doi:10.4249/scholarpedia.3881 · Cited by 105
Self-citationLearning Continuous Attractors in Recurrent Networks
Seung, H. S. (1997) · neural information processing systems · Cited by 95
Learning and Selecting Features Jointly with Point-wise Gated Boltzmann Machines
Sohn, K., Zhou, G., Lee, C., & Lee, H. (2013) · Cited by 93
Decision trees do not generalize to new variations
Bengio, Y., Delalleau, O., & Simard, C. (2010) · Computational Intelligence · doi:10.1111/j.1467-8640.2010.00366.x · Cited by 89 · cited 3×
“Indeed, we want to keep the advantages of distributed representations and avoid the limited local generalization suffered by decision trees (Bengio et al., 2010).”
BackgroundLearning Deep Boltzmann Machines using Adaptive MCMC
Salakhutdinov, R. (2010) · International Conference on Machine Learning · Cited by 88
Parallel tempering is efficient for learning restricted Boltzmann machines
Cho, K., Raiko, T., & Ilin, A. (2010) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2010.5596837 · Cited by 82
Quickly Generating Representative Samples from an RBM-Derived Process
Breuleux, O., Bengio, Y., & Vincent, P. (2011) · Neural Computation · doi:10.1162/neco_a_00158 · Cited by 82
Self-citationLearning in Markov Random Fields using Tempered Transitions
Salakhutdinov, R. (2009) · Cited by 81
Regularized estimation of image statistics by Score Matching
Kingma, D. P., & LeCun, Y. (2010) · neural information processing systems · Cited by 79
A Generative Process for Sampling Contractive Auto-EncodersPreprint
Rifai, S., Bengio, Y., Dauphin, Y., & Vincent, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6434 · Cited by 75
Self-citationSlow, Decorrelated Features for Pretraining Complex Cell-like Networks
Bengio, Y., & Bergstra, J. (2009) · neural information processing systems · Cited by 75
Self-citationUnsupervised Models of Images by Spike-and-Slab RBMs
Bengio, Y., Courville, A. C., & Bergstra, J. (2011) · International Conference on Machine Learning · Cited by 67
Self-citationDeep Boltzmann Machines and the Centering Trick
Montavon, G., & Müller, K.-R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_33 · Cited by 67
Disentangling Factors of Variation via Generative Entangling
Desjardins, G., Courville, A. C., & Bengio, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1210.5474 · Cited by 64
Self-citationEmergence of Object-Selective Features in Unsupervised Feature Learning
Coates, A., Karpathy, A., & Ng, A. Y. (2012) · neural information processing systems · Cited by 60
Non-Local Manifold Parzen Windows
Bengio, Y., Larochelle, H., & Vincent, P. (2005) · neural information processing systems · Cited by 58
Self-citationSpike-and-Slab Sparse Coding for Unsupervised Feature DiscoveryPreprint
Goodfellow, I., Courville, A. C., & Bengio, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1201.3382 · Cited by 54
Self-citationMulti-column Deep Neural Networks for Image Classification
Cireşan, D., Meier, U., & Schmidhuber, J. (2012) · arXiv · doi:10.48550/arxiv.1202.2745 · Cited by 53
No More Pesky Learning RatesPreprint
Schaul, T., Zhang, S., & LeCun, Y. (2012) · arXiv (Cornell University) · Cited by 51
Communication/Computation Tradeoffs in Consensus-Based Distributed OptimizationPreprint
Tsianos, K. I., Lawlor, S., & Rabbat, M. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1209.1076 · Cited by 50
Large-Scale Feature Learning With Spike-and-Slab Sparse Coding
Goodfellow, I., Courville, A. C., & Bengio, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6407 · Cited by 49
Self-citationSequential Labeling Using Deep-Structured Conditional Random Fields
Yu, D., Wang, S., & Deng, L. (2010) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2010.2075990 · Cited by 46
Centering Neural Network Gradient Factors
Schraudolph, N. N. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35289-8_14 · Cited by 45
Nonlocal Estimation of Manifold Structure
Bengio, Y., Monperrus, M., & Larochelle, H. (2006) · Neural Computation · doi:10.1162/neco.2006.18.10.2509 · Cited by 45
Self-citationSCALING LARGE LEARNING PROBLEMS WITH HARD PARALLEL MIXTURES
Collobert, R., Bengio, Y., & Bengio, S. (2003) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001403002411 · Cited by 45
Self-citationCentering Neural Network Gradient Factors
Schraudolph, N. N. (1998) · Lecture notes in computer science · doi:10.1007/3-540-49430-8_11 · Cited by 45
Language recognition using deep-structured conditional random fields
Yu, D., Wang, S., Karam, Z. N., & Deng, L. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5495072 · Cited by 38
Exploring Compositional High Order Pattern Potentials for Structured Output Learning
Li, Y., Tarlow, D., & Zemel, R. S. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.14 · Cited by 35
Kindermann, R. P., & Snell, J. L. (1980) · Contemporary mathematics - American Mathematical Society · doi:10.1090/conm/001/01 · Cited by 31
Texture Modeling with Convolutional Spike-and-Slab RBMs and Deep ExtensionsPreprint
Luo, H., Carrier, P. L., Courville, A. C., & Bengio, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.5687 · Cited by 29
Self-citationLEARNING ITERATIVE IMAGE RECONSTRUCTION IN THE NEURAL ABSTRACTION PYRAMID
Behnke, S. (2001) · International Journal of Computational Intelligence and Applications · doi:10.1142/s1469026801000342 · Cited by 27
Learning Feature Hierarchies with Centered Deep Boltzmann MachinesPreprint
Montavon, G., & Müller, K. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1203.3783 · Cited by 26
Joint Training of Deep Boltzmann MachinesPreprint
Goodfellow, I., Courville, A. C., & Bengio, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1212.2686 · Cited by 25
Self-citationJoint Training Deep Boltzmann Machines for ClassificationPreprint
Goodfellow, I., Courville, A. C., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3568 · Cited by 24
Self-citationParallelizable Sampling of Markov Random Fields
Martens, J., & Sutskever, I. (2010) · International Conference on Artificial Intelligence and Statistics · Cited by 24
Evolving Culture vs Local MinimaPreprint
Bengio, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1203.2990 · Cited by 22
Self-citationImplicit Density Estimation by Local Moment Matching to Sample from Auto-EncodersPreprint
Bengio, Y., Alain, G., & Rifai, S. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.0057 · Cited by 15
Self-citationRéseaux de neurones à relaxation entraînés par critère d'autoencodeur débruitant
Savard, F. (2011) · doi:10.71781/10007 · Cited by 7
Cited by
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503
A survey of the recent architectures of deep convolutional neural networks
Khan, A., Sohail, A., Zahoora, U., & Qureshi, A. S. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09825-6 · Cited by 3,137
Deep learning applications and challenges in big data analytics
Najafabadi, M. M., Villanustre, F., Khoshgoftaar, T. M., Seliya, N., Wald, R., & Muharemagic, E. (2015) · Journal of Big Data · doi:10.1186/s40537-014-0007-7 · Cited by 2,594
Deep Metric Learning Using Triplet Network
Hoffer, E., & Ailon, N. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-24261-3_7 · Cited by 2,134
Deep learning for sensor-based activity recognition: A survey
Wang, J., Chen, Y., Hao, S., Peng, X., & Hu, L. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.02.010 · Cited by 2,014
Metadata
- Authors
- Bengio, Y.
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer Berlin Heidelberg
- Published
- 2013
- Pages
- 1-37
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W1613249581
- Cited by
- 513
- References
- 157
- FWCI
- 20.38
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.