Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations
The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
There has been much interest in unsupervised learning of hierarchical generative models such as deep belief networks. Scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep belief network, a hierarchical generative model which scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max-pooling, a novel technique which shrinks the representations of higher layers in a probabilistically sound way. Our experiments show that the algorithm learns useful high-level visual features, such as object parts, from unlabeled images of objects and natural scenes. We demonstrate excellent performance on several visual recognition tasks and show that our model can perform hierarchical (bottom-up and top-down) inference over full-sized images.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Deep learning techniques have outperformed many conventional methods in computer vision [13] and audio classification [14].”
From Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition (2016) · cites this paper as [13]
In the citing paper
“Convolutional RBMs are an extension of regular RBMs [24].”
From Multimodal emotion recognition using deep learning architectures (2016) · cites this paper as [24]
Citations
Cites
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 9×
“DBNs have been successfully used to learn high-level structure in a wide variety of domains, including handwritten digits (Hinton et al., 2006) and human motion capture data (Taylor et al.”
BackgroundMethodInfluentialBackpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916
Emergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694 · cited 4×
“…while the second layer consisted of 100 bases, each one 10x10 as well.5 As shown in Figure 2 (top), the learned .rst layer bases are oriented, localized edge .lters; this result is consistent with much prior work (Olshausen & Field, 1996; Bell & Sejnowski, 1997; Ranzato et al., 2006).”
BackgroundResultInfluentialTraining Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 2×
“Instead, one typically uses the contrastive divergence approximation (Hinton, 2002), which has been shown to work well in practice.”
BackgroundGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Neural Information Processing Systems
(2018) · The Deep Learning Revolution · doi:10.7551/mitpress/11474.003.0014 · Cited by 3,730
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
The “independent components” of natural scenes are edge filters
Bell, A. J., & Sejnowski, T. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00121-1 · Cited by 2,201
Self-taught learning: transfer learning from unlabeled data
Raina, R., Battle, A., Lee, H., Packer, B., & Ng, A. Y. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273592 · Cited by 1,510
Self-citationHierarchical Bayesian inference in the visual cortex
Lee, T. S., & Mumford, D. (2003) · Journal of the Optical Society of America A · doi:10.1364/josaa.20.001434 · Cited by 1,473 · cited 2×
“Lee and Mumford (2003) proposed that the human visual cortex can conceptually be modeled as performing \hierarchical Bayesian inference." For example, if you observe a face image with its left half in dark illumina-Figure 6. Hierarchical probabilistic inference.”
BackgroundSVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition
Zhang, H., Berg, A. C., Maire, M., & Malik, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.301 · Cited by 1,114
Efficient Learning of Sparse Representations with an Energy-Based Model
Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0147 · Cited by 1,106
Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition
Ranzato, M., Huang, F. J., Boureau, Y.-L., & LeCun, Y. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383157 · Cited by 1,086
Editorial: On Machine Learning
Langley, P. (1986) · Machine Learning · doi:10.1023/a:1022687019898 · Cited by 951
(2007) · Cited by 849
Self-citationShape Matching and Object Recognition Using Low Distortion Correspondences
Berg, A. C., Berg, T. L., & Malik, J. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.320 · Cited by 845
Large-scale deep unsupervised learning using graphics processors
Raina, R., Madhavan, A., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553486 · Cited by 742 · cited 2×
“(However, see also (Raina et al., 2009).)”
MethodSelf-citationModeling Human Motion Using Binary Latent Variables
Taylor, G. W., Hinton, G. E., & Roweis, S. T. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0173 · Cited by 689
Learning The Discriminative Power-Invariance Trade-Off
Varma, M., & Ray, D. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408875 · Cited by 520
Multiclass Object Recognition with Sparse, Localized Features
Mutch, J., & Lowe, D. G. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR'06) · doi:10.1109/cvpr.2006.200 · Cited by 453
Deep learning via semi-supervised embedding
Weston, J., Ratle, F., & Collobert, R. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390303 · Cited by 380
Representation of Angles Embedded within Contour Stimuli in Area V2 of Macaque Monkeys
Ito, M., & Komatsu, H. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.4364-03.2004 · Cited by 273
Shift-invariant sparse coding for audio classification
Grosse, R., Raina, R., Kwong, H., & Ng, A. Y. (2007) · Uncertainty in Artificial Intelligence · Cited by 220
Self-citationDeep Learning with Kernel Regularization for Visual Recognition
Yu, K., Xu, W. H., & Gong, Y. (2008) · Cited by 84
Shirai, Y. (1984) · Journal of the Robotics Society of Japan · doi:10.7210/jrsj.2.593 · Cited by 11
Research works printed but not linked: 2
- Desjardins, G., & Bengio, Y. (2008). Empirical evaluation of convolutional RBMs for vision (Technical Report).
- 10.5555/1032643.1033069 · doi:10.5555/1032643.1033069
Cited by
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
“Deep belief networks (Lee et al., 2009) have also been shown to work well in learning hierarchical representations.”
BackgroundMethod
Metadata
- Authors
- Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 26th Annual International Conference on Machine Learning
- Publisher
- ACM
- Published
- 2009-06-14
- Pages
- 609-616
- OpenAlex
- W2130325614
- Cited by
- 2,412
- References
- 27
- Citations by year
- 2026: 7 · 2025: 27 · 2024: 57 · 2023: 77 · 2022: 117 · 2021: 153 · 2020: 217 · 2019: 225 · 2018: 246 · 2017: 247 · 2016: 270 · 2015: 236 · 2014: 172 · 2013: 130 · 2012: 92
- FWCI
- 62.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Machine Learning (publishedVersion)
- Funders
- Defense Advanced Research Projects Agency (FA8750-05-2-0249)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- convolutional deep belief network, unsupervised learning, hierarchical generative models, translation invariance, top-down inference, hierarchical representation, visual recognition, natural scene recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.