Building high-level features using large scale unsupervised learning
The DOI matches its Crossref record; 48 of the 50 references it cites were found in a registry or an index.
Abstract
We consider the problem of building high-level, class-specific feature detectors from only unlabeled data. For example, is it possible to learn a face detector using only unlabeled images? To answer this, we train a deep sparse autoencoder on a large dataset of images (the model has 1 billion connections, the dataset has 10 million 200×200 pixel images downloaded from the Internet). We train this network using model parallelism and asynchronous SGD on a cluster with 1,000 machines (16,000 cores) for three days. Contrary to what appears to be a widely-held intuition, our experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not. Control experiments show that this feature detector is robust not only to translation but also to scaling and out-of-plane rotation. We also find that the same network is sensitive to other high-level concepts such as cat faces and human bodies. Starting from these learned features, we trained our network to recognize 22,000 object categories from ImageNet and achieve a leap of 70% relative improvement over the previous state-of-the-art.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“[41] famously found evidence of a ‘cat neuron’, a phenomenon that has been remarkably replicable across newer models and analyses [14,42,43]”
From Symbols and grounding in large language models (2023) · cites this paper as [41]
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 4×
“The test set consists of 37,000 images sampled from two datasets: Labeled Faces In the Wild dataset (Huang et al., 2007) and ImageNet dataset (Deng et al., 2009).”
BackgroundMethodInfluentialGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 3×
“This is a stark differ-ence between our approach and previous work (LeCun et al., 1998; Jarrett et al., 2009; Lee et al., 2009).”
BackgroundMethodResultLearning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 3×
“Successful feature learning algorithms and their applications can be found in recent literature using a variety of approaches such as RBMs (Hinton et al., 2006), autoencoders (Hinton & Salakhutdinov, 2006; Bengio et al., 2007), sparse coding (Lee et al., 2007) and K-means (Coates et al., 2011).”
BackgroundMethodReceptive fields, binocular interaction and functional architecture in the cat’s visual cortex
Hubel, D. H., & Wiesel, T. N. (1962) · The Journal of Physiology · doi:10.1113/jphysiol.1962.sp006837 · Cited by 13,680 · cited 1×
“In this section, we describe the algorithm that we use to learn features from the unlabeled training set.”
MethodEmergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694 · cited 5×
“One shortcoming of early approaches such as sparse coding (Olshausen & Field, 1996) is that their architectures are shallow and typically capture low-level concepts (e.g., edge “Gabor” filters) and simple invariances.”
BackgroundMethodInfluentialGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments
Huang, G. B., Mattar, M., Berg, T. L., & Miller, E. G. (2008) · HAL · Cited by 4,822
An analysis of single-layer networks in unsupervised feature learning
Coates, A., Ng, A. Y., & Lee, H. (2011) · International Conference on Artificial Intelligence and Statistics · Cited by 4,675
Receptive fields of single neurones in the cat's striate cortex
Hubel, D. H., & Wiesel, T. N. (1959) · The Journal of Physiology · doi:10.1113/jphysiol.1959.sp006308 · Cited by 4,653 · cited 1×
“According to their study, sparse coding can be trained on unlabeled natural images to yield receptive fields akin to V1 simple cells (Hubel & Wiesel, 1959).”
BackgroundLarge Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Self-citationTheoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems
Dayan, P., & Abbott, L. F. (2001) · Cited by 4,099
Hierarchical models of object recognition in cortex
Riesenhuber, M., & Poggio, T. (1999) · Nature Neuroscience · doi:10.1038/14819 · Cited by 3,309 · cited 2×
“Our style of stacking a series of uniform modules, switching between selectivity and tolerance layers, is reminiscent of Neocognition and HMAX (Fukushima & Miyake, 1982; LeCun et al., 1998; Riesenhuber & Poggio, 1999).”
BackgroundImageNet: A large-scale hierarchical image database
Deng, J. H., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412 · cited 6×
“In this section, we describe the algorithm that we use to learn features from the unlabeled training set.”
BackgroundMethodResultInfluentialEfficient sparse coding algorithms
Lee, H., Battle, A., Raina, R., & Ng, A. Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0105 · Cited by 2,394 · cited 2×
“Our work is inspired by recent successful algorithms in unsupervised feature learning and deep learning [20, 9, 1, 24, 18].”
MethodWhat is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · cited 8×
“Next, to achieve invariance to local deformations, we employ local L2 pooling (Hyv¨arinen et al., 2009; Gregor & LeCun, 2010; Le et al., 2010) and lo-cal contrast normalization (Jarrett et al., 2009).”
BackgroundMethodResultInfluentialInvariant visual representation by single neurons in the human brain
Quiroga, R. Q., Reddy, L., Kreiman, G., Koch, C., & Fried, I. (2005) · Nature · doi:10.1038/nature03687 · Cited by 1,988 · cited 1×
“…in the brain is an area of active investigation, but current experimental evidence suggests the possibility that some neurons in the temporal cortex are highly selective for object categories such as faces or hands (Desimone et al., 1984), and perhaps even specific people (Quiroga et al., 2005).”
BackgroundHow does the brain solve visual object recognition?
DiCarlo, J. J., Zoccolan, D., & Rust, N. C. (2012) · Neuron · doi:10.1016/j.neuron.2012.01.010 · Cited by 1,790 · cited 2×
“(1) Here, λ is a tradeoff parameter between sparsity and reconstruction; m, k are the number of examples and pooling units in a layer respectively; Hj is the vector of weights of the j-th pooling unit.”
BackgroundSelf-taught learning: transfer learning from unlabeled data
Raina, R., Battle, A., Lee, H., Packer, B., & Ng, A. Y. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273592 · Cited by 1,510 · cited 1×
“Using unlabeled data in the wild to learn features is the key idea behind the self-taught learning framework (Raina et al., 2007).”
BackgroundStimulus-selective properties of inferior temporal neurons in the macaque
Desimone, R., Albright, T., Gross, C., & Bruce, C. (1984) · The Journal of Neuroscience · doi:10.1523/jneurosci.04-08-02051.1984 · Cited by 1,423 · cited 2×
“” The extent of class-specificity of neurons in the brain is an area of active investigation, but current experimental evidence suggests the possibility that some neurons in the temporal cortex are highly selective for object categories such as faces or hands [1], and perhaps even specific people [2].”
BackgroundUnsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition
Ranzato, M., Huang, F. J., Boureau, Y.-L., & LeCun, Y. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383157 · Cited by 1,086 · cited 2×
“Our work is inspired by recent successful algorithms in unsupervised feature learning and deep learning (Hinton et al., 2006; Bengio et al., 2007; Ranzato et al., 2007; Lee et al., 2007).”
MethodInfluentialDeep, Big, Simple Neural Nets for Handwritten Digit Recognition
Cireşan, D. C., Meier, U., Gambardella, L. M., & Schmidhuber, J. (2010) · Neural Computation · doi:10.1162/neco_a_00052 · Cited by 1,078 · cited 3×
“1 Our input data are 200x200 images, much larger than typical 32x32 images used in deep learning and unsupervised feature learning (Krizhevsky, 2009; Ciresan et al., 2010; Le et al., 2010; Coates et al., 2011).”
BackgroundMethodOn optimization methods for deep learning
Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., Le, Q. V., & Ng, A. Y. (2011) · International Conference on Machine Learning · Cited by 1,076
Self-citation(2007) · Cited by 937
(2007) · Cited by 849
Building high-level features using large scale unsupervised learning
Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G. S., Dean, J., Le, Q. V., & Ng, A. Y. (2012) · International Conference on Machine Learning · Cited by 823
Self-citationTraffic sign recognition with multi-scale Convolutional Networks
Sermanet, P., & LeCun, Y. (2011) · The 2011 International Joint Conference on Neural Networks · doi:10.1109/ijcnn.2011.6033589 · Cited by 759
Fukushima, K., & Miyake, S. (1982) · Pattern Recognition · doi:10.1016/0031-3203(82)90024-3 · Cited by 742 · cited 1×
“This style of stacking a series of uniform structures is reminiscent of HMAX [25] and Neocognition [6] approaches.”
BackgroundLarge-scale deep unsupervised learning using graphics processors
Raina, R., Madhavan, A., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553486 · Cited by 742 · cited 4×
“1 Our input data are 200x200 images, much larger than typical 32x32 images used in deep learning and unsupervised feature learning (Krizhevsky, 2009; Ciresan et al., 2010; Le et al., 2010; Coates et al., 2011).”
BackgroundMethodInfluentialWsabie: Scaling up to large vocabulary image annotation
Weston, J., Samy Bengio, A., & Usunier, N. (2011) · Cited by 677
Why is Real-World Visual Object Recognition Hard?
Pinto, N., Cox, D. D., & DiCarlo, J. J. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0040027 · Cited by 537 · cited 2×
“This exper-iment shows that patches, being detected as faces by the OpenCV face detector, account for less than 3% of the 100,000 sampled patches”
BackgroundMethodWhat Does Classifying More Than 10,000 Image Categories Tell Us?
Deng, J., Berg, A. C., Li, K., & Fei-Fei, L. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15555-0_6 · Cited by 486 · cited 1×
“We followed the experimental protocols specified by (Deng et al., 2010; Sanchez & Perronnin, 2011), in which, the datasets are randomly split into two halves for training and validation.”
MethodPakkenberg, B. (2003) · Experimental Gerontology · doi:10.1016/s0531-5565(02)00151-1 · Cited by 485
Measuring Invariances in Deep Networks
Goodfellow, I., Lee, H., Le, Q. V., Saxe, A., & Ng, A. Y. (2009) · neural information processing systems · Cited by 461
Self-citationHyvärinen, A., Hurri, J., & Hoyer, P. O. (2009) · Computational Imaging and Vision · doi:10.1007/978-1-84882-491-1 · Cited by 398 · cited 5×
“y types of pooling such as mean pooling, max pooling [17] and square-squareroot pooling [15] but in this work we will use the last type of pooling because it allows the learning of invariant features [13, 15]. For learned invariances when the same algorithm is trained on natural images, see http://cs.stanford.edu/˘quo…”
InfluentialICA with Reconstruction Cost for Efficient Overcomplete Feature Learning
Le, Q. V., Karpenko, A., Ngiam, J., & Ng, A. Y. (2011) · neural information processing systems · Cited by 369
Self-citationTiled convolutional neural networks
Ngiam, J., Chen, Z., Chia, D., Koh, P. W., Le, Q. V., & Ng, A. Y. (2010) · neural information processing systems · Cited by 361
Self-citationSlow feature analysis yields a rich repertoire of complex cell properties
Berkes, P., & Wiskott, L. (2005) · Journal of Vision · doi:10.1167/5.6.9 · Cited by 280 · cited 2×
“The second approach is to perform numerical optimization to find the optimal stimulus (Berkes & Wiskott, 2005; Erhan et al., 2009; Le et al., 2010).”
BackgroundMethodHigh-dimensional signature compression for large-scale image classification
Sanchez, J., & Perronnin, F. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995504 · Cited by 272 · cited 2×
“We followed the experimental protocols specified by (Deng et al., 2010; Sanchez & Perronnin, 2011), in which, the datasets are randomly split into two halves for training and validation.”
BackgroundMethodSENSC: a Stable and Efficient Algorithm for Nonnegative Sparse Coding
LI, L., & ZHANG, Y.-J. (2009) · Acta Automatica Sinica · doi:10.3724/sp.j.1004.2009.01257 · Cited by 223
Cat Head Detection - How to Effectively Exploit Shape and Texture Features
Zhang, W., Sun, J. Z., & Tang, X. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88693-8_59 · Cited by 184 · cited 2×
“scribed in (Zhang et al., 2008).”
BackgroundBuilding high-level features using large scale unsupervised learning
Le, Q. V., Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G. S., Dean, J., & Ng, A. Y. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1112.6209 · Cited by 163
Self-citationNonlinear image representation using divisive normalization
Lyu, S., & Simoncelli, E. P. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587821 · Cited by 146 · cited 2×
“This exper-iment shows that patches, being detected as faces by the OpenCV face detector, account for less than 3% of the 100,000 sampled patches”
BackgroundMethodA new benchmark for stereo-based pedestrian detection
Keller, C. G., Enzweiler, M., & Gavrila, D. M. (2011) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2011.5940480 · Cited by 95 · cited 1×
“Negative and positive examples in our human body dataset are subsampled at random from a benchmark dataset (Keller et al., 2009).”
BackgroundEmergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields
Gregor, K., & LeCun, Y. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1006.0448 · Cited by 56 · cited 2×
“Next, to achieve invariance to local deformations, we employ local L2 pooling (Hyvärinen et al., 2009; Gregor & LeCun, 2010; Le et al., 2010) and local contrast normalization (Jarrett et al., 2009).”
BackgroundMethodLayer-wise analysis of deep networks with Gaussian kernels
Montavon, G., Müller, K., & Braun, M. L. (2010) · neural information processing systems · Cited by 17
Research works printed but not linked: 2
- erhan, 2009, Visualizing Higher-layer Features of Deep Networks
- bengio, 2007, Greedy layerwise training of deep networks, NIPS
Cited by
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
“ffic sign competition (Sec. 5.19). See further improvements in Sec. 5.22. Also in 2012, the biggest NN so far (109 free parameters) was trained in unsupervised mode (Sec. 5.7, 5.15) on unlabeled data (Le et al., 2012), then applied to ImageNet. The codes across its top layer were used to train a simple supervised class…”
MethodInfluentialReview of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Biamonte, J., Wittek, P., Pancotti, N., Rebentrost, P., Wiebe, N., & Lloyd, S. (2017) · Nature · doi:10.1038/nature23474 · Cited by 4,779
“In the past decade, particularly in the past ve years, the combination of powerful computers and special-purpose information processors capable of implementing deep networks with billions of weights [3], together with their application to very large data sets, has revealed that such deep learning networks are capable…”
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
“Explanation of neural network behavior on the level of single neurons is done in [27] and [28].”
BackgroundUnsupervised Feature Learning via Non-parametric Instance Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664
Metadata
- Authors
- Le, Q. V.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Le, Quoc V.
- ORCID
- Le, Quoc V.
- Venue
- 2013 IEEE International Conference on Acoustics, Speech and Signal Processing
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 8595-8598
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2120480077
- Cited by
- 1,695
- References
- 50
- Citations by year
- 2026: 6 · 2025: 21 · 2024: 50 · 2023: 82 · 2022: 93 · 2021: 114 · 2020: 140 · 2019: 148 · 2018: 184 · 2017: 208 · 2016: 196 · 2015: 189 · 2014: 159 · 2013: 88
- FWCI
- 134.47
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- unsupervised feature learning, deep sparse autoencoder, face detection, asynchronous SGD, model parallelism, object recognition, ImageNet, rotation and translation invariance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.