Learning to represent visual input
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
One of the central problems in computational neuroscience is to understand how the object-recognition pathway of the cortex learns a deep hierarchy of nonlinear feature detectors. Recent progress in machine learning shows that it is possible to learn deep hierarchies without requiring any labelled data. The feature detectors are learned one layer at a time and the goal of the learning procedure is to form a good generative model of images, not to predict the class of each image. The learning procedure only requires the pairwise correlations between the activations of neuron-like processing units in adjacent layers. The original version of the learning procedure is derived from a quadratic ‘energy’ function but it can be extended to allow third-order, multiplicative interactions in which neurons gate the pairwise interactions between other neurons. A technique for factoring the third-order interactions leads to a learning module that again has a simple learning rule based on pairwise correlations. This module looks remarkably like modules that have been proposed by both biologists trying to explain the responses of neurons and engineers trying to create systems that can recognize objects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Self-citationNeural networks and physical systems with emergent collective computational abilities.
Hopfield, J. J. (1982) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.79.8.2554 · Cited by 19,270
Factor graphs and the sum-product algorithm
Kschischang, F. R., Frey, B. J., & Loeliger, H.-A. (2001) · IEEE Transactions on Information Theory · doi:10.1109/18.910572 · Cited by 6,486 · cited 2×
“The use of factors that send different messages to different places is a form of belief propagation (Kschischang et al. 2001).”
BackgroundInfluentialFukushima, K. (1980) · Biological Cybernetics · doi:10.1007/bf00344251 · Cited by 5,759 · cited 1×
“A good way to perform this computation in a network of neuron-like processing elements is to use a hierarchy of progressively more complicated feature detectors (Selfridge 1958; Fukushima 1980; LeCun et al. 1998; Serre et al. 2007).”
MethodTraining Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 2×
“This gives a very simple learning rule called ‘contrastive divergence’ (Hinton 2002) D w ij / k v i h j l data (cid:2) k v i h j l reconstruction : ð 5 : 6 Þ This rule does not maximize the probability that the RBM would generate the training data, but it does work surprisingly well in a wide…”
BackgroundSelf-citationAn Introduction to Variational Methods for Graphical Models
Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999) · Machine Learning · doi:10.1023/a:1007665907178 · Cited by 3,791
Spatiotemporal energy models for the perception of motion
Adelson, E. H., & Bergen, J. R. (1985) · Journal of the Optical Society of America A · doi:10.1364/josaa.2.000284 · Cited by 3,442
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
Self-citationLee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412
A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
Neal, R. M., & Hinton, G. E. (1998) · Learning in Graphical Models · doi:10.1007/978-94-011-5014-9_12 · Cited by 2,187 · cited 1×
“It does, however, perform gradient ascent in a closely related quantity called the (negative) variational free energy (Zemel 1994; Neal & Hinton 1998) that differs from the log probability of v by the Kullback Liebler divergence between q(hjv, W ) and p(hjv, W ) FðvjW Þ ¼ log pðvjW Þ X h qðhjv;W Þ…”
BackgroundSelf-citationThe "Wake-Sleep" Algorithm for Unsupervised Neural Networks
Hinton, G. E., Dayan, P., Frey, B. J., & Neal, R. M. (1995) · Science · doi:10.1126/science.7761831 · Cited by 1,070
Self-citationCrick, F., & Mitchison, G. (1983) · Nature · doi:10.1038/304111a0 · Cited by 865
Cited by
Whatever next? Predictive brains, situated agents, and the future of cognitive science
Clark, A. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000477 · Cited by 5,924
Pupil fluctuations track fast switching of cortical states during quiet wakefulness
Reimer, J., Froudarakis, E., Cadwell, C. R., Yatsenko, D., Denfield, G. H., & Tolias, A. S. (2014) · Neuron · doi:10.1016/j.neuron.2014.09.033 · Cited by 717
“Interestingly, theoretical work has suggested that alternating between bottom-up and top-down processing may be an efficient learning mechanism for the cortex to build a representation of the external world (Hinton, 2010).”
Change Detection in Synthetic Aperture Radar Images Based on Deep Neural Networks
Gong, M., Zhao, J., Liu, J., Miao, Q., & Jiao, L. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2435783 · Cited by 638
Buesing, L., Bill, J., Nessler, B., & Maass, W. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002211 · Cited by 384
“We used for this purpose a standard Boltzmann machine learning rule known as contrastive divergence [41,65].”
BackgroundMethodHappily entangled: prediction, emotion, and the embodied mind
Miller, M., & Clark, A. (2018) · Synthese · doi:10.1007/s11229-017-1399-7 · Cited by 114
“The idea that the brain is (at least in part, and at least sometimes) acting as some form of prediction engine has a long history, stretching from early work on perception (Helmholz, 1860; MacKay, 1956; Neisser, 1967; Gregory, 1980) all the way to recent work in deep learning (Hinton, 2007, 2010).”
Background
Metadata
- Authors
- Hinton, G. E.
- Venue
- Philosophical Transactions of the Royal Society B: Biological Sciences
- Publisher
- The Royal Society
- Published
- 2010-01-12
- Volume
- 365
- Issue
- 1537
- Pages
- 177-184
- ISSN
- 0962-8436 (ISSN-L)
- 1471-2970
- OpenAlex
- W2119574220
- Cited by
- 91
- References
- 13
- FWCI
- 8.31
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.