The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
An Efficient Learning Procedure for Deep Boltzmann Machines
Open accessAbstract
We present a new learning algorithm for Boltzmann machines that contain many layers of hidden variables. Data-dependent statistics are estimated using a variational approximation that tends to focus on a single mode, and data-independent statistics are estimated using persistent Markov chains. The use of two quite different techniques for estimating the two types of statistic that enter into the gradient of the log likelihood makes it practical to learn Boltzmann machines with multiple hidden layers and millions of parameters. The learning can be made more efficient by using a layer-by-layer pretraining phase that initializes the weights sensibly. The pretraining also allows the variational inference to be initialized sensibly with a single bottom-up pass. We present results on the MNIST and NORB data sets showing that deep Boltzmann machines learn very good generative models of handwritten digits and 3D objects. We also show that the features discovered by deep Boltzmann machines are a very effective way to initialize the hidden layers of feedforward neural nets, which are then discriminatively fine-tuned.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Optimization by Simulated Annealing
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983) · Science · doi:10.1126/science.220.4598.671 · Cited by 45,134
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Self-citationStochastic relaxation, gibbs distributions, and the bayesian restoration of images.
Geman, S., & Geman, D. (1984) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1984.4767596 · Cited by 17,949
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Self-citationAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
(2010) · International Conference on Machine Learning · Cited by 13,995
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 12×
“Instead of viewing this learning procedure as a gross approximation to maximum likelihood learning, it can be viewed as a much better approximation to minimizing the difference of two divergences (Hinton, 2002), and so it is called contrastive divergence (CD) learning.”
BackgroundMethodInfluentialSelf-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841 · cited 1×
“…nonlinear NCA (Salakhutdinov & Hinton, 2007), linear NCA (Goldberger, Roweis, Hinton, & Salakhutdinov, 2004), a stack of greedily pretrained autoencoders (Bengio, Lamblin, Popovici, & Larochelle, 2007), and K-nearest neighbors, particularly when the number of labeled examples is only 600.”
BackgroundAn Introduction to Variational Methods for Graphical Models
Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999) · Machine Learning · doi:10.1023/a:1007665907178 · Cited by 3,791
A Practical Guide to Training Restricted Boltzmann Machines
Hinton, G. E. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_32 · Cited by 2,852 · cited 3×
“For example, it ought to be possible to learn the variance of the noise model of the visible units in a Gaussian-Bernoulli RBM, but this is typically very difficult for reasons explained in Hinton (2010).”
BackgroundMethodSelf-citationAdvances in Neural Information Processing Systems 20
Bonilla, E. V., Chai, K. M. A., & Williams, C. K. I. (2008) · neural information processing systems · Cited by 2,846
Salakhutdinov, R., & Hinton, G. E. (2009) · International Conference on Artificial Intelligence and Statistics · Cited by 2,605
Self-citationParallel & distributed processing
Slusallek, P., Shirley, P., Mark, W., Stoll, G., & Wald, I. (2005) · ACM SIGGRAPH 2005 Courses on - SIGGRAPH '05 · doi:10.1145/1198555.1198750 · Cited by 2,568
(1998) · Springer eBooks · doi:10.1007/978-94-011-5014-9 · Cited by 2,418
Information processing in dynamical systems: foundations of harmony theory
Smolensky, P. (1986) · The MIT Press eBooks · Cited by 2,233
A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
Neal, R. M., & Hinton, G. E. (1998) · Learning in Graphical Models · doi:10.1007/978-94-011-5014-9_12 · Cited by 2,187
Self-citationADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 16
Storkey, A. (2004) · neural information processing systems · Cited by 2,085
Advances in Neural Information Processing Systems 18
Murray, I., Mackay, D., Zoubin Ghahramani, A., & Skilling, J. (2006) · Cited by 1,927
Restricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Self-citationNeighbourhood Components Analysis
Goldberger, J., Hinton, G. E., Roweis, S. T., & Salakhutdinov, R. (2004) · Cited by 1,855
Self-citationAdvances in Neural Information Processing Systems 21
Griffiths, T. L., Lucas, C. G., Williams, J. J., & Kalish, M. L. (2009) · neural information processing systems · Cited by 1,854
Acoustic Modeling Using Deep Belief Networks
Mohamed, A.-R., Dahl, G. E., & Hinton, G. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,727
Self-citationLearning methods for generic object recognition with invariance to pose and lighting
LeCun, Y., Huang, F. J., & Bottou, L. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315150 · Cited by 1,324
(1993) · Cited by 1,173
Neal, R. M. (2001) · Statistics and Computing · doi:10.1023/a:1008923215028 · Cited by 1,172
Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition
Ranzato, M., Huang, F. J., Boureau, Y.-L., & LeCun, Y. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383157 · Cited by 1,086
(2007) · Cited by 937
A feedforward architecture accounts for rapid categorization
Serre, T., Oliva, A., & Poggio, T. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0700622104 · Cited by 918
Advances in Neural Information Processing Systems 7
Tesauro, G., Touretzky, D. S., & Leen, T. K. (1995) · Cited by 869
Training restricted Boltzmann machines using approximations to the likelihood gradient
Tieleman, T. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390290 · Cited by 837
On Contrastive Divergence Learning.
Carreira-Perpiñán, M. Á., & Hinton, G. E. (2005) · Cited by 630
Self-citationTraining Invariant Support Vector Machines
Decoste, D., & Schölkopf, B. (2002) · Machine Learning · doi:10.1023/a:1012454411458 · Cited by 588
Replicated Softmax: an Undirected Topic Model
Hinton, G. E., & Salakhutdinov, R. (2009) · neural information processing systems · Cited by 538
Self-citation(2007) · The MIT Press eBooks · doi:10.7551/mitpress/7496.001.0001 · Cited by 536
Connectionist learning of belief networks
Neal, R. M. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90065-6 · Cited by 513
Efficient Learning of Deep Boltzmann Machines
Salakhutdinov, R., & Larochelle, H. (2010) · International Conference on Artificial Intelligence and Statistics · Cited by 502
Self-citationHinton, G. E., & Sejnowski, J. (1983) · Cited by 498
Self-citationLearning a Nonlinear Embedding by Preserving Class Neighbourhood Structure
Salakhutdinov, R., & Hinton, G. E. (2007) · International Conference on Artificial Intelligence and Statistics · Cited by 440
Self-citationExponential Family Harmoniums with an Application to Information Retrieval
Welling, M., Rosen‐Zvi, M., & Hinton, G. E. (2004) · Cited by 423
Self-citationOn the quantitative analysis of deep belief networks
Salakhutdinov, R., & Murray, I. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390266 · Cited by 411
Self-citationHerding dynamical weights to learn
Welling, M. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553517 · Cited by 401
To recognize shapes, first learn to generate images
Hinton, G. E. (2007) · Progress in Brain Research · doi:10.1016/s0079-6123(06)65034-6 · Cited by 309
Self-citationUsing fast weights to improve persistent contrastive divergence
Tieleman, T., & Hinton, G. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553506 · Cited by 274 · cited 1×
“Tieleman and Hinton (2009) argue that the fact that the parameters are being updated using the data-independent statistics gathered from the persistent chains means that the mixing rate of the chains with their parameters fixed is not what limits the maximum acceptable learning rate.”
BackgroundSelf-citationPhone Recognition with the Mean-Covariance Restricted Boltzmann Machine
Dahl, G. E., Ranzato, M., Mohamed, A., & Hinton, G. E. (2010) · Cited by 266
Self-citationParametric Inference for imperfectly observed Gibbsian fields
Younès, L. (1989) · Probability Theory and Related Fields · doi:10.1007/bf00341287 · Cited by 157
Tempered Markov Chain Monte Carlo for training of Restricted Boltzmann Machines
Desjardins, G., Courville, A. C., Bengio, Y., Vincent, P., & Delalleau, O. (2010) · International Conference on Artificial Intelligence and Statistics · Cited by 153
On the convergence of markovian stochastic algorithms with rapidly decreasing ergodicity rates
Laurent Younes, B. (1999) · Stochastics and stochastics reports · doi:10.1080/17442509908834179 · Cited by 134
Hinton, G. E. (2013) · Cognitive Science · doi:10.1111/cogs.12049 · Cited by 132
Self-citationModeling image patches with a directed hierarchy of Markov random fields
Osindero, S., & Hinton, G. E. (2007) · Cited by 103
Self-citationA minimum description length framework for unsupervised learning
Zemel, R. S. (1994) · Cited by 95
Implicit Mixtures of Restricted Boltzmann Machines
Nair, V. B., & Hinton, G. E. (2008) · neural information processing systems · Cited by 90
Self-citationLearning Deep Boltzmann Machines using Adaptive MCMC
Salakhutdinov, R. (2010) · International Conference on Machine Learning · Cited by 88
Self-citationLearning in Markov Random Fields using Tempered Transitions
Salakhutdinov, R. (2009) · Cited by 81
Self-citationEvaluating probabilities under high-dimensional latent variable models
Murray, I., & Salakhutdinov, R. (2008) · ERA · Cited by 80
Self-citationReverse engineering the cognitive brain
Cauwenberghs, G. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1313114110 · Cited by 50
Computational Neuroscience: Theoretical Insights into Brain Function
Cisek, P., Drew, T., Kalaska, J. F., central, G. D. R. S. L. S. N., & Montréal, U. D. (2007) · Progress in brain research · doi:10.1016/s0079-6123(06)x6500-7 · Cited by 46
Measuring the usefulness of hidden units in Boltzmann machines with mutual information
Berglund, M., Raiko, T., & Cho, K. (2014) · Neural Networks · doi:10.1016/j.neunet.2014.09.004 · Cited by 37
Boltzmann Machine Learning Using Mean Field Theory and Linear Response Correction
Kappen, H. J., & Rodrı́guez, F. B. (1997) · Radboud Repository (Radboud University) · Cited by 30
Diffusion Networks, Products of Experts, and Factor Analysis
Marks, T. K., & Movellan, J. R. (2001) · Cited by 24
Spike-Timing-Dependent Construction
Lightheart, T., Grainger, S., & Lu, T. (2013) · Neural Computation · doi:10.1162/neco_a_00501 · Cited by 5
Cited by
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Deep learning for computer vision: A brief review
Voulodimos, A., Doulamis, N., Doulamis, A., & Protopapadakis, E. (2018) · Computational Intelligence and Neuroscience · doi:10.1155/2018/7068349 · Cited by 3,403
Remote Sensing Image Scene Classification: Benchmark and State of the Art
Cheng, G., Han, J., & Lu, X. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2675998 · Cited by 2,926
Deep learning for visual understanding: A review
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.116 · Cited by 2,504
“To increase the ef fi - ciency of DBMs, some researchers introduced an approximate inference algorithm [66,67], which utilizes a separate “ recognition ” model to initialize the values of the latent variables in all layers, thus effectively accelerating the inference.”
MethodDaDianNao: A Machine-Learning Supercomputer
Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., Li, L., Chen, T., Xu, Z., Sun, N., et al. (2014) · 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1109/micro.2014.58 · Cited by 1,492
Metadata
- Authors
- Salakhutdinov, R., & Hinton, G.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2012-08
- Volume
- 24
- Issue
- 8
- Pages
- 1967-2006
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2071709160
- Cited by
- 484
- References
- 66
- FWCI
- 27.86
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.