The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
An Efficient Learning Procedure for Deep Boltzmann Machines
Open accessAbstract
We present a new learning algorithm for Boltzmann machines that contain many layers of hidden variables. Data-dependent statistics are estimated using a variational approximation that tends to focus on a single mode, and data-independent statistics are estimated using persistent Markov chains. The use of two quite different techniques for estimating the two types of statistic that enter into the gradient of the log likelihood makes it practical to learn Boltzmann machines with multiple hidden layers and millions of parameters. The learning can be made more efficient by using a layer-by-layer pretraining phase that initializes the weights sensibly. The pretraining also allows the variational inference to be initialized sensibly with a single bottom-up pass. We present results on the MNIST and NORB data sets showing that deep Boltzmann machines learn very good generative models of handwritten digits and 3D objects. We also show that the features discovered by deep Boltzmann machines are a very effective way to initialize the hidden layers of feedforward neural nets, which are then discriminatively fine-tuned.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Optimization by Simulated Annealing
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983) · Science · doi:10.1126/science.220.4598.671 · Cited by 45,134
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Self-citationStochastic relaxation, gibbs distributions, and the bayesian restoration of images.
Geman, S., & Geman, D. (1984) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1984.4767596 · Cited by 17,949
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Self-citationAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
(2010) · International Conference on Machine Learning · Cited by 13,995
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Cited by
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Deep learning for computer vision: A brief review
Voulodimos, A., Doulamis, N., Doulamis, A., & Protopapadakis, E. (2018) · Computational Intelligence and Neuroscience · doi:10.1155/2018/7068349 · Cited by 3,403
Remote Sensing Image Scene Classification: Benchmark and State of the Art
Cheng, G., Han, J., & Lu, X. (2017) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2675998 · Cited by 2,926
Deep learning for visual understanding: A review
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.116 · Cited by 2,504
“To increase the ef fi - ciency of DBMs, some researchers introduced an approximate inference algorithm [66,67], which utilizes a separate “ recognition ” model to initialize the values of the latent variables in all layers, thus effectively accelerating the inference.”
MethodDaDianNao: A Machine-Learning Supercomputer
Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., Li, L., Chen, T., Xu, Z., Sun, N., et al. (2014) · 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture · doi:10.1109/micro.2014.58 · Cited by 1,492
Deep learning for computational biology
Angermueller, C., Pärnamaa, T., Parts, L., & Stegle, O. (2016) · Molecular Systems Biology · doi:10.15252/msb.20156651 · Cited by 1,473
“…for modelling images (Box 2 ), recurrent neural networks for sequential data (Sutskever, 2013 ; Lipton, 2015 ), or restricted Boltzmann machines (Salakhutdinov & Larochelle, 2010 ; Hinton, 2012 ) and autoencoders (Hinton & Salakhutdinov, 2006 ; Alain et al , 2012 ; Kingma & Welling, 2013 ) for…”
A Survey on Deep Learning: Algorithms, Techniques, and Applications
Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M.-L., Chen, S.-C., & Iyengar, S. S. (2019) · ACM Computing Surveys · doi:10.1145/3234150 · Cited by 1,403
Deep Learning for Single Image Super-Resolution: A Brief Review
Yang, W., Zhang, X., Tian, Y., Wang, W., Xue, J.-H., & Liao, Q. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2919431 · Cited by 1,154
Bao, W., Yue, J., & Rao, Y. (2017) · PLOS ONE · doi:10.1371/journal.pone.0180944 · Cited by 1,124
Hierarchical Feature Representation and Multimodal Fusion with Deep Learning for AD/MCI Diagnosis
Suk, H.-I., Lee, S.-W., & Shen, D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.06.077 · Cited by 968
“…procedure after the initial bottom-up pass incorporates top-down feedback, which allows DBM to use higher-level knowledge to resolve uncertainty about intermediate-level features, thus creating better data-dependent representations and statistics for learning (Salakhutdinov and Hinton, 2012).”
InfluentialNweke, H. F., Teh, Y. W., Al-garadi, M. A., & Alo, U. R. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.03.056 · Cited by 930
Fadlullah, Z. M., Tang, F., Mao, B., Kato, N., Akashi, O., Inoue, T., & Mizutani, K. (2017) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2017.2707140 · Cited by 810
Deep Learning in Drug Discovery
Gawehn, E., Hiss, J. A., & Schneider, G. (2016) · Molecular Informatics · doi:10.1002/minf.201501008 · Cited by 723
Cross-modal Retrieval with Correspondence Autoencoder
Feng, F., Wang, X., & Li, R. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654902 · Cited by 560
Hallucinations and strong priors
Corlett, P. R., Horga, G., Fletcher, P. C., Alderson-Day, B., Schmack, K., & Powers, A. R. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.12.001 · Cited by 534
Gearbox fault diagnosis based on deep random forest fusion of acoustic and vibratory signals
Li, C., Sanchez, R.-V., Zurita, G., Cerrada, M., Cabrera, D., & Vásquez, R. E. (2016) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2016.02.007 · Cited by 470
A Deep Learning Scheme for Motor Imagery Classification based on Restricted Boltzmann Machines
Lu, N., Li, T., Ren, X., & Miao, H. (2017) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2016.2601240 · Cited by 419
Kato, N., Fadlullah, Z. M., Mao, B., Tang, F., Akashi, O., Inoue, T., & Mizutani, K. (2017) · IEEE Wireless Communications · doi:10.1109/mwc.2016.1600317wc · Cited by 400
Sum-Product Networks: A New Deep Architecture
Poon, H., & Domingos, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1202.3732 · Cited by 352
Deep Learning for Medical Image-Based Cancer Diagnosis
Jiang, X., Hu, Z., Wang, S., & Zhang, Y. (2023) · Cancers · doi:10.3390/cancers15143608 · Cited by 345
Grotjahn, R., Black, R., Leung, R., Wehner, M. F., Barlow, M., Bosilovich, M., Gershunov, A., Gutowski, W. J., Gyakum, J. R., Katz, R. W., et al. (2016) · Climate Dynamics · doi:10.1007/s00382-015-2638-6 · Cited by 331
Sum-product networks: A new deep architecture
Poon, H., & Domingos, P. (2011) · 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) · doi:10.1109/iccvw.2011.6130310 · Cited by 315
Li, C., Sanchez, R.-V., Zurita, G., Cerrada, M., Cabrera, D., & Vásquez, R. E. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.06.008 · Cited by 295
Deep learning methods and applications for electrical power systems: A comprehensive review
Ozcanli, A. K., Yaprakdal, F., & Baysal, M. (2020) · International Journal of Energy Research · doi:10.1002/er.5331 · Cited by 270
Neural Variability and Sampling-Based Probabilistic Representations in the Visual Cortex
Orbán, G., Berkes, P., Fiser, J., & Lengyel, M. (2016) · Neuron · doi:10.1016/j.neuron.2016.09.038 · Cited by 255
Deep neural networks-based rolling bearing fault diagnosis
Chen, Z., Deng, S., Chen, X., Li, C., Sanchez, R.-V., & Qin, H. (2017) · Microelectronics Reliability · doi:10.1016/j.microrel.2017.03.006 · Cited by 254
Active inference on discrete state-spaces: A synthesis
Da Costa, L., Parr, T., Sajid, N., Veselic, S., Neacsu, V., & Friston, K. (2020) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2020.102447 · Cited by 226
“Examples of generative models that do not fall within the current discrete state-space, continuous state-space [8,18,30,182–185] or mixed [27–29] models – currently implemented in active inference – include Markov decision trees [74,186] and Boltzmann machines [77,187,188].”
BackgroundOn the Expressive Power of Deep Architectures
Bengio, Y., & Delalleau, O. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24412-4_3 · Cited by 182
Deep Learning Intervention for Health Care Challenges: Some Biomedical Domain Considerations
Tobore, I., Li, J., Yuhang, L., Al-Handarish, Y., Kandwal, A., Nie, Z., & Wang, L. (2019) · JMIR mHealth and uHealth · doi:10.2196/11966 · Cited by 180
Early Alzheimer’s disease diagnosis based on EEG spectral images using deep learning
Bi, X., & Wang, H. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.02.005 · Cited by 169
Fu, Y., Li, C., Yu, F. R., Luan, T. H., & Zhang, Y. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3083927 · Cited by 169
Prediction error, ketamine and psychosis: An updated model
Corlett, P. R., Honey, G. D., & Fletcher, P. C. (2016) · Journal of Psychopharmacology · doi:10.1177/0269881116650087 · Cited by 142
van Tulder, G., & de Bruijne, M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2526687 · Cited by 142
Deep Learning: A Bayesian Perspective
Polson, N. G., & Sokolov, V. (2017) · Bayesian Analysis · doi:10.1214/17-ba1082 · Cited by 120
Guo, J., Zhang, W., Fan, W., & Li, W. (2018) · Journal of Management Information Systems · doi:10.1080/07421222.2018.1523564 · Cited by 85
Issa, E. B., Cadieu, C. F., & DiCarlo, J. J. (2018) · eLife · doi:10.7554/elife.42870 · Cited by 76
Khoshboresh Masouleh, M., & Shah-Hosseini, R. (2019) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2019.07.009 · Cited by 51
Luo, X., Feng, Y., & Zhang, M. (2021) · IEEE Access · doi:10.1109/access.2021.3075344 · Cited by 47
Breaking away from labels: The promise of self-supervised machine learning in intelligent health
Spathis, D., Perez-Pozuelo, I., Marques-Fernandez, L., & Mascolo, C. (2022) · Patterns · doi:10.1016/j.patter.2021.100410 · Cited by 39
An Instruction Set Architecture for Machine Learning
Chen, Y., Lan, H., Du, Z., Liu, S., Tao, J., Han, D., Luo, T., Guo, Q., Li, L., Xie, Y., et al. (2018) · ACM Transactions on Computer Systems · doi:10.1145/3331469 · Cited by 14
Evolving Long Short-Term Memory Networks
Lobo Neto, V. C., Passos, L. A., & Papa, J. P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-50417-5_25 · Cited by 9
A deep neural network for audio-visual person recognition
Alam, M. R., Bennamoun, M., Togneri, R., & Sohel, F. (2015) · 2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS) · doi:10.1109/btas.2015.7358754 · Cited by 8
Metadata
- Authors
- Salakhutdinov, R., & Hinton, G.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 2012-08
- Volume
- 24
- Issue
- 8
- Pages
- 1967-2006
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2071709160
- Cited by
- 484
- References
- 66
- FWCI
- 27.86
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.