The "Wake-Sleep" Algorithm for Unsupervised Neural Networks
The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
An unsupervised learning algorithm for a multilayer network of stochastic neurons is described. Bottom-up "recognition" connections convert the input into representations in successive hidden layers, and top-down "generative" connections reconstruct the representation in one layer from the representation in the layer above. In the "wake" phase, neurons are driven by recognition connections, and generative connections are adapted to increase the probability that they would reconstruct the correct activity vector in the layer below. In the "sleep" phase, neurons are driven by generative connections, and recognition connections are adapted to increase the probability that they would produce the correct activity vector in the layer above.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A first meta-structure pops up when considering the classical Wake-Sleep algorithm [52], described above as a model for NREM sleep (that we extended by random perturbations in the replay).”
From How Adversarial REM Dreams May Facilitate Creativity, and Why We Become Aware of Them (2024) · cites this paper as [52]
In the citing paper
“In this algorithm, the FF recognition pathway is trained during the Sleep phase so that it inverts the FB generative pathway. The generative FB pathway is itself trained during the Wake phase so that it learns to reproduce the sensory activity from the internal representation generated by the FF pathway.”
From How Adversarial REM Dreams May Facilitate Creativity, and Why We Become Aware of Them (2024) · cites this paper as [52]
In the citing paper
“Bayesian inference and related theories have been proposed as a more appropriate theoretical framework for reasoning about top-down visual processing in the brain. 1,2,4,6,26,27”
From Hierarchical Bayesian inference in the visual cortex (2003) · cites this paper as [2]
In the citing paper
“In one common form of unsupervised learning, higher brain areas attempt to produce samples that are statistically similar to those actually seen in lower layers.”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Hinton et al., 1995
In the citing paper
“For example, the wake-sleep algorithm (Hinton et al., 1995) requires the sleep mode to sample potential data points whose distribution should then match the observed distribution.”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Hinton et al., 1995
Citations
Cites
Ballard, D. H., & Brown, C. M. (1982) · Cited by 5,303
A massively parallel architecture for a self-organizing neural pattern recognition machine
Carpenter, G. A., & Grossberg, S. (1987) · Computer Vision, Graphics, and Image Processing · doi:10.1016/s0734-189x(87)80014-2 · Cited by 2,525
Barlow, H. B. (1989) · Neural Computation · doi:10.1162/neco.1989.1.3.295 · Cited by 1,650
Stochastic Complexity in Statistical Inquiry
Rissanen, J. (1998) · WORLD SCIENTIFIC eBooks · doi:10.1142/0822 · Cited by 1,539
Forward models: supervised learning with a distal teacher
Jordan, M. I., & Rumelhart, D. E. (1992) · Cognitive Science · doi:10.1207/s15516709cog1603_1 · Cited by 1,451
Dayan, P., Hinton, G. E., Neal, R. M., & Zemel, R. S. (1995) · Neural Computation · doi:10.1162/neco.1995.7.5.889 · Cited by 1,207
Self-citationHasselmo, M. E., & Bower, J. M. (1993) · Trends in Neurosciences · doi:10.1016/0166-2236(93)90159-j · Cited by 427
A forward-inverse optics model of reciprocal connections between visual cortical areas
Kawato, M., Hayakawa, H., & Inui, T. (1993) · Network: Computation in Neural Systems · doi:10.1088/0954-898x_4_4_001 · Cited by 151
Grenander, U. (1978) · Springer eBooks · Cited by 89
Research works printed but not linked: 3
- Hinton, G. E., Advances in Neural Information Processing Systems 6: 3 (1994).
- Mumford, D., Large-Scale Theories of the Cortex: 125 (1994).
- Ullman, S., Large-Scale Theories of the Cortex: 257 (1994).
Cited by
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Self-citationAuto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Self-citationLearning the parts of objects by non-negative matrix factorization
Lee, D. D., & Seung, H. S. (1999) · Nature · doi:10.1038/44565 · Cited by 13,868
Whatever next? Predictive brains, situated agents, and the future of cognitive science
Clark, A. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000477 · Cited by 5,924
Emergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694
Deep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879
Self-citationGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841
A theory of cortical responses.
Friston, K. (2005) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2005.1622 · Cited by 4,578
“B (2005) from conventional statistical models (e.g. factor and cluster analysis) and those motivated by Bayesian inference and learning (e.g. Dayan et al. 1995; Hinton et al. 1995).”
BackgroundMethodGenerative Adversarial Networks
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2661 · Cited by 4,477
An Introduction to Variational Methods for Graphical Models
Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999) · Machine Learning · doi:10.1023/a:1007665907178 · Cited by 3,791
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239
“In the fine-tuning stage, the DBNs are trained with labeled da ta by the up-down algorithm which is a contrastive version of the wake-sleep algorithm [57].”
MethodDeep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637
“The wake-sleep algorithm [14, 6] introduced the idea of training inference and generative probabilistic models against each other.”
MethodUnsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629
An Introduction to Variational Autoencoders
Kingma, D. P., & Welling, M. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000056 · Cited by 2,609
“The wake-sleep algorithm [Hinton et al., 1995] is, to the best of our knowledge, the only other on-line learning method in the literature that is applicable to the same general class of continuous latent”
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550
Building Machines That Learn and Think Like People
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2016) · arXiv · doi:10.48550/arxiv.1604.00289 · Cited by 2,550
Generative Adversarial Networks
(2022) · The Science of Deep Learning · doi:10.1017/9781108891530.013 · Cited by 2,402
Tononi, G., & Cirelli, C. (2014) · Neuron · doi:10.1016/j.neuron.2013.12.025 · Cited by 2,356
“Indeed, the strengthening of fantasies is a known problem in neural networks that learn based on a wake-sleep algorithm in which feedforward (‘‘recognition’’) connections that match feedback (‘‘generative’’) connections are potentiated in the sleep phase (Hinton et al., 1995).”
BackgroundBayesian integration in sensorimotor learning
Körding, K. P., & Wolpert, D. M. (2004) · Nature · doi:10.1038/nature02169 · Cited by 2,004
How does the brain solve visual object recognition?
DiCarlo, J. J., Zoccolan, D., & Rust, N. C. (2012) · Neuron · doi:10.1016/j.neuron.2012.01.010 · Cited by 1,790
“A related but distinct idea is that the hierarchy of areas plays a key role at a much slower time scale—in particular, for learning to properly configure a largely feedforward ‘‘serial chain’’ processing system (Hinton et al., 1995).”
Acoustic Modeling Using Deep Belief Networks
Mohamed, A.-R., Dahl, G. E., & Hinton, G. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,727
Self-citationUsing goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
Hierarchical Bayesian inference in the visual cortex
Lee, T. S., & Mumford, D. (2003) · Journal of the Optical Society of America A · doi:10.1364/josaa.20.001434 · Cited by 1,473
Tutorial on Variational Autoencoders
Doersch, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05908 · Cited by 1,470
A Survey on Deep Learning: Algorithms, Techniques, and Applications
Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M.-L., Chen, S.-C., & Iyengar, S. S. (2019) · ACM Computing Surveys · doi:10.1145/3234150 · Cited by 1,403
Adaptive Computation and Machine Learning
(2012) · Boosting · doi:10.7551/mitpress/8291.003.0026 · Cited by 1,402
Ahmed, R., Sreeram, V., Mishra, Y., & Arif, M. D. (2020) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2020.109792 · Cited by 1,259
“This is a continuous process that progresses until it reaches the output layer [216]; thus, it can learn patterns progressively.”
BackgroundDayan, P., Hinton, G. E., Neal, R. M., & Zemel, R. S. (1995) · Neural Computation · doi:10.1162/neco.1995.7.5.889 · Cited by 1,207
Self-citationInfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.03657 · Cited by 1,156
Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models
Lawrence, N. D. (2005) · Journal of Machine Learning Research · doi:10.5555/1046920.1194904 · Cited by 1,156
Exploring Strategies for Training Deep Neural Networks
Larochelle, H., Bengio, Y., Louradour, J., & Lamblin, P. (2009) · doi:10.1145/1577069.1577070 · Cited by 1,095
Kriegeskorte, N. (2015) · Annual Review of Vision Science · doi:10.1146/annurev-vision-082114-035447 · Cited by 1,093
Data-Efficient Image Recognition with Contrastive Predictive Coding
Hénaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S. M. A., & Oord, A. V. D. (2019) · arXiv · doi:10.48550/arxiv.1905.09272 · Cited by 1,014
Learning multiple layers of representation
Hinton, G. E. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.09.004 · Cited by 1,011
Self-citationUnsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005
“The first class of algorithms focus on learning generative models with strong priors [20, 46].”
BackgroundToward Controlled Generation of Text
Hu, Z., Yang, Z., Liang, X., Salakhutdinov, R., & Xing, E. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00955 · Cited by 119
Replay and Time Compression of Recurring Spike Sequences in the Hippocampus
Nádasdy, Z., Hirase, H., Czurkó, A., Csicsvari, J., & Buzsáki, G. (1999) · The Journal of Neuroscience · doi:10.1523/jneurosci.19-21-09497.1999 · Cited by 932
“These observations support the suggestion that time-compressed neuronal patterns during SPW bursts are generated within the hippocampus and are a consequence of firing patterns in the wake brain (Buzsáki, 1989; Chrobak and Buzsaki, 1994; Bibbig et al., 1995; Hinton et al., 1995; McClelland et al., 1995; Skaggs and McN…”
BackgroundLillicrap, T. P., Santoro, A., Marris, L., Akerman, C. J., & Hinton, G. (2020) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-0277-3 · Cited by 908
Self-citationNeural Networks and Deep Learning
Theodoridis, S. (2015) · Machine Learning · doi:10.1016/b978-0-12-801522-3.00018-5 · Cited by 900
Separating Style and Content with Bilinear Models
Tenenbaum, J. B., & Freeman, W. T. (2000) · Neural Computation · doi:10.1162/089976600300015349 · Cited by 840
Deep unsupervised learning using nonequilibrium thermodynamics
Sohl‐Dickstein, J. (2024) · doi:10.57702/qrfwyszh · Cited by 57
Sleep, learning, and dreams: off-line memory reprocessing
Stickgold, R., Hobson, J. A., Fosse, R., & Fosse, M. (2001) · Science · doi:10.1126/science.1063530 · Cited by 762
Learning and inference in the brain
Friston, K. (2003) · Neural Networks · doi:10.1016/j.neunet.2003.06.005 · Cited by 724
“The goal of generative models is “to learn representations that are economical to describe but allow the input to be reconstructed accurately” (Hinton et al., 1995).”
BackgroundDeep learning in optical metrology: a review
Zuo, C., Qian, J., Feng, S., Yin, W., Li, Y., Fan, P., Han, J., Qian, K., & Chen, Q. (2022) · Light: Science & Applications · doi:10.1038/s41377-022-00714-x · Cited by 711
Statistically optimal perception and learning: from behavior to neural representations
Fiser, J., Berkes, P., Orbán, G., & Lengyel, M. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.01.003 · Cited by 672
Whittington, J. C. R., Muller, T. H., Mark, S., Chen, G., Barry, C., Burgess, N., & Behrens, T. E. J. (2020) · Cell · doi:10.1016/j.cell.2020.10.024 · Cited by 660
Riesenhuber, M., & Poggio, T. (2000) · Nature Neuroscience · doi:10.1038/81479 · Cited by 658
An Introduction to Variational Autoencoders
Kingma, D. P., & Welling, M. (2019) · doi:10.1561/9781680836233 · Cited by 643
Proximal Policy Optimization in StarCraft
Liu, Y. (2025) · doi:10.12794/metadc1505267 · Cited by 615
Why Fiction May be Twice as True as Fact: Fiction as Cognitive and Emotional Simulation
Oatley, K. (1999) · Review of General Psychology · doi:10.1037/1089-2680.3.2.101 · Cited by 585
Wainberg, M., Merico, D., Delong, A., & Frey, B. J. (2018) · Nature Biotechnology · doi:10.1038/nbt.4233 · Cited by 585
Self-citationStochastic neural computation. I. Computational elements
Brown, B. D., & Card, H. C. (2001) · IEEE Transactions on Computers · doi:10.1109/12.954505 · Cited by 555
Computational Neuroscience of Vision
Rolls, E., & Deco, G. (2001) · doi:10.1093/acprof:oso/9780198524885.001.0001 · Cited by 553
An Introduction to Variational Methods for Graphical Models
Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1998) · Learning in Graphical Models · doi:10.1007/978-94-011-5014-9_5 · Cited by 551
Hallucinations and strong priors
Corlett, P. R., Horga, G., Fletcher, P. C., Alderson-Day, B., Schmack, K., & Powers, A. R. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.12.001 · Cited by 534
Generative Moment Matching Networks
Li, Y., Swersky, K., & Zemel, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.02761 · Cited by 517
Deep Learning of Representations: Looking Forward
Bengio, Y. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-39593-2_1 · Cited by 513
Attias, H. (1999) · Neural Computation · doi:10.1162/089976699300016458 · Cited by 504
Efficient Learning of Deep Boltzmann Machines
Salakhutdinov, R., & Larochelle, H. (2010) · International Conference on Artificial Intelligence and Statistics · Cited by 501
Kar, K., Kubilius, J., Schmidt, K., Issa, E. B., & DiCarlo, J. J. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0392-5 · Cited by 496
Why do we sleep?11Published on the World Wide Web on 7 November 2000.
Sejnowski, T. J., & Destexhe, A. (2000) · Brain Research · doi:10.1016/s0006-8993(00)03007-9 · Cited by 489
Fundamentals, materials, and machine learning of polymer electrolyte membrane fuel cell technology
Wang, Y., Seo, B., Wang, B., Zamel, N., Jiao, K., & Adroher, X. C. (2020) · Energy and AI · doi:10.1016/j.egyai.2020.100014 · Cited by 484
Deterministic annealing EM algorithm
Ueda, N., & Nakano, R. (1998) · Neural Networks · doi:10.1016/s0893-6080(97)00133-0 · Cited by 469
A refined model of sleep and the time course of memory formation
Walker, M. P. (2005) · Behavioral and Brain Sciences · doi:10.1017/s0140525x05000026 · Cited by 468
Beyond Phrenology: What Can Neuroimaging Tell Us About Distributed Circuitry?
Friston, K. (2002) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.25.112701.142846 · Cited by 465
Song replay during sleep and computational rules for sensorimotor vocal learning
Dave, A. S., & Margoliash, D. (2000) · Science · doi:10.1126/science.290.5492.812 · Cited by 458
The hippocampal sharp wave-ripple in memory retrieval for immediate use and consolidation
Joo, H. R., & Frank, L. M. (2018) · Nature Reviews Neuroscience · doi:10.1038/s41583-018-0077-1 · Cited by 430
Top-down predictions in the cognitive brain
Kveraga, K., Ghuman, A. S., & Bar, M. (2007) · Brain and Cognition · doi:10.1016/j.bandc.2007.06.007 · Cited by 423
Attentive Neural ProcessesPreprint
Kim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, A., Rosenbaum, D., Vinyals, O., & Teh, Y. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.05761 · Cited by 414
On the Origin of Deep Learning
Wang, H., & Raj, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.07800 · Cited by 83
Buzsáki, G. (2019) · doi:10.1093/oso/9780190905385.001.0001 · Cited by 358
A Review of Deep Learning Methods and Applications for Unmanned Aerial Vehicles
Carrio, A., Sampedro, C., Rodriguez-Ramos, A., & Campoy, P. (2017) · Journal of Sensors · doi:10.1155/2017/3296874 · Cited by 356
Structured Inference Networks for Nonlinear State Space Models
Krishnan, R., Shalit, U., & Sontag, D. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10779 · Cited by 346
Ghahramani, Z. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-28650-9_5 · Cited by 346
Modeling the manifolds of images of handwritten digits
Hinton, G. E., Dayan, P., & Revow, M. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.554192 · Cited by 330
Self-citationFunctional integration and inference in the brain
Friston, K. (2002) · Progress in Neurobiology · doi:10.1016/s0301-0082(02)00076-x · Cited by 330
3D-RCNN: Instance-Level 3D Object Reconstruction via Render-and-Compare
Kundu, A., Li, Y., & Rehg, J. M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00375 · Cited by 328
Mean Field Theory for Sigmoid Belief Networks
Saul, L. K., Jaakkola, T., & Jordan, M. I. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.251 · Cited by 327
Deep Learning Advances in Computer Vision with 3D Data: A Survey
Ioannidou, A., Chatzilari, E., Nikolopoulos, S., & Kompatsiaris, I. (2018) · ACM Computing Surveys · doi:10.1145/3042064 · Cited by 327
A deep belief network based fault diagnosis model for complex chemical processes
Zhang, Z., & Zhao, J. (2017) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2017.02.041 · Cited by 309
To recognize shapes, first learn to generate images
Hinton, G. E. (2007) · Progress in Brain Research · doi:10.1016/s0079-6123(06)65034-6 · Cited by 309
Self-citationSynthetic data augmentation for surface defect detection and classification using deep learning
Jain, S., Seth, G., Paruthi, A., Soni, U., & Kumar, G. (2022) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-020-01710-x · Cited by 302
Variational inference for Monte Carlo objectivesPreprint
Mnih, A., & Rezende, D. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.06725 · Cited by 293
Dynamic Model of Visual Recognition Predicts Neural Response Properties in the Visual Cortex
Rao, R. P. N., & Ballard, D. H. (1997) · Neural Computation · doi:10.1162/neco.1997.9.4.721 · Cited by 290
“While the derivation of the algorithm necessitates some approximations and simplifications, which may sometimes prevent it from converging to the correct posterior distribution, experimental results using the algorithm have been positive [Hinton et al., 1995].”
MethodResultInfluentialNeural Autoregressive Distribution Estimation
Uría, B., Côté, M.-A., Gregor, K., Murray, I., & Larochelle, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.02226 · Cited by 19
Waking and dreaming consciousness: neurobiological and functional considerations.
Hobson, J. A., & Friston, K. J. (2012) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2012.05.003 · Cited by 287
“Indeed, this separation into two phases of optimization underlies the wake-sleep algorithm in machine learning that minimizes free-energy to provide a parsimonious and unsupervised model of data (Hinton et al., 1995).”
BackgroundWhat is the mammalian dentate gyrus good for?
Treves, A., Tashiro, A., Witter, M. P., & Moser, E. I. (2008) · Neuroscience · doi:10.1016/j.neuroscience.2008.04.073 · Cited by 282
“Sleep / wake algorithms , studied in machine learning , separate a wake phase in which activity reflects inputs from the sensory world and is propagated forward , and a sleep phase in which it reflects internal " models of the world " and is propagated backward ( Hinton et al , 1995 ) .”
A Cost-Sensitive Deep Belief Network for Imbalanced Classification
Zhang, C., Tan, K. C., Li, H., & Hong, G. S. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2832648 · Cited by 282
Towards Biologically Plausible Deep Learning
Bengio, Y., Lee, D.-H., Bornschein, J., Mesnard, T., & Lin, Z. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04156 · Cited by 276
Single Units and Visual Cortical Organization
Lennie, P. (1998) · Perception · doi:10.1068/p270889 · Cited by 276
Zhou, Z.-H., Chawla, N. V., Jin, Y., & Williams, G. J. (2014) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2014.2350953 · Cited by 275
Opportunities and challenges for quantum-assisted machine learning in near-term quantum computers
Perdomo-Ortiz, A., Benedetti, M., Realpe-Gómez, J., & Biswas, R. (2018) · Quantum Science and Technology · doi:10.1088/2058-9565/aab859 · Cited by 274
Plasticity in single neuron and circuit computations
Destexhe, A., & Marder, E. (2004) · Nature · doi:10.1038/nature03011 · Cited by 274
Reinforcement learning in artificial and biological systems
Neftci, E. O., & Averbeck, B. B. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0025-4 · Cited by 268
Connectionist modelling in psychology: A localist manifesto
Page, M. (2000) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00003356 · Cited by 267
Metadata
- Authors
- Hinton, G. E., Dayan, P., Frey, B. J., & Neal, R. M.
- Affiliations
- University of Toronto (CA)
- ORCID
- Dayan, Peter
- Venue
- Science
- Published
- 1995-05-26
- Volume
- 268
- Issue
- 5214
- Pages
- 1158-1161
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1993845689
- Cited by
- 1,070
- References
- 9
- Citations by year
- 2026: 18 · 2025: 33 · 2024: 36 · 2023: 57 · 2022: 57 · 2021: 68 · 2020: 63 · 2019: 79 · 2018: 79 · 2017: 64 · 2016: 52 · 2015: 42 · 2014: 37 · 2013: 19 · 2012: 18
- FWCI
- 28.04
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- unsupervised learning, stochastic neurons, multilayer neural networks, top-down connections, representation learning, generative models, neural network adaptation
- MeSH terms
- Algorithms, Probability, Stochastic Processes, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.