Deep belief networks

Abstract

Recently, Hinton et al.[6] derived a way to perform fast, greedy learning of deep belief networks (DBN) one layer at a time, with the top two layers forming an undirected bipartite graph (associate memory). The learning procedure consists of training a stack of Restricted Boltzmann Machines (RBM’s) each having only one layer of latent (hidden) feature detectors. The learned feature activations of one RBM are used as the “data” for training the next RBM in the stack. The important aspect of this layer-wise training procedure is that, provided the number of features per layer does not decrease, [6] showed that each extra layer increases a variational lower bound on the log probability of data. So layer-by-layer training can be repeated several times 1 to learn a deep, hierarchical model in which each layer of features captures strong high-order correlations between the activities of features in the layer below. We will discuss three ideas based on greedily learning a hierarchy of features: Nonlinear Dimensionality Reduction. The DBN framework allows us to make nonlinear autoencoders work considerably better [7] than widely used methods such as PCA, SVD, and LLE. The standard way to train autoencoders is to use backpropagation to reduce the reconstruction error. It is difficult, however, to optimize the weights in non-linear autoencoders that have multiple hidden layers with many million parameters [3, 5]. We use our greedy learning algorithm to pretrain autoencoders. This pretraining stage discovers useful features efficiently. After the pretraining stage, the model is “unfolded” to produce encoder and decoder networks that initially use the same weights. The global fine-tuning stage then uses backpropagation through the whole autoencoder to fine-tune the weights for optimal reconstruction. The key idea is that the greedy learning algorithm