r44 - 28 Jan 2008 - 09:40:50 - YoshuaBengioYou are here: TWiki >  Public Web  > DeepBeliefNetworks

# Deep Belief Networks

Deep Belief Networks (here referred to as DBNs) are deep neural networks, trained in a greedy layer-wise fashion. They were introduced by Hinton et al. (2006). See this review paper for a detailed introduction to such models.

Each layer of the network tries to model the distribution of its input, using unsupervised training in a Restricted Boltzmann Machine:

• H = hidden causes,
• P(h|x) = representation of x.

The unsupervised greedy layer-wise training serves as initialization, replacing the traditional random initialization of multi-layer networks.

Layer-wise training of a DBN

## Restricted Boltzmann Machines

The probabilities of the states that a Boltzmann Machine can take is defined by its energy function:

Inference is easy: factorizes ()

No explaining away.

Training is easy: Contrastive Divergence (Hinton 2002)

Let be the observed input. Imagine Gibbs Markov chain :

Then, for :

where expectation over + sampling with small is easy.

This is contrastive divergence and works (Carreira-Perpiñán 2005).

For more detailed equations, and extension to Gaussian and truncated-exponential units, please see DBNEquations.

## Implementation

The detailed procedure for training a whole DBN is detailed on DBNPseudoCode.

It was implemented using the PLearn machine learning library. The class for a DBN is DeepBeliefNetwork, and it uses different other classes, deriving from RBMLayer (RBMBinomialLayer, RBMGaussianLayer, RBMTruncExpLayer, and RBMMixedLayer) and RBMConnection (RBMMatrixConnection, RBMMixedConnection).

## Examples

Here are two examples of PLearn scripts implementing Deep Belief Nets, applied to MNIST digit recognition task.

You will need to change the path to your data and the hyper-parameters (and possibly the network architecture), either by editing the args class at the beginning of the script, or by passing the appropriate command-line arguments (see the scripts for more details).

-- PascalLamblin - 22 Jun 2007

Show attachmentsHide attachments
Topic attachments
I Attachment Action Size Date Who Comment
png DBNs.png manage 36.8 K 21 Jun 2007 - 22:27 PascalLamblin Layer-wise training of a DBN
png RBM.png manage 7.2 K 21 Jun 2007 - 23:10 PascalLamblin A Restricted Boltzmann Machine
pyplearn example_mnist.pyplearn manage 5.1 K 21 Jun 2007 - 21:13 PascalLamblin Example of .pyplearn script implementing a Deep Belief Net, applied to MNIST digit recognition dataset
pyplearn example_mnist_earlystopping.pyplearn manage 6.4 K 21 Jun 2007 - 21:15 PascalLamblin Example of .pyplearn script implementing a Deep Belief Net, applied to MNIST digit recognition dataset, with early-stopping on a validation set during supervised phases
Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs |  | More topic actions