r1000 - 03 Jan 2008 - 21:11:55 - PascalLamblinYou are here: TWiki >  Public Web  > DeepBeliefNetworks > DBNEquations

# Contrastive divergence multi-layer RBMs

## Boltzmann machines

Random variable (may be) observed, is (always) hidden. Their joint is given by the Boltzmann distribution associated with an energy function :

where Z is the appropriate normalization constant:

.

In ordinary Boltzmann machines, the energy function is a quadratic polynomial. Let z=(x,h), then

where . There is no need for a constant term since it would cancel out in .

If is observed while remains hidden, the likelihood involves a sum over all configurations of :

In unrestricted Boltzmann machines, summing over (in the numerator) and over (in the denominator ) are both intractable. We will see below that the sum over becomes tractable when interactions between hidden units are removed (this is the Restricted Boltzmann Machine).

Hence, the exact gradient of wrt or is also intractable in a general Boltzmann machine. The gradient can be written as a sum of the corresponding two terms:

The derivation of this result is easy (we do a similar derivation for the restricted Boltzmann machine below).

The standard way to estimate the gradient, to avoid these sums, is to perform an MCMC scheme to obtain one or more samples from with training set and from .

## Restricted Boltzmann machines

If we set the weight between and to and the weight between and to , we obtain a Restricted Boltzmann Machine (RBM). The advantage of an RBM is that all the 's become independent when conditioning on , and (symmetrically) all the become independent when conditioning on .

### Energy functions for Restricted Boltzmann Machines

(Note that )

• energy term for binomial unit with value and inputs , parameters (, ):

• energy term for fixed-variance Gaussian unit with value and inputs , parameters :

Note how these blow up when is too small. We may want to use instead, with fixed.

• energy term for softmax units with value and inputs , parameters :

### Update rule

Likelihood gradient for a RBM with observed inputs x, hidden outputs y:

use

For ANY energy-based (Boltzmann) distribution:

where E is over the model's distribution

The positive phase tries to lower the energy of observed while the negative phase tries to increase the energy of all .

For a RBM, factorizes into and energy is a sum over , and so that

with the parameters associated with .

With Contrastive Divergence we replace the expectation over by a sample taken after 1 (or more) Gibbs sampling steps

and the pair serves as that sample in the case of 1 step (= CD1'').

• output binomial unit <-> input binomial unit
• weight :
• positive phase contribution:
• negative phase contribution:
• bias :
• positive phase contribution:
• negative phase contribution:

• output binomial unit <-> input Gaussian unit
• bias and weight as above
• parameter :
• positive phase contribution:
• negative phase contribution:

• output softmax unit <-> input binomial unit
same formulas as for binomial units, except that is computed differently (with softmax instead of sigmoid)

## Using and training the last layers

### Using

If we train the network in a supervised fashion, we introduce a layer containing the outputs (or targets), Y. Let's call the last layer L and the previous layer P, and define:

         [______ L ______]
/         \
[_______ P _______] [___ Y ___]


R.V. (=sample) of the last layer =

R.V. (=sample) of the previous (next-to-last) layer =

R.V. of the supervised layer =

energy parameters between and : ; energies

energy parameters between and : ; energies

"output" (expectation) of next-to-last layer = (given the inputs of the network)

• The activation of is computed from , not :

with

(see below for explanation)

• The expectations of , from the activations ( is a multinomial units set)

• fprop = expectations ( activations )

### Training

There are two ways of learning the parameters , , and

• We compute the , where is the observed target, and we backprop all the way.

• By using contrastive divergence:
• we consider [ Y, P ] = X a big layer, beyond L
• we put [onehot(k), p(P)] = x in X, as input of L
• (X,L) forms an RBM, which we train using contrastive divergence

### Why?

The output probabilities are computed as follows:

This formula can be derived by considering that P, L, and Y are binary random variables following the Boltzmann distribution with energy:

During training, both P and Y are observed, so that E is linear in L, i.e. P(L|P,Y) is a product of : the are conditionally independant given P and Y.

This corresponds to an undirected graphical model with full connectivity between each and each (and similarly between and each ), but no connection among the or among the 's. Because of this factorization we obtain that

and

where

Since , we obtain that

which gives the above formula for P(Y=onehot(k)|inputs), if we replace P by its expectation p(P).

-- PascalLamblin - 19 Jun 2007

Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r1000 < r999 < r998 < r997 < r996 | More topic actions
Public.DBNEquations moved from Neurones.DBNEquations on 21 Jun 2007 - 20:09 by PascalLamblin - put it back