Contrastive divergence multi-layer RBMs
Boltzmann machines
Random variable

(may be) observed,

is (always) hidden.
Their joint is given by the Boltzmann distribution
associated with an energy function

:
where Z is the appropriate normalization constant:
.
In ordinary Boltzmann machines, the energy function is a
quadratic polynomial. Let z=(x,h), then
where

. There is no need for a constant term
since it would cancel out in

.
If

is observed while

remains hidden, the likelihood
involves a sum over all configurations of

:
In unrestricted Boltzmann machines, summing over

(in the numerator)
and over

(in the denominator

) are both intractable. We will see
below that the sum over

becomes tractable when interactions between
hidden units are removed (this is the Restricted Boltzmann Machine).
Hence, the exact gradient of

wrt

or

is also intractable in a general Boltzmann machine.
The gradient can be written as a sum of the corresponding two terms:
The derivation of this result is easy (we do a similar derivation
for the restricted Boltzmann machine below).
The standard way to estimate the gradient, to avoid these sums, is
to perform an MCMC scheme to obtain one or more samples from

with

training set and from

.
Restricted Boltzmann machines
If we set the weight between

and

to

and the weight between

and

to

, we obtain a
Restricted Boltzmann Machine (RBM).
The advantage of an RBM is that all the

's become independent
when conditioning on

, and (symmetrically) all the

become independent
when conditioning on

.
Energy functions for Restricted Boltzmann Machines
(Note that

)
- energy term for binomial unit
with value
and inputs
, parameters (
,
):
- energy term for fixed-variance Gaussian unit
with value
and inputs
, parameters
:
Note how these blow up when
is too small. We may want to use
instead, with

fixed.
- energy term for softmax units
with value
and inputs
, parameters
:
Update rule
Likelihood gradient for a RBM with observed inputs x, hidden outputs y:
use
For ANY energy-based (Boltzmann) distribution:
where E is over the model's distribution
The positive phase tries to lower the energy of observed

while the negative phase tries to increase the energy of all

.
For a RBM,

factorizes into

and energy is a sum over

, and

so that
with

the parameters associated with

.
With Contrastive Divergence we replace the expectation over

by a sample taken after 1 (or more) Gibbs sampling steps
and the pair

serves as that sample in the case of 1 step (= ``CD1'').
- output binomial unit
<-> input binomial unit
- weight
:
- positive phase contribution:
- negative phase contribution:
- bias
:
- positive phase contribution:
- negative phase contribution:
- output binomial unit
<-> input Gaussian unit
- bias
and weight
as above
- parameter
:
- positive phase contribution:
- negative phase contribution:
- output softmax unit
<-> input binomial unit
same formulas as for binomial units, except that
is computed differently (with softmax instead of sigmoid)
Using and training the last layers
Using
If we train the network in a supervised fashion, we introduce a layer
containing the outputs (or targets), Y.
Let's call the last layer L and the previous layer P, and define:
[______ L ______]
/ \
[_______ P _______] [___ Y ___]
R.V. (=sample) of the last layer =
R.V. (=sample) of the previous (next-to-last) layer =
R.V. of the supervised layer =
energy parameters between

and

:

; energies
energy parameters between

and

:

; energies
"output" (expectation) of next-to-last layer =

(given the inputs of the
network)
- The activation of
is computed from
, not
:
with
(see below for explanation)
- The expectations of
, from the activations (
is a multinomial units set)
- fprop = expectations ( activations )
Training
There are two ways of learning the parameters

,

,

and
- By simple gradient descent:
- We compute the
, where
is the observed target, and we backprop all the way.
- By using contrastive divergence:
- we consider [ Y, P ] = X a big layer, beyond L
- we put [onehot(k), p(P)] = x in X, as input of L
- (X,L) forms an RBM, which we train using contrastive divergence
Why?
The output probabilities are computed as follows:
This formula can be derived by considering that P, L, and Y are binary
random variables following the Boltzmann distribution with energy:
During training, both P and Y are observed, so that E is linear in L,
i.e. P(L|P,Y) is a product of

:
the

are conditionally independant given P and Y.
This corresponds to an undirected graphical model with full connectivity
between each

and each

(and similarly between

and each

),
but no connection among the

or among the

's. Because of this
factorization we obtain that
and
where
Since

, we obtain that
which gives the above formula for P(Y=onehot(k)|inputs), if we replace P by
its expectation p(P).
--
PascalLamblin - 19 Jun 2007