This is the RBM update procedure for binomial units. It also works for exponential and truncated exponential units, and for the linear parameters of a Gaussian unit (using the appropriate sampling procedure for Q and P). It can be readily adapted for the variance parameter of Gaussian units.
v[0]
is a sample from the training distribution for the RBM
epsilon
is a learning rate for the stochastic gradient descent in Contrastive Divergence
W
is the RBM weight matrix, of dimension (number of hidden units, number of inputs)
b
is the RBM biases vector for hidden units
c
is the RBM biases vector for input units
RBMupdate(v[0], epsilon, W, b, c): for all hidden units i: compute Q(h[0][i] = 1 | v[0]) # for binomial units, sigmoid(b[i] + sum_j(W[i][j] * v[0][j])) sample h[0][i] from Q(h[0][i] = 1 | v[0]) for all visible units j: compute P(v[1][j] = 1 | h[0]) # for binomial units, sigmoid(c[j] + sum_i(W[i][j] * h[0][i])) sample v[1][j] from P(v[1][j] = 1 | h[0]) for all hidden units i: compute Q(h[1][i] = 1 | v[1]) # for binomial units, sigmoid(b[i] + \sum_j(W[i][j] * v[1][j])) W += epsilon * (h[0] * v[0]' - Q(h[1][.] = 1 | v[1]) * v[1]') b += epsilon * (h[0] - Q(h[1][.] = 1 | v[1])) c += epsilon * (v[0] - v[1])$
Train a DBN in a purely unsupervised way, with the greedy layer-wise procedure in which each added layer is trained as an RBM by contrastive divergence.
X
is the input training distribution for the network
epsilon
is a learning rate for the stochastic gradient descent in Contrastive Divergence
L
is the number of layers to train
n=(n[1], ...,n[L])
is the number of hidden units in each layer
W[i]
is the weight matrix for level i
, for i
from 1
to L
b[i]
is the bias vector for level i
, for i
from 0
to L
PreTrainUnsupervisedDBN(X, epsilon, L, n, W, b): initialize b[0]=0 for l=1 to L: initialize W[i]=0, b[i]=0 while not stopping criterion: sample g[0]=x from X for i=1 to l-1: sample g[i] from Q(g[i]|g[i-1]) RBMupdate(g[l-1], epsilon, W[l], b[l], b[l-1])
After a DBN has been initialized by pre-training, this procedure will
optimize all the parameters with respect to the supervised criterion C
,
using stochastic gradient descent.
Z
is the supervised training distribution for the DBN, with (input,target) samples (x,y)
C
is a training criterion, a function that takes a network output f(x)
and a target y
and returns a scalar differentiable in f(x)
epsilon_C
is a learning rate for the stochastic gradient descent on supervised cost C
L
is the number of layers
n=(n[1], ..., n[L])
is the number of hidden units in each layer
W[i] is the weight matrix for level =i
, for i
from 1
to L
b[i]
is the bias vector for level i
, for i
from 0
to L
V
is a weight matrix for the supervised output layer of the network
c
is the bias vector for the supervised output layer
DBNSupervisedFineTuning(Z, C, epsilon_C, L, n, W, b, V, c): Recursively define mean-field propagation mu[i](x)=Expectation(g[i]|g[i-1]=mu[i-1](x)) where mu[0](x)=x, and Expectation(g[i]|g[i-1]=mu[i-1]) is the expected value of g[i] under the RBM conditional distribution Q(g[i]|g[i-1]), when the values of g[i-1] are replaced by the mean-field values mu[i-1](x) # In the case where g[i] has binomial units: # Expectation(g[i][j]|g[i-1]=mu[i-1]) = sigmoid(b[i][j] + sum_k W[i][j][k] mu[i-1][k](x) Define the network output function f(x) = V * mu[L](x)' + c Iteratively minimize the expected value of C(f(x),y) for pairs (x,y) sampled from Z by tuning parameters W, b, V, c. This can be done by stochastic gradient descent with learning rate epsilon_C, using an appropriate stopping criterion such as early stopping on a validation set.
Train a DBN for a supervised learning task, by first performing pre-training
of all layers (except the output weights V
), followed by supervised fine-tuning
to minimize a criterion C
.}\
Z
is the supervised training distribution for the DBN, with (input,target) samples (x,y)
C
is a training criterion, a function that takes a network output f(x)
and a target y
and returns a scalar differentiable in f(x)
epsilon_CD
is a learning rate for the stochastic gradient descent with Contrastive Divergence
epsilon_C
is a learning rate for the stochastic gradient descent on supervised cost C
L
is the number of layers
n=(n[1], ..., n[L])
is the number of hidden units in each layer
W[i] is the weight matrix for level =i
, for i
from 1
to L
b[i]
is the bias vector for level i
, for i
from 0
to L
V
is a weight matrix for the supervised output layer of the network
c
is the bias vector for the supervised output layer
TrainSupervisedDBN(Z, C, epsilon_CD, epsilon_C, L, n, W, b, V): let X the marginal over the input part of Z PreTrainUnsupervisedDBN(X, epsilon_CD, L, n, W, b) DBNSupervisedFineTuning(Z, C, epsilon_C, L, n, W, b, V, c)
When the units of the last layer are binomial, and if the target is a class label,
instead of training the last layer of a DBN using unsupervised Contrastive Divergence,
we can train a joint RBM with (g[L-1], y) as input.
In TrainSupervisedDBN
, We then replace PreTrainUnsupervisedDBN
by PreTrainSupervisedDBN
.
The network configuration is this one:
[______ L ______] / \ [______ L-1 ______] [___ Y ___]
Train the first layers of a DBN in a purely unsupervised way, with the greedy layer-wise procedure in which each added layer is trained as an RBM by contrastive divergence. Then, train the last layer as an RBM modelling the joint distribution of the previous layer and the target.
Z
is the supervised training distribution for the DBN, with (input,target) samples (x,y)
epsilon
is a learning rate for the stochastic gradient descent in Contrastive Divergence
L
is the total number of layers to train
n=(n[1], ...,n[L])
is the number of hidden units in each layer
n_t
is the number of different class labels
W[i]
is the weight matrix for level i
, for i
from 1
to L
b[i]
is the bias vector for level i
, for i
from 0
to L
V
is the weight matrix between the last layer and target layer
c
is the bias vector for target layer
PreTrainSupervisedDBN(Z, epsilon, L, n, W, b, V, c): let X the marginal over the input part of Z initialize b[0]=0 for l=1 to L-1: initialize W[i]=0, b[i]=0 while not stopping criterion: sample g[0]=x from X for i=1 to l-1: sample g[i] from Q(g[i]|g[i-1]) RBMupdate(g[l-1], epsilon, W[l], b[l], b[l-1]) initialize W[L]=0, b[L]=0, V=0, c=0 while not stopping criterion: sample (g[0]=x, y) from Z for i=1 to L-1: sample g[i] from Q(g[i]|g[i-1]) RBMupdate((g[L-1],y), epsilon, (W[L],V'), b[L], (b[L-1],c))
-- PascalLamblin - 21 Jun 2007