Greedy Layer-Wise Training of Deep Networks
par/by Yoshua Bengio
Département d'Informatique et de Recherche Opérationnelle
Université de Montréal
Deep multi-layer neural networks have many levels of non-linearities allowing them
to compactly represent highly non-linear and highly-varying functions. However,
until recently it was not clear how to train such deep networks, since gradientbased
optimization starting from random initialization appears to often get stuck
in poor solutions. Hinton et al. recently introduced a greedy layer-wise unsupervised
learning algorithmfor Deep Belief Networks (DBN), a generative model with
many layers of hidden causal variables.
In the context of the above optimization
problem, we present an empirical study of this algorithm and explore variants to better understand
its success and extend it to cases where the inputs are continuous or where
the structure of the input distribution is not revealing enough about the variable to
be predicted in a supervised task. Our experiments also confirm the hypothesis that
the greedy layer-wise unsupervised training strategy mostly helps the optimization,
by initializing weights in a region near a good local minimum, giving rise to internal
distributed representations that are high-level abstractions of the input bringing
better generalization.