Effective learning of deep belief nets, with application to handwritten digits and face images

par/by Simon Osindero
Department of Computer Science
University of Toronto

In many interesting generative models, inference and parameter learning are problematic due to the complicated dependencies that can arise between hidden variables when we are presented with an observation. In this talk, I will explain how a bipartite undirected graphical model can be viewed as a directed belief network with infinitely many layers and tied parameters. The insight gained from this equivalence can be used to motivate a fast, greedy algorithm that can learn deep networks one layer at a time. This allows us to bypass many of the inference problems that arise if we try to learn the full model all at once. The fast, greedy algorithm can be used to initialize further global learning procedures that fine-tune the parameters. After such fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model can give better classification performance than discriminative learning algorithms. A similar style of network can be applied to a wide range of datasets and I will also present some recent results in which we use our method to build a generative model of face images.