Wake-sleep Algorithm for Representational Learning

par/by Hamid Reza Maei
Physiology & Program in Neuroscience
University of Toronto

How the representations of objects form in the brain is a high-level question. Since last decade, scientists have been trying to develop unsupervised learning algorithms to address representational learning. Here, I will describe an unsupervised learning method on multilayer neural network of stochastic neurons using wake-sleep algorithm. During the wake phase, the bottom-up recognition connections convert the input data to higher representations and top-down generative weights are adjusted such that they can reconstruct the input data. In sleep phase neurons are driven by generative connections and recognition connections are adjusted to increase the probability of the correct activity in the layer above. The main challenge of wake-sleep algorithm (as directed graphical model) is that the exact inference is intractable and therefore approximation methods have been applied. I will compare the performance of the model trained by wake-sleep algorithm with the performance of a naïve model such as naïve Bayes classifier on a real dataset (20 news group data). At the end of the talk I will describe, briefly, how restricting the architecture of network can improve the performance of the model.