Deep belief networks are universal approximators and a recurrent neural network that learns to remember

par/by Ilya Sutskever
Department of Computer Science
University of Toronto

The talk will consist of two parts. In the first part I will prove that narrow deep belief networks can approximate any distribution over binary vectors to arbitrary accuracy, and describe a simple greedy algorithm that can learn such networks in an impractical manner.

In the second part I will introduce a new kind of recurrent neural networks that can learn long term dependencies much better than standard recurrent neural networks.