Modelling high-dimensional sequential data
using distributed hidden state

par/by Geoffrey Hinton
University of Toronto
and
Canadian Institute for Advanced Research

I will describe a new model for sequential data. The model uses a large number of hidden binary variables that are symmetrically connected to the observable variables to form a restricted Boltzmann machine. In addition, the hidden variables are linked together by directed connections to form a sigmoid belief net. I will describe how the model is trained and show that it can learn a good generative model of human motion from motion capture data. We will show that the same type of model is very good at predicting the next word in a sentence and we will describe ways in which the model can be improved.

Joint work with Graham Taylor, Andriy Mnih and Ilya Sutskever