Learning relational dictionaries

Modelling two images with one set of hidden variables makes it possible to learn spatio-temporal features that can represent the relationship between the images.

The key ingredient to make this work is to let the three groups of variables interact multiplicatively, leading to a gated sparse coding model.

Factored gated Boltzmann machines ([Memisevic, Hinton, 2010]) implement this idea in the form of a probabilistic model with a simple Hebbian learning rule. But one can apply the approach to other sparse coding techniques.


A Python implementation of a factored gated Boltzmann machine is available here (requires numpy).

An example Python script that instantiates and trains the model on pairs of shifted random images is available here (uses matplotlib for plotting).
(inputimages.txt, outputimages.txt)

(Note: The image files are approximately 40MB each. You can get both image sets in a small compressed tar (tgz) file here.)

Example 1: Learning a dictionary to represent translation

After about 200 epochs of training on the shifted random images (this can easily take an hour or so - don't get discouraged) the learned filters should look something like this:

The model has learned a 2d Fourier transform as the optimal filters for representing translations. (Filters for the input image are shown on the left, filters for the output image on the right.)

Example 2: Learning a dictionary to represent rotation

When training on rotations, the model learns a polar ("circular") generalization of the Fourier transform:

Training on scalings and other types of transformation leads to other types of transformation-specific basis functions.
After learning about transformations it is possible to make analogies, understand the meaning of "the same" and to perform transformation invariant classification [Memisevic, Hinton, 2010].


2010 Memisevic, R., Hinton, G.
Learning to Represent Spatial Transformations with Factored Higher-Order Boltzmann Machines.
Neural Computation June 2010, Vol. 22, No. 6: 1473-1492. [pdf], [tech-report (pdf)], [errata]
The paper introduces the model and shows some simple applications.
Basis functions that the model learned from rotations are shown on the June 2010 edition of Neural Computation (click to enlarge):

2007 Memisevic, R. and Hinton, G. E.
Unsupervised learning of image transformations.
Computer Vision and Pattern Recognition (CVPR 2007). [pdf], [tech-report (pdf)]
Describes a simpler, non-factorized version of the model and a convolutional version.

2008 Memisevic, Roland
Non-linear latent factor models for revealing structure in high-dimensional data.
Doctoral dissertation, University of Toronto. [pdf]
Describes some extensions and a non-probabilistic version of the model.

2011 Memisevic, R.
Gradient-based learning of higher-order image features. (ICCV 2011)
Introduces the gated autoencoder, which provides an efficent way to learn relational features. [pdf]

2013 Memisevic, R.
Learning to relate images.
IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)
Detailed analysis and introduction to gated feature learning and bi-linear models. [preprint pdf]

Webside from multiview feature learning tutorial at cvpr 2012

Slides from IPAM summer school on deep learning at UCLA 2012: Slides part 1 and Slides part 2. (The full schedule with links to video.)

Slides and code from CIFAR summer school in Toronto 2013.