AAAI 2013

Deep Learning of Representations: a AAAI 2013 Tutorial

Yoshua Bengio

U. Montreal

July 14th, 2013, Bellevue, Washington, USA

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although domain knowledge can be used to help design representations, learning can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms. We view the ultimate goal of these algorithms as disentangling the unknown underlying factors of variation that explain the observed data.This tutorial reviews the basics of feature learning and deep learning, as well as recent work relating these subjects to probabilistic modeling and manifold learning. An objective is to raise questions and issues about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning.

Outline:

Motivations and Scope
- Feature / Representation learning
- Distributed representations
- Exploiting unlabeled data
- Deep representations
- Multi-task / Transfer learning
- Invariance vs Disentangling
Algorithms
- Probabilistic models and RBM variants
- Training computational graphs and auto-encoder variants
- Geometry and probabilistic interpretations of auto-encoders
- Deep variants
Practical Considerations
- Tips and tricks
- Hyper-Parameters
Challenges and Concerns
- Computational Scaling & Distributed Training
- Mixing between modes
- Inference and getting rid of anonymous latent variables
- Disentangling and sparsity
- Culture to fight effective local minima

Final version of the slides (pdf)
Final version of the bibliographic references (pdf)
Review paper having a large overlap with the content of the tutorial: Bengio, Courville and Vincent, "Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives", IEEE Trans. PAMI, 2013.