Institute for Pure and Applied Mathematics (IPAM)
Representation Learning and Deep Learning
U. Montreal
July, 2012, UCLA
The success of machine learning algorithms generally depends on data
representation, and we hypothesize that this is because different
representations can entangle and hide more or less the different
explanatory factors of variation behind the data. Although domain
knowledge can be used to help design representations, learning can also
be used, and the quest for AI is motivating the design of more powerful
representation-learning algorithms. We view the ultimate goal of these
algorithms as disentangling the unknown underlying factors of variation
that explain the observed data.This tutorial reviews the basics of
feature learning and deep learning, as well as recent work relating
these subjects to probabilistic modeling and manifold learning.
An objective is to raise questions and issues about the appropriate
objectives for learning good representations, for computing
representations (i.e., inference), and the geometrical connections
between representation learning, density estimation and manifold
learning.
These lectures include also an overview of the application of
deep learning in Natural Language Processing, and a discussion
of connections between the optimization issues of local minima
in deep architectures and the evolution of culture, as playing a role
to reduce that optimization difficulty.
Outline:
- Motivations and Scope
- Feature / Representation learning
- Distributed representations
- Exploiting unlabeled data
- Deep representations
- Multi-task / Transfer learning
- Invariance vs Disentangling
- Algorithms
- Probabilistic models and RBM variants
- Auto-encoder variants (sparse, denoising, contractive)
- Explaining away, sparse coding and Predictive Sparse Decomposition
- Deep variants
- Sampling from regularized auto-encoders
- Regularized auto-encoders as implicit density estimators
- Analysis, Issues and Practice
- Tips and tricks
- Partition function gradient
- Inference
- Mixing between modes
- Geometry and probabilistic interpretations of auto-encoders
- Open questions
- Applications of Deep Learning to NLP
- Neural Language Model
- Dealing with the Output Bottleneck
- Applications, Modeling Semantics
- Recursive Neural Networks
- Culture vs Local Minima
VIDEOS OF LECTURES
LARGE FILE: final version of the slides (pdf, 48M)
(updated 20/07/2012, 10h15 PT)
MORE CONVENIENT - IN PARTS:
- slides, part 1, motivations (pdf, 15M)
- slides, part 2, algorithms / pre-training, RBMs(pdf, 10M)
- slides, part 3, algorithms / auto-encoders, depth (pdf, 12M)
- slides, part 4, questions & issues (pdf, 5M)
- slides, part 5, NLP (pdf, 10M)
- slides, part 6, NLP / recursive nets (pdf, 4M)
- slides, part 7, culture vs local minima (pdf, 4M)
Bibliographic references(pdf, 68k)