Deep Learning Workshop: Foundations and Future Directions
Hyatt hotel, Vancouver, Canada, December 6th, 2007, 2pm-5:30pm, Georgia B room, 2nd floor
Organizers: Yoshua Bengio, Yann LeCun, Ruslan Salakhutdinov and Hugo Larochelle
This is the afternoon part of
NIPS'2007 special
Neuro-Thursday, sponsored by the Canadian Institute For Advanced Research (
CIFAR).
Workshop description
Theoretical results strongly suggest that in order to learn the
kind of complicated functions that can represent high-level abstractions
(e.g. in vision, language, and other AI-level tasks), one may need "deep
architectures", which are composed of multiple levels of non-linear
operations (such as in neural nets with many hidden layers). Searching
the parameter space of deep architectures is a difficult optimization
task, but learning algorithms (e.g. Deep Belief Networks) have recently
been proposed to tackle this problem with notable success, beating
the state-of-the-art in certain areas.
This workshop is intended to bring together researchers interested in
the question of deep learning in order to review the current algorithms'
principles and successes, but also to identify the challenges, and to
formulate promising directions of investigation. Besides the algorithms
themselves, there are many fundamental questions that need to be
addressed: What would be a good formalization of deep learning? What
new ideas could be exploited to make further inroads to that difficult
optimization problem? What makes a good high-level representation or abstraction?
What type of problem is deep learning appropriate for?
The workshop is also going to be an opportunity to celebrate
Geoff Hinton's 60th birthday and his contributions to theories of neural information processing.
Registration Procedure
Registration is required in order to attend the workshop.
Free registration for an evening bus going to Whisler is also offered to workshop participants. This
bus trip is different from the one offered on the NIPS registration page (i.e. you do not
need to ask for it when registering to NIPS).
The number of places at the Hyatt is limited, so early registration is highly recommended!
The registration page can be found
here.
Schedule
Videos of all workshop talks are available
here.
2:00pm - 2:05pm Introductory remarks |
avi|
2:05pm - 2:25pm
Yee-Whye Teh, Gatsby Unit :
Setting the Stage: Complementary Prior and Variational Bounds |
pdf part1.avi part2.avi|
2:25pm - 2:45pm
John Langford, Yahoo Research:
(Lack of) Deep Learning Theory |
pdf avi|
2:45pm - 3:05pm
Yoshua Bengio, University of Montreal:
Optimizing Deep Architectures |
pdf part1.avi part2.avi|
3:05pm - 3:25pm
Yann Le Cun, New York University:
Learning a Deep Hierarchy of Sparse Invariant Features |
pdf part1.avi part2.avi|
3:25pm - 3:45pm
Martin Szummer, Microsoft Research:
Deep Networks for Information Retrieval |
pdf part1.avi part2.avi|
3:45pm - 4:00pm Coffee break
4:00pm - 4:20pm
Max Welling, University of California:
(Infinite) Deep Networks |
ppt part1.avi part2.avi|
4:20pm - 4:40pm
Rajat Raina, Stanford University:
Self-taught Learning |
ppt pptx avi|
4:40pm - 5:00pm
Geoff Hinton, University of Toronto:
How to do Backpropagation in a Brain |
ppt avi|
5:00pm - 5:30pm Discussion |
avi|
Relevant literature
First paper introducing Deep Belief Networks (as generative models):
- A fast learning algorithm for deep belief nets | pdf ps.gz html |
Hinton, G. E., Osindero, S. and Teh, Y.
Neural Computation (2006)
Review paper on deep architectures and details of Deep Belief Nets and Restricted Boltzmann Machines:
- Learning Deep Architectures for AI | link |
Bengio, Y. Foundations and Trends in Machine Learning (to appear, 2009).
Book chapter about the philosophy behind deep architecture model, motivating them in the context of Artificial Intelligence
- Scaling Learning Algorithms towards AI | pdf |
Bengio, Y. and LeCun, Y.
Book chapter in "Large-Scale Kernel Machines"
Deep Belief Networks as a simple way of initializing a deep feed-forward neural network:
- To recognize shapes, first learn to generate images | pdf |
Hinton, G. E.
Technical Report (2006)
General study of the framework of initializing a deep feed-forward neural network using a greedy layer-wise procedure:
- Greedy Layer-Wise Training of Deep Networks | pdf tech-report-pdf |
Bengio, Y., Lamblin, P., Popovici, P., Larochelle, H.
NIPS 2006
An application of greedy layer-wise learning of a deep autoassociator for dimensionality reduction:
- Reducing the dimensionality of data with neural networks | pdf support-pdf code |
Hinton, G. E. and Salakhutdinov, R. R.
Science 2006
A way to use the greedy layer-wise learning procedure to learn a useful embeding for k nearest neighbor classification:
- Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure | pdf |
Salakhutdinov, R. R. and Hinton, G. E.
AISTATS 2007
Different theoretical results about Restricted Boltzmann Machines (RBMs) and Deep Belief Networks, like the universal approximation property of RBMs:
- Representational Power of Restricted Boltzmann Machines and Deep Belief Networks | pdf |
Le Roux, N. and Bengio, Y.
Technical Report, to appear in Neural Computation.
A novel way of using greedy layer-wise learning for Convolutional Networks:
- Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition | pdf |
Ranzato, M'A, Huang, F-J, Boureau, Y-L, and Le Cun, Y.
CVPR 2007
How to generalize Restricted Boltzmann Machines to types of data other than binary using exponential familly distribution:
- Exponential Family Harmoniums with an Application to Information Retrieval | pdf ps |
Welling, M., Rosen-Zvi, M. and Hinton, G. E.
NIPS 2004
An evaluation of deep networks on many datasets related to vision:
- An Empirical Evaluation of Deep Architectures on Problems with Many Factors of Variation | pdf html |
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., Bengio, Y.
ICML 2007
Application of deep learning in the context of information retrieval:
- Semantic Hashing | pdf |
Salakhutdinov, R. R. and Hinton, G. E.
IRGM 2007