This is a selective survey of our main achievements in the last few years.
Our main achievements have been in breaking through difficult optimization problems that previously plagued some machine learning algorithms, in particular those relevant to our previous MITACS research, i.e., to learn highly-varying functions, such as deep architectures.
These successes have come from two fronts. First, Bengio and Vincent have participated since 2006 in the development of a new family of learning algorithms which were ignored before because of the difficult optimization problem they involved: deep architectures [ 1,2,3,4,5, 6, 7 ] This involves unsupervised learning as one of the key components of the success. The ultimate objective was to develop better predictive models for our partners. Second, Schuurmans after developing convex relaxations of unsupervised learning problems that allow reasonable global training algorithms to be achieved, has achieved a significant breakthrough in developing global, exact training algorithms for non-convex unsupervised learning problems, which completely avoids the use of approximations.
On the front of large datasets, another of the challenges present in our 2006 proposal, we have made progress on a number of fronts. In particular, Chipman[31,32 ] has a computationally efficient Bayesian Ensemble Learning algorithm that draws bridges between the computational statistics and the machine learning communities. Bengio has developped online variants of the successful greedy layerwise algorithm for deep architecture, enabling the application of this training principle on very large datasets.
Knowledge Exchange and Technology Transfer
In March 2009, Chipman organized a Team workshop on mining network data. Over 20 researchers and students, including four members of CSEC participated in this research event designed to develop interest in the emerging area of modelling network data.
In April 2010, Chipman delivered to CSEC a report entitled "Mixed-Membership Stochastic Block-Models for Transactional Networks", and authored by Hugh Chipman and M. Mahdi Shafiei, to summarize the results of the research contract with CSEC. Shafiei is a PDF working with Chipman and funded by MITACS and CSEC. This follows up on two visits to Ottawa (where the main offices of CSEC are located), one in September 2009 and the other in April 2010.
During 2008-2010, Bengio worked with colleague Eck on a project with Apstat, involving three MITACS interns (2 PhDs and 2 post-doc fellows), on the application of deep learning to a task in acoustic signal classification. A paper on this work was submitted to the Computational Intelligence journal and has just been accepted. The co-authors include 3 Apstat employees and 4 university researchers.
During the 2009-2010 academic year, Bengio established relations with Ubisoft as a new industrial partner. Four graduate students were hired by Ubisoft as interns in that period, and the relationship worked so well that three grant proposals were submitted (a CRD in March, a strategic grant in April, and an Industrial Research Chair in August).
In 2009-2010 Schuurmans and his students have been designing and implementing a pilot system (for personnel scheduling) for our partner the Alberta Health Services (the provincial health authority in Alberta). They have conducted five half-day workshops (4 in Edmonton and 1 in Calgary) for the relevant key personnel within AHS, and are working towards a formal assessment of their prototype by AHS.
5 selected recent publications from our group:
- D. Erhan, Y. Bengio, A. Courville, P.A. Manzagol, P .Vincent, S. Bengio (2010), "Why Does Unsupervised Pre-training Help Deep Learning?", Journal of Machine Learning Research, 11:625–660, Feb.
- Bengio, Y., (2009), "Learning deep architectures for AI", Foundations and Trends in Machine Learning, 2:1(1–127), 2009. Also published as a book by Now Publishers.
- Xu, L., White, M. and Schuurmans, D. (2009), "Optimal reverse prediction: a unified perspective on supervised, unsupervised and semi-supervised learning", In Proceedings of the International Conference on Machine Learning (ICML'09). (Honorable Mention as Best Overall Paper.)
- Chipman, H.A., George, E. I. and. McCulloch, R. E. (2010), "BART: Bayesian additive regression trees", Annals of Applied Statistics, 4, 266-298.
- Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Wortman Vaughan, J. (2010), "A theory of learning from different domains", Machine Learning 79(1-2): 151-175.