Roland Memisevic I am interested in AI systems that can see, understand and interact with humans. I believe that the key to more human-like AI is enabling it to hold an audio-visual dialogue with real people, in real-time, about the real world. I received the PhD from the University of Toronto in 2008 working with Geoffrey Hinton. email: memisevr[at]iro[dot]umontreal[dot]ca Bio, CV

• This recent slide deck (Oct 2020) describes our work at Twenty Billion Neurons, a company I co-founded in 2015


• Check out 20BN Sense, our open source software that can turn an input stream of pixels into gestures, actions, and other human behaviours


• We've built an end-to-end AI-powered fitness companion as a show-case application
  


• Demo videos showing our situated avatars based on the same technology

In the news

• CNBC published an article about our AI fitness coach
  
  
• My interview at the Chris Collin's show
  
  
• Bloomberg wrote about one of our first AI avatar demos (this was at NeurIPS)
  

Publications

2022

• 2022 Memisevic, R.
  Metaphors we learn by
  [pdf]


• 2022 Memisevic, R., Panchal, S., Lee, M.
  Decision Making as Language Generation
  NeurIPS Foundation Models for Decision Making workshop 2022
  [pdf]
  

2020

• 2020 Mercier, A., Berger, G., Panchal, S., Letsch, F., Boehm, C., Bax, I. Memisevic, R.
  Putting the skeleton back in the closet
  [html] [bibtex]
  

2019

• 2019 Materzynska, J., Berger G., Bax, I., Memisevic, R.
  The jester dataset: A large-scale video dataset of human gestures
  ICCV workshop HANDS 2019
  [pdf]

2018

• 2018 Mahdisoltani, F., Berger, G., Gharbieh, W., Memisevic, R., Fleet, D.
  The more fine-grained, the better for transfer learning
  NeurIPS Workshop on Modeling and Decision-Making in the Spatiotemporal Domain 2018
  [pdf]


• 2018 Goyal, R., Mahdisoltani, F., Berger, G., Gharbieh, W., Bax, I., Memisevic R.
  Evaluating visual" common sense" using fine-grained classification and captioning tasks
  ICLR workshop
  [pdf]
  


• 2018 Mahdisoltani, F., Berger, G., Gharbieh, W., Fleet, D., Memisevic R.
  On the effectiveness of task granularity for transfer learning
  arXiv [pdf]

2017

• 2017 Goyal, R., Kahou, S., Michalski, V. Materzynska, J., Westphal, *, Bax., I., Memisevic, R.
  * see paper for additional authors.
  The "something something" video database for learning and evaluating visual common sense
  International Conference on Computer Vision (ICCV 2017).
  arXiv [pdf]


• 2017 Ebrahimi Kahou, S., Michalski, V., Memisevic, R., Pal, C., Vincent, P.
  RATM: Recurrent attentive tracking model
  CVPR workshop [pdf]

2016

• 2016 Berger, G., Memisevic R.
  Incorporating long-range consistency in CNN-based texture generation
  arXiv [pdf] (accepted at ICLR 2017)
  
  
• 2016 Lin, Z., Courbariaux M., Memisevic, R., Bengio Y.
  Neural Networks with Few Multiplications
  International Conference on Learning Representations (ICLR 2016)
  [pdf]
  
  
• 2016 Krueger, D., Memisevic, R.
  Regularizing RNNs by Stabilizing Activations
  International Conference on Learning Representations (ICLR 2016)
  [pdf]
  
  
• 2016 Im, D., Belghazi, M., Memisevic, R.
  Conservativeness of Untied Auto-Encoders
  Thirtieth AAAI Conference on Artificial Intelligence (AAAI 2016)
  
  
• 2016 de Vries, H., Memisevic, R., Courville, A.
  Deep Learning Vector Quantization
  European Symposium on Artificial Neural Networks 2016
  
  
• 2016 Zhang ,S., Wu, Y., Che, T., Lin, Z, Memisevic, R., Salakhutdinov, R., Bengio, Y.
  Architectural Complexity Measures of Recurrent Neural Networks
  Neural Information Processing Systems (NIPS 2016)
  

2015

• 2015 Kamyshanska, H., Memisevic, R.
  The potential energy of an autoencoder
  IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)
  preprint
  
  
• 2015 Konda, K., Memisevic, R., Krueger, D.
  Zero-bias autoencoders and the benefits of co-adapting features
  International Conference on Learning Representations (ICLR 2015) [code]
  
  
• 2015 Konda, K., Memisevic, R.
  Learning visual odometry with a convolutional network
  International Conference on Computer Vision Theory and Applications (VISAPP 2015)
  [pdf]
  
  
• 2015 Jean, S., Cho, K., Memisevic, R., Bengio, Y.
  On using very large target vocabulary for neural machine translation
  Annual meeting of the Association for Computational Linguistics (ACL 2015)
  
  
• 2015 Konda, K., Chandrasekariah, P., Memisevic, R., Triesch, J.
  Real-time activity recognition via deep learning of motion features
  European Symposium on Artificial Neural Networks (ESANN2015)
  
  
• 2015 Kahou, S.E., Michalski, V., Konda, K., Memisevic, R., and Pal, C.
  Recurrent Neural Networks for Emotion Recognition in Video
  To appear in the proceedings of the 17th ACM International Conference on Multimodal Interaction (ICMI 2015).
  
  
• 2015 Kahou, S.E., Michalski V., Memisevic R.
  RATM: Recurrent Attentive Tracking Model
  arXiv:1510.08660 [pdf]
  
  
• 2015 Kahou, S. E., Bouthillier, X., *, P., Courville, A., Vincent, P., Memisevic, R., Pal, C., Bengio, Y.
  EmoNets: Multimodal deep learning approaches for emotion recognition in video
  * see paper for additional authors.
  Journal on Multimodal User Interfaces
  [pdf]

2014

• 2014 Michalski, V., Memisevic, R., Konda, K.
  Modeling Deep Temporal Dependencies with Recurrent "Grammar Cells"
  Neural Information Processing Systems (NIPS 2014) [pdf]
  an early version of this paper available on arXiv:1402.2333 [pdf]
  
  
• 2014 Konda, K., Memisevic, R., Michalski, V.
  Learning to encode motion using spatio-temporal synchrony
  International Conference on Learning Representations (ICLR 2014)
  [pdf]
  
  
• 2014 Memisevic, R., Konda, K., Krueger, D.
  Zero-bias autoencoders and the benefits of co-adapting features
  arXiv:1402.3337
  [pdf]
  
  
• 2014 Konda, K., Memisevic, R.
  A unified approach to learning depth and motion features (ICVGIP 2014)
  

2013

• 2013 Konda, K., Memisevic, R.
  Unsupervised learning of depth and motion.
  arXiv:1312.3429v2
  [pdf]
  
  
• 2013 Kahou, S. E., Pal, C., Bouthillier, X., *, Memisevic, R., Vincent, P., Courville, A. and Bengio, Y.
  Combining Modality Specific Deep Neural Networks for Emotion Recognition in Video
  ICMI 2013
  * see paper for additional authors.
  The paper describes our model that won the ICMI 2013 Grand Challenge on Emotion Recognition in the Wild.
  [pdf]
  
  
• 2013 Kamyshanska, H., Memisevic, R.
  On autoencoder scoring.
  International Conference on Machine Learning (ICML 2013) (oral)
  [pdf] [bibtex]
  
  
• 2013 Memisevic, R., Exarchakis, G.
  Learning invariant features by harnessing the aperture problem.
  International Conference on Machine Learning (ICML 2013)
  [pdf] [bibtex]
  
  
• 2013 Memisevic, R.
  Learning to relate images.
  IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)
  Special issue on Learning Deep Architectures
  [preprint pdf] © IEEE [bibtex]
  
  
• 2013 Bauer, F., Memisevic, R.
  Feature grouping from spatially constrained multiplicative interaction.
  International Conference on Learning Representations (ICLR 2013) (oral)
  [pdf]
  
  

2012

• 2012 Memisevic, R.
  On multi-view feature learning.
  International Conference on Machine Learning (ICML 2012) (oral)
  [pdf] [bibtex] [code]
  
  
• 2012 Memisevic, R., Sigal, L., Fleet, D.
  Shared Kernel Information Embedding for Discriminative Inference.
  IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)
  [pdf] © IEEE [code] [bibtex]
  
  

2011

• 2011 Memisevic, R.
  Gradient-based learning of higher-order image features.
  International Conference on Computer Vision (ICCV 2011).
  [pdf] [bibtex] © IEEE [Website]
  
  
• 2011 Susskind, J., Memisevic, R., Hinton, G., Pollefeys, M.
  Modeling the joint density of two images under a variety of transformations.
  Computer Vision and Pattern Recognition (CVPR 2011).
  [pdf] [bibtex] [code] © IEEE
  
  
• 2011 Memisevic, R.
  Learning to relate images: Mapping units, complex cells and simultaneous eigenspaces.
  arXiv:1110.0107v1 [pdf] [bibtex]
  
  
• 2011 Memisevic, R., Conrad, C.
  Depth as a latent variable.
  Frontiers in Neuroscience Conference Abstract: BC11
  [link]
  
  
• 2011 Memisevic, R.
  On spatio-temporal sparse coding: Analysis and an algorithm.
  NIPS workshop on Deep Learning and Unsupervised Feature Learning 2011
  [pdf]
  
  
• 2011 Memisevic, R., Conrad, C.
  Stereopsis via Deep Learning.
  NIPS workshop on Deep Learning and Unsupervised Feature Learning 2011
  [pdf]
  
  

2010

• 2010 Memisevic, R., Hinton, G.
  Learning to Represent Spatial Transformations with Factored Higher-Order Boltzmann Machines.
  June 2010 edition, Vol. 22, No. 6: 1473-1492, Journal Neural Computation.
  [pdf], [bibtex], [Website]
  related 2009 technical report: [pdf]
  
  
• 2010 Memisevic, R., Zach, C., Hinton, G., Pollefeys, M.
  Gated Softmax Classification.
  Neural Information Processing Systems (NIPS 2010).
  [pdf], [bibtex], [Website].
  

2009 and before

• 2009 Sigal, L., Memisevic, R., Fleet, D.
  Shared kernel information embedding for discriminative inference.
  Computer Vision and Pattern Recognition (CVPR 2009 © IEEE).
  [pdf]
  
  
• 2008 Memisevic, R.
  Non-linear latent factor models for revealing structure in high-dimensional data.
  PhD thesis, University of Toronto 2008.
  [pdf], [bibtex]
  
  
• 2007 Memisevic, R. and Hinton, G. E.
  Unsupervised learning of image transformations.
  Computer Vision and Pattern Recognition (CVPR 2007 © IEEE).
  [pdf], [bibtex]
  (Related Technical Report: [pdf])
  
  
• 2007 Samulowitz, H. and Memisevic, R.
  Learning to solve QBF.
  Twenty-Second Conference on Artificial Intelligence (AAAI 2007).
  [pdf][bibtex]
  
  
• 2006 Memisevic, R.
  An introduction to structured discriminative learning. Technical report.
  University of Toronto, 2006. [ps],[pdf]
  
  
• 2006 Memisevic, R.
  Dual optimization of conditional probability models.
  NIPS Workshop on Kernel Methods and Structured Domains
  (Technical report, University of Toronto, 2006: [ps],[pdf])
  
  
• 2006 Memisevic, R.
  Kernel Information Embeddings.
  23rd International Conference on Machine Learning (ICML 2006).
  [pdf],[bibtex], [more info], [python code], [GPU version].
  
  
• 2006 Memisevic, R.
  Propagating Errors and Beliefs for Large Scale Nonlinear Structure Prediction.
  North East Student Colloquium on Artificial Intelligence. Ithaca, NY.
  [pdf],[bibtex]
  
  
• 2005 Meinicke, P., Klanke, S., Memisevic, R., Ritter, H.
  Principal Surfaces from Unsupervised Kernel Regression.
  IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)
  Vol. 27, no. 9, pp. 1379-1391. [bibtex]
  
  
• 2005 Memisevic, R. and Hinton, G. E.
  Improving dimensionality reduction with spectral gradient descent.
  Journal Neural Networks.
  18, pp 702-710.
  [online version],[bibtex]
  
  
• 2005 Memisevic, R. and Hinton, G. E.
  Embedding via clustering: Using spectral information to guide dimensionality reduction.
  International Joint Conference on Neural Networks 2005.
  
  
• 2004 Memisevic, R. and Hinton, G. E.
  Multiple Relational Embedding.
  Advances in Neural Information Processing Systems (NIPS 2004).
  [ps.gz],[pdf],[bibtex]
  
  
• 2003 Memisevic, R.
  Unsupervised Kernel Regression for Nonlinear Dimensionality Reduction.
  Master's Thesis, Bielefeld University, 2003.
  [pdf]

Professional activities

• Senior PC, PC or reviewer for CVPR 2004, 2005, 2010, 2011 (PC), 2012 (PC); NIPS 2006, 2007, 2008, 2009, 2011; ICCV 2010, 2011 (PC); ECCV 2010; DAGM 2011 (Senior PC), 2012 (Senior PC); ICML 2006, 2008 (PC), 2012 (PC).
• Reviewer for IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Journal of Machine Learning Research (JMLR), Journal Image and Vision Computing (IMAVIS), Journal Constraint, Journal Machine Learning, IEEE Transactions on Image Processing, International Journal of Adaptive Control and Signal Processing.
• Grant reviewer, Swiss National Science Foundation.
• Invited speaker at the CIFAR ncap summer school 2013, Toronto.
• Invited speaker at the IPAM summer school on deep learning, UCLA.
• I gave tutorials at CVPR 2012 and DAGM 2011 on higher-order feature learning and on spatio-temporal learning [CVPR tutorial website].
  
  

Teaching

• Winter 2016: Machine Learning for Vision
• Winter 2015: Databases (IFT2821)
• Winter 2015: Learning for vision (IFT6268)
• Fall 2014: Machine Learning (Fondements de l'apprentissage machine) (IFT3395/6390)
  
• Fall 2013: Learning for vision (IFT 6268, based on IFT 6085 taught in the winter), University of Montreal.
  
• Winter 2013: Visual Feature Learning (IFT 6085), University of Montreal.
  
• Winter 2011/12: Machine Learning, University of Frankfurt.
  
• Spring 2011: Digital Image Processing (in German), University of Frankfurt.
  
• Before 2011: Teaching Assistant at the University of Toronto: Introduction to Neural Networks and Machine Learning (CSC321, 2006 and 2007), Software Design (CSC207, 2006), Machine Learning (CSC2515, 2005 and 2006).

Miscellaneous

• Some fun figures.
• A Python module implementing hidden Markov models: hmm.py
  Supports discrete or Gaussian observations and training on a single or on multiple observation sequences.
• minimize.py is a Python adaptation of Carl Rasmussen's matlab function minimize.m
• Slides and a few code-snippets for a ten-minute mini-tutorial on switching from Matlab to Python.

Contact information