About learning distributed representations for words
Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Jauvin. A Neural Probabilistic Language Model, Journal of Machine Learning Research, 3(Feb):1137-1155, 2003. ps
About auto-encoders
Hinton, G.E. and Zemel, R.S. Minimizing description length in an unsupervised neural network (technical report?) pdf
R. Hadsell, S. Chopra, Y. LeCun?, "Dimensionality Reduction by Learning an Invariant Mapping," in Proc. of Computer Vision and Pattern Recognition Conference (CVPR 2006), 2006. pdf
Learning about relations between symbols
Paccanaro, A., and Hinton, G.E. Learning Distributed Representations by Mapping Concepts and Relations into a Linear Space ICML-2000, Proceedings of the Seventeenth International Conference on Machine Learning, Langley P. (Ed.), 711-718, Stanford University, Morgan Kaufmann Publishers, San Francisco. pdf
2000 Paccanaro, A. and Hinton, G.E Extracting Distributed Representations of Concepts and Relations from Positive and Negative Propositions Proceedings of the International Joint Conference on Neural Networks, IJCNN 2000 pdf
2005 Memisevic, R. and Hinton, G. E. Multiple Relational Embedding. Advances in Neural Information Processing Systems, 17, MIT Press, Cambridge, MA pdf
About Monte-Carlo methods
MathWorld? entry on Monte-Carlo methods: www and Monte-Carlo integration: www
Nando de Freitas seminar on Monte-Carlo methods www
About graphical models
Graphical models: Probabilistic inference. M. I. Jordan and Y. Weiss. In M. Arbib (Ed.), The Handbook of Brain Theory and Neural Networks, 2nd edition. Cambridge, MA: MIT Press, 2002. ps.gz
Graphical models. M. I. Jordan. Statistical Science (Special Issue on Bayesian Statistics), 19, 140-155, 2004. ps
A Comparison of Algorithms for Inference and Learning in Probabilistic Graphical Models Brendan J. Frey, and Nebojsa Jojic IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 27, NO. 9, Sept. 2005 pdf
Hinton, G. E., Dayan, P., To, A. and Neal R. M. The Helmholtz machine through time. F. Fogelman-Soulie and R. Gallinari (editors) ICANN-95, 483-490 pdf
About Boltzmann machines and related energy-based models
One of the founding papers:
"Learning and Relearning in Boltzmann Machines" by G. E. Hinton and T. J. Sejnowski pdf
spiking Boltzmann machines paper with ideas to deal with temporal data: pdf
Yann LeCun? and Fu Jie Huang, "Loss Functions for Discriminative Training of Energy-Based Models," in Proc. of the 10-th International Workshop on Artificial Intelligence and Statistics (AIStats'05) , 2005. pdf
About Products of Experts, Restricted Boltzmann Machines and Contrastive Divergence
Hinton, G.E. Products of experts Proceedings of the Ninth International Conference on Artificial Neural Networks [ICANN 99 Vol 1 pages 1-6]. pdf
2000. Hinton, G.E. Training Products of Experts by Minimizing Contrastive Divergence Technical Report: GCNU TR 2000-004 pdf
Carreira-Perpignan, M. A. and Hinton. G. E. On Contrastive Divergence Learning. In: Artificial Intelligence and Statistics, 2005, Barbados pdf
Yee-Whye Teh, Geoffrey Hinton Rate-coded Restricted Boltzmann Machines for Face Recognition Advances in Neural Information Processing Systems 13, MIT Press, Cambridge, MA, 2001. pdf
Hinton, G. E. and Teh, Y. W. Discovering Multiple Constraints that are Frequently Approximately Satisfied. Proceedings of Uncertainty in Artificial Intelligence (UAI-2001), pp 227-234. pdf
2002 Welling, M. and Hinton, G. E. A New Learning Algorithm for Mean Field Boltzmann Machines. International Joint Conference on Neural Networks, Madrid. ps.gz
Hinton, G. E. (2002) Training Products of Experts by Minimizing Contrastive Divergence. Neural Computation, 14, pp 1771-1800. pdf
2004 Hinton, G. E., Welling, M. and Mnih, A. Wormholes Improve Contrastive Divergence. Advances in Neural Information Processing Systems, 16, MIT Press, Cambridge, MA pdf
Welling, M,, Rosen-Zvi, M. and Hinton, G. E. (2005) Exponential Family Harmoniums with an Application to Information Retrieval. Advances in Neural Information Processing Systems, 17, MIT Press, Cambridge, MA pdf
About deep belief networks as such
Bengio, Y. (2009) Learning deep Architectures for AI. To appear in Foundations and Trends in Machine Learning, 2009. www
This is a review paper on deep architectures and deep belief networks in general.
Early version: wake-sleep algorithm and Helmholtz machine
Hinton, G. E., Dayan, P., Frey, B. J. and Neal, R. The wake-sleep algorithm for unsupervised Neural Networks. Science, 268, 1158-1161, 1995. pdf
Frey, B. J., Hinton, G. E. and Dayan, P. Does the wake-sleep algorithm learn good density estimators? Advances in Neural Information Processing Systems 8. MIT Press, Cambridge, MA. pdf
Hinton, G. E. and Ghahramani, Z. Generative models for discovering sparse distributed representations Philosophical Transactions of the Royal Society of London, B, 352: 1177-1190, 1997. pdf
The first Deep Nets papers
2006 Hinton, G. E., Osindero, S. and Teh, Y. A fast learning algorithm for deep belief nets.(submitted to Neural Computation) ps.gzpdf
yet another version of the deep belief net paper, easier to read:
Hinton, G. E., Osindero, S., Welling, M. and Teh, Y. Unsupervised Discovery of Non-linear Structure using Contrastive Backpropagation. to appear in Cognitive Science, 30:4, 2006. pdf
One the use of DBN for training deep auto-encoders:
Hinton, G. E. and Salakhutdinov, R. R Reducing the dimensionality of data with neural networks. Science, Vol. 313. no. 5786, pp. 504 - 507, 28 July 2006. pdf
Papers at NIPS'2006:
Greedy Layer-Wise Training of Deep Networks Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle Advances in Neural Information Processing Systems 19, B. Scholkopf, J. Platt and T. Hoffman editors, MIT Press, Cambridge, MA, 2007. pdf
Modeling Human Motion Using Binary Latent Variables Graham Taylor, Geoff Hinton, Sam Roweis Advances in Neural Information Processing Systems 19, B. Scholkopf, J. Platt and T. Hoffman editors, MIT Press, Cambridge, MA, 2007. pdf
The recent Deep Nets papers
Yoshua Bengio, Learning deep architectures for AI, in: Foundations and Trends in Machine Learning, volume to appear, 2009. www
Joseph Turian , James Bergstra and Yoshua Bengio, Quadratic Features and Deep Architectures for Chunking, in: North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL HLT), 2009. www
Yoshua Bengio, Jerome Louradour, Ronan Collobert and Jason Weston, Curriculum Learning, number 1330, 2009. www
Yoshua Bengio and Olivier Delalleau, Justifying and Generalizing Contrastive Divergence, in: Neural Computation, volume 21, number 6, pages 1601-1621, 2009. www
Dumitru Erhan, Pierre-Antoine Manzagol , Yoshua Bengio, Samy Bengio and Pascal Vincent, The Difficulty of Training Deep Architectures and the effect of Unsupervised Pre-Training, in: Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS 2009), Clearwater (Florida), USA, 2009. www
Dumitru Erhan, Yoshua Bengio, Aaron Courville and Pascal Vincent, Visualizing Higher-Layer Features of a Deep Network, number 1341, 2009. www
Hugo Larochelle, Dumitru Erhan and Pascal Vincent, Deep Learning using Robust Interdependent Codes, in: Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS 2009), Clearwater (Florida), USA, 2009. www
Hugo Larochelle, Yoshua Bengio, Jerome Louradour and Pascal Lamblin, Exploring Strategies for Training Deep Neural Networks, in: The Journal of Machine Learning Research, pages 1-40, 2009. www
Guillaume Desjardins and Yoshua Bengio, Empirical Evaluation of Convolutional RBMs for Vision, number 1327, 2008. www
Hugo Larochelle and Yoshua Bengio, Classification using Discriminative Restricted Boltzmann Machines, in: International Conference on Machine Learning proceedings, 2008. www
Nicolas Le Roux and Yoshua Bengio, Representational Power of Restricted Boltzmann Machines and Deep Belief Networks, in: Neural Computation, volume 20, number 6, pages 1631-1649, 2008. www
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. pages 1096-1103, 2008. www