r24 - 30 Sep 2009 - 18:12:21 - MichaelMandelYou are here: TWiki >  Public Web  > DeepNetworks > ReadingOnDeepNetworks

Preliminary stuff that can be useful, depending on your background

Another bibliography: DeepNetworksBibliography.

About neural networks

See the page on neural nets introductory material: NeuralNetsIntroductoryMaterial

Also see the page on recurrent neural nets: RecurrentNeuralNetBibliography

About distributed representations

About learning distributed representations for words

  • Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Jauvin. A Neural Probabilistic Language Model, Journal of Machine Learning Research, 3(Feb):1137-1155, 2003. ps

About auto-encoders

  • Hinton, G.E. and Zemel, R.S. Minimizing description length in an unsupervised neural network (technical report?) pdf
  • R. Hadsell, S. Chopra, Y. LeCun?, "Dimensionality Reduction by Learning an Invariant Mapping," in Proc. of Computer Vision and Pattern Recognition Conference (CVPR 2006), 2006. pdf

Learning about relations between symbols

  • Paccanaro, A., and Hinton, G.E. Learning Distributed Representations by Mapping Concepts and Relations into a Linear Space ICML-2000, Proceedings of the Seventeenth International Conference on Machine Learning, Langley P. (Ed.), 711-718, Stanford University, Morgan Kaufmann Publishers, San Francisco. pdf
  • 2000 Paccanaro, A. and Hinton, G.E Extracting Distributed Representations of Concepts and Relations from Positive and Negative Propositions Proceedings of the International Joint Conference on Neural Networks, IJCNN 2000 pdf
  • 2005 Memisevic, R. and Hinton, G. E. Multiple Relational Embedding. Advances in Neural Information Processing Systems, 17, MIT Press, Cambridge, MA pdf

About Monte-Carlo methods

  • MathWorld? entry on Monte-Carlo methods: www and Monte-Carlo integration: www
  • Nando de Freitas seminar on Monte-Carlo methods www

About graphical models

  • Graphical models: Probabilistic inference. M. I. Jordan and Y. Weiss. In M. Arbib (Ed.), The Handbook of Brain Theory and Neural Networks, 2nd edition. Cambridge, MA: MIT Press, 2002. ps.gz
  • Graphical models. M. I. Jordan. Statistical Science (Special Issue on Bayesian Statistics), 19, 140-155, 2004. ps
  • A Comparison of Algorithms for Inference and Learning in Probabilistic Graphical Models Brendan J. Frey, and Nebojsa Jojic IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 27, NO. 9, Sept. 2005 pdf
  • Hinton, G. E., Dayan, P., To, A. and Neal R. M. The Helmholtz machine through time. F. Fogelman-Soulie and R. Gallinari (editors) ICANN-95, 483-490 pdf

About Boltzmann machines and related energy-based models

One of the founding papers:

  • "Learning and Relearning in Boltzmann Machines" by G. E. Hinton and T. J. Sejnowski pdf
  • spiking Boltzmann machines paper with ideas to deal with temporal data: pdf
  • Yann LeCun? and Fu Jie Huang, "Loss Functions for Discriminative Training of Energy-Based Models," in Proc. of the 10-th International Workshop on Artificial Intelligence and Statistics (AIStats'05) , 2005. pdf

About Products of Experts, Restricted Boltzmann Machines and Contrastive Divergence

  • Hinton, G.E. Products of experts Proceedings of the Ninth International Conference on Artificial Neural Networks [ICANN 99 Vol 1 pages 1-6]. pdf
  • 2000. Hinton, G.E. Training Products of Experts by Minimizing Contrastive Divergence Technical Report: GCNU TR 2000-004 pdf
  • Carreira-Perpignan, M. A. and Hinton. G. E. On Contrastive Divergence Learning. In: Artificial Intelligence and Statistics, 2005, Barbados pdf
  • Yee-Whye Teh, Geoffrey Hinton Rate-coded Restricted Boltzmann Machines for Face Recognition Advances in Neural Information Processing Systems 13, MIT Press, Cambridge, MA, 2001. pdf
  • Hinton, G. E. and Teh, Y. W. Discovering Multiple Constraints that are Frequently Approximately Satisfied. Proceedings of Uncertainty in Artificial Intelligence (UAI-2001), pp 227-234. pdf
  • 2002 Welling, M. and Hinton, G. E. A New Learning Algorithm for Mean Field Boltzmann Machines. International Joint Conference on Neural Networks, Madrid. ps.gz
  • Hinton, G. E. (2002) Training Products of Experts by Minimizing Contrastive Divergence. Neural Computation, 14, pp 1771-1800. pdf
  • 2004 Hinton, G. E., Welling, M. and Mnih, A. Wormholes Improve Contrastive Divergence. Advances in Neural Information Processing Systems, 16, MIT Press, Cambridge, MA pdf
  • Welling, M,, Rosen-Zvi, M. and Hinton, G. E. (2005) Exponential Family Harmoniums with an Application to Information Retrieval. Advances in Neural Information Processing Systems, 17, MIT Press, Cambridge, MA pdf

About deep belief networks as such

  • Bengio, Y. (2009) Learning deep Architectures for AI. To appear in Foundations and Trends in Machine Learning, 2009. www

This is a review paper on deep architectures and deep belief networks in general.

Early version: wake-sleep algorithm and Helmholtz machine

  • Hinton, G. E., Dayan, P., Frey, B. J. and Neal, R. The wake-sleep algorithm for unsupervised Neural Networks. Science, 268, 1158-1161, 1995. pdf
  • Frey, B. J., Hinton, G. E. and Dayan, P. Does the wake-sleep algorithm learn good density estimators? Advances in Neural Information Processing Systems 8. MIT Press, Cambridge, MA. pdf
  • Hinton, G. E. and Ghahramani, Z. Generative models for discovering sparse distributed representations Philosophical Transactions of the Royal Society of London, B, 352: 1177-1190, 1997. pdf

The first Deep Nets papers

  • 2006 Hinton, G. E., Osindero, S. and Teh, Y. A fast learning algorithm for deep belief nets.(submitted to Neural Computation) ps.gz pdf

yet another version of the deep belief net paper, easier to read:

  • Hinton, G. E., Osindero, S., Welling, M. and Teh, Y. Unsupervised Discovery of Non-linear Structure using Contrastive Backpropagation. to appear in Cognitive Science, 30:4, 2006. pdf

One the use of DBN for training deep auto-encoders:

  • Hinton, G. E. and Salakhutdinov, R. R Reducing the dimensionality of data with neural networks. Science, Vol. 313. no. 5786, pp. 504 - 507, 28 July 2006. pdf

Papers at NIPS'2006:

  • Greedy Layer-Wise Training of Deep Networks Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle Advances in Neural Information Processing Systems 19, B. Scholkopf, J. Platt and T. Hoffman editors, MIT Press, Cambridge, MA, 2007. pdf
  • Modeling Human Motion Using Binary Latent Variables Graham Taylor, Geoff Hinton, Sam Roweis Advances in Neural Information Processing Systems 19, B. Scholkopf, J. Platt and T. Hoffman editors, MIT Press, Cambridge, MA, 2007. pdf

The recent Deep Nets papers

  • Yoshua Bengio, Learning deep architectures for AI, in: Foundations and Trends in Machine Learning, volume to appear, 2009. www

  • Joseph Turian , James Bergstra and Yoshua Bengio, Quadratic Features and Deep Architectures for Chunking, in: North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL HLT), 2009. www
  • Yoshua Bengio, Jerome Louradour, Ronan Collobert and Jason Weston, Curriculum Learning, number 1330, 2009. www
  • Yoshua Bengio and Olivier Delalleau, Justifying and Generalizing Contrastive Divergence, in: Neural Computation, volume 21, number 6, pages 1601-1621, 2009. www
  • Dumitru Erhan, Pierre-Antoine Manzagol , Yoshua Bengio, Samy Bengio and Pascal Vincent, The Difficulty of Training Deep Architectures and the effect of Unsupervised Pre-Training, in: Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS 2009), Clearwater (Florida), USA, 2009. www
  • Dumitru Erhan, Yoshua Bengio, Aaron Courville and Pascal Vincent, Visualizing Higher-Layer Features of a Deep Network, number 1341, 2009. www
  • Hugo Larochelle, Dumitru Erhan and Pascal Vincent, Deep Learning using Robust Interdependent Codes, in: Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS 2009), Clearwater (Florida), USA, 2009. www
  • Hugo Larochelle, Yoshua Bengio, Jerome Louradour and Pascal Lamblin, Exploring Strategies for Training Deep Neural Networks, in: The Journal of Machine Learning Research, pages 1-40, 2009. www
  • Guillaume Desjardins and Yoshua Bengio, Empirical Evaluation of Convolutional RBMs for Vision, number 1327, 2008. www
  • Hugo Larochelle and Yoshua Bengio, Classification using Discriminative Restricted Boltzmann Machines, in: International Conference on Machine Learning proceedings, 2008. www
  • Nicolas Le Roux and Yoshua Bengio, Representational Power of Restricted Boltzmann Machines and Deep Belief Networks, in: Neural Computation, volume 20, number 6, pages 1631-1649, 2008. www
  • Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. pages 1096-1103, 2008. www
Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r24 < r23 < r22 < r21 < r20 | More topic actions
Public.ReadingOnDeepNetworks moved from Neurones.ReadingOnDeepNetworks on 05 Feb 2009 - 14:32 by FredericBastien - put it back
 
Home
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback