r4 - 05 Feb 2009 - 10:24:03 - FredericBastienYou are here: TWiki > Public Web > NeuralNetsIntroductoryMaterial

A neural network tutorial (with no math):
http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html

A slightly more formal introduction: http://ieee.uow.edu.au/~daniel/software/libneural/BPN_tutorial/BPN_English/BPN_English/BPN_English.html

The original article on back-propagation. http://www.iro.umontreal.ca/~pift6266/A06/refs/backprop_old.pdf

Hinton, G.E. Preface to the special issue on connectionist symbol processing, Artificial Intelligence 46, 1-4 http://www.cs.toronto.edu/~hinton/absps/connectionist.pdf

How backprop for feed-forward and recurrent nets can be derived cleanly from a Lagrangian formalism:

Y. LeCun^{?}, "A theoretical framework for Back-Propagation," in Proceedings of the 1988 Connectionist Models Summer School, (D. Touretzky, G. Hinton, and T. Sejnowski, eds.), (CMU, Pittsburgh, Pa), pp. 21-28, 1988.
http://yann.lecun.com/exdb/publis/pdf/lecun-88.pdf

The bias-variance dilemma for neural networks, by Geman et al, Neural Computation 1991 paper. http://www.iro.umontreal.ca/~pift6266/A06/refs/neural_nets_bias_variance.pdf Explores the issue of overfitting and underfitting with non-parametric models in general.

Michael Jordan's encyclopedic paper on neural networks, graphical models... Neural networks. M. I. Jordan and C. Bishop. In Tucker, A. B. (Ed.), CRC Handbook of Computer Science, Boca Raton, FL: CRC Press, 1997. http://www.cs.berkeley.edu/~jordan/papers/crc.ps.Z

Lecture notes from Rich Zemel's classes on machine learning and datamining: http://www.cs.toronto.edu/~zemel/Courses/CS411/lects.html

Lecture notes from Geoffrey Hinton's classes on neural networks: http://www.cs.toronto.edu/~hinton/csc2535/lectures.html http://www.cs.toronto.edu/~hinton/csc321/lectures.html

Lecture notes from Sam Roweis undergraduate class in machine learning: http://www.cs.toronto.edu/~roweis/csc2515/lectures.html

Statistical data-mining tutorials:
http://www.autonlab.org/tutorials/

I suggest the following: probability for data miners, probability density functions, Gaussians,
maximum likelihood estimation, cross-validation, neural networks, AI class introduction

If you really want to implement multi-layer neural networks back-prop, this paper is for you:
all the tricks and the theory behind them to efficiently train neural networks with backpropagation, including how to compute the optimal learning rate, how to back-propagate second derivatives, and other sundries.

Y. LeCun^{?}, L. Bottou, G. Orr, and K. Muller, "Efficient BackProp^{?}," in Neural Networks: Tricks of the trade, (G. Orr and Muller K., eds.), 1998.
http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf

Y. Le Cun and Y. Bengio. Pattern Recognition and Neural Networks. In M. A. Arbib, editor, The Handbook of Brain Theory and Neural Networks, pages 711-714. MIT Press, 1995. http://www.iro.umontreal.ca/%7Elisa/pointeurs/handbook-patrec.ps

Dietterich, T. G. (2002). Machine Learning for Sequential Data: A Review. In T. Caelli (Ed.) Lecture Notes in Computer Science. Springer-Verlag. http://www.cs.orst.edu/~tgd/publications/mlsd-ssspr.pdf

Neal, R. M. (1990). Learning stochastic feedforward networks. Technical Report CRG-TR-90-7, Dept. of Computer Science, University of Toronto http://www.cs.toronto.edu/~hinton/absps/sff.pdf

Hinton, G. E., Dayan, P., Frey, B. J. and Neal, R. The wake-sleep algorithm for unsupervised Neural Networks. Science, 268, 1158-1161. http://www.cs.toronto.edu/~hinton/absps/ws.pdf

Y. Le Cun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998. http://www.iro.umontreal.ca/~lisa/pointeurs/lecun-98.ps.gz

-- YoshuaBengio - 30 May 2006

Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r4 < r3 < r2 < r1 | More topic actions

Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.

Ideas, requests, problems regarding TWiki? Send feedback

Ideas, requests, problems regarding TWiki? Send feedback