Topmoumoute Online Natural Gradient Algorithm
par/by Nicolas Le Roux
Département d'Informatique et de Recherche Opérationnelle
Université de Montréal
Many machine learning algorithms, such as neural networks, are trained
using gradient descent. Natural gradient descent is a variant of the
usual gradient descent which has been popularized by Amari. At the time,
its justification was to be the direction of steepest descent in an
underlying Riemannian space.
We first present a statistical approach which justifies the use of
natural gradient descent for limiting overfitting. Then, we will develop
an efficient approach of this gradient, making it practical for online
learning.