Topmoumoute Online Natural Gradient Algorithm

par/by Nicolas Le Roux
Département d'Informatique et de Recherche Opérationnelle
Université de Montréal



Many machine learning algorithms, such as neural networks, are trained using gradient descent. Natural gradient descent is a variant of the usual gradient descent which has been popularized by Amari. At the time, its justification was to be the direction of steepest descent in an underlying Riemannian space.

We first present a statistical approach which justifies the use of natural gradient descent for limiting overfitting. Then, we will develop an efficient approach of this gradient, making it practical for online learning.