Non-parametric learning algorithms

The term non-parametric has been used in different ways in the statistics and machine learning literature.

Here is the definition we prefer, because it allows including mixture models and neural networks when their size is data-selected.

We say that a learning algorithm is non-parametric if the complexity of the functions it can learn is allowed to grow as the amount of training data is increased.

The term non-parametric has been restricted in some publications to learning algorithms in which the learned function is expressed directly in terms of the training examples, e.g., the nearest-neighbor classifier or a Support Vector Machine.

A parametric learning algorithm involves a fixed family of functions, indexed by a fixed number of parameters (independent of the number of training examples), e.g. a linear regression, or a multivariate Gaussian distribution. The choice of that family of function reflects a prior belief that the function to be learned is either in that family or not far from it.

When we lack sufficient knowledge to make strong assumptions about the target function, it seems preferable to resort to non-parametric learning algorithms. Prior knowledge about the target function can also be included in learning algorithms, but as the amount of training data grows, these algorithms generally have the nice property of consistency (they converge to the target function).

Some 'parametric' models have a knob that controls the number of parameters, e.g. the number of components in a mixture model, or the number of hidden neurons in a neural network. When this knob is selected based on the data, e.g. by cross-validation or similar techniques, we consider such models to be non-parametric, and in fact consistency has been proven for them.

-- YoshuaBengio - 18 Aug 2008

This topic: Public > WebHome > ResearchVision > NonParametric
History: r2 - 18 Aug 2008 - 17:11:02 - YoshuaBengio
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback