Conditional Density Estimation with Extreme Observations

Julie Carreau and Yoshua Bengio

My project is to develop methods of density estimation based on subexponential distributions such as those found in the extreme value theory. The goal is to model an unknown, asymmetric, fat-tailed density (eventually a conditional density or the noise in a regression setting). Current methods in robust statistics downweight the observations in the tails symmetrically and are therefore biased in the case of asymmetry.

In order to make as few distributional assumptions as possible, I would like to use a non-parametric method. However, the estimation of the density becomes more and more difficult as we try to model further in the tail of the density. A compromise would be to use a non-parametric method to model the center of the density and a parametric method to model the tail of the density, above a given threshold.

The estimation of the distribution in the upper tail could be based on the following theory. Distribution with Pareto-type tails are defined by 1 - F(x) = x^{-alpha} L(x), where L(x) is a slowly varying function and alpha controls the thickness of the tail. In order to estimate alpha, one can use the technics from extreme value theory (EVT).

The EVT is based on the modelling of maxima and minima behaviour. The limit laws for maxima (minima) of random variables are one of three types: Frechet, Gumbel or Weibull. The estimation of alpha is based on sample maxima. Another possibility is the generalised Pareto distribution (GPD) which models excesses over a given threshold.

The difficulty with the GPD is to set an appropriate threshold. One possibility is to use order statistics. In the conditional setting, I propose to use regression quantile. The level of quantile is chosen through cross-validation. The GPD allows then to model parametrically the tails of the density.