Higher Order Contractive Auto-Encoder
| Type of publication: | Inproceedings |
| Citation: | Salah+al-2011 |
| Booktitle: | European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD) |
| Year: | 2011 |
| Abstract: | We propose a novel regularizer when training an auto-encoder for unsupervised feature extraction. We explicitly encourage the latent representation to contract the input space by regularizing the norm of the Jacobian (analytically) and the Hessian (stochastically) of the en- coder’s output with respect to its input, at the training points. While the penalty on the Jacobian’s norm ensures robustness to tiny corrup- tion of samples in the input space, constraining the norm of the Hes- sian extends this robustness when moving further away from the sample. From a manifold learning perspective, balancing this regularization with the auto-encoder’s reconstruction objective yields a representation that varies most when moving along the data manifold in input space, and is most insensitive in directions orthogonal to the manifold. The second order regularization, using the Hessian, penalizes curvature, and thus favors smooth manifold. We show that our proposed technique, while remaining computationally efficient, yields representations that are sig- nificantly better suited for initializing deep architectures than previously proposed approaches, beating state-of-the-art performance on a number of datasets. |
| Keywords: | |
| Authors | |
| Added by: | [ADM] |
| Total mark: | 0 |
|
Attachments
|
|
|
Notes
|
|
|
|
|
|
Topics
|
|
|
|
|
