Here are some images of the filters (weights) learned by the neurons of the first hidden layer of different deep networks. These features were learned from 10000 MNIST digit images, randomly drawn from the original MNIST with replacement (don't ask why ).

These images where obtained using Matlab's image function. Given a matrix `M`

where each row corresponds to the input weights of a neuron, I used the following command to obtain the following images:

i=begin; for j=1:128; subplot(8,16,j); axis off; box on; image(reshape((M(i+j,:)+a)/(2*a)*64,d1,d2)'); colormap('gray');axis off; box on; end;

which takes the neurons at position `begin + 1`

up to `begin + 128`

and plots their input weights. Black means a value of the weight that is smaller or equal to `-a`

, white
means a weight that is greater or equal to `a`

, and gray means a weight 0. The weights are drawn in `d1`

by `d2`

pixel images (for the MNIST images, `d1`

and `d2`

are 28).

Some test errors are also reported in the following sections, but should be taken with a grain of salt. Indeed, very few hyper-parameters were tested for the different architectures. Also the test error was evaluated on only 2000 samples drawn from the MNIST data and, again, the sampling was done with replacements. This makes the estimated error optimistic, since there are some samples in the test set that are also present in the training set (again, don't ask why this was done...).

Here, `a`

is equal to 3, as usually used by Hinton. The DBN had 500 units in the first and second hidden layer, and 2000 units in the third layer. It obtains 1.8% classification error on a 2000 samples drawn from the MNIST data (again, with replacement, so this error is optimistic):

For a `eps`

format of this figure: dbn_hid1_subset_corrected.eps

Here, `a`

is equal to 21. This value was chosen because it is such that . The DBN had 500 units in the first and second hidden layer, and 2000 units in the third layer. It obtains 2.25% on the 2000 samples test set used in the previous section:

For a `eps`

format of this figure: dbn_hid1_trunc_exp_subset_corrected.eps

Here is the same figure, wiht `a`

equal to 3:

For a `eps`

format of this figure: dbn_hid1_trunc_exp_subset_scale2_corrected.eps

Here, `a`

is equal to 23. This value was chosen because it roughly corresponds to the maximum absolute value over all weights. The DBN had 500 units in the first and second hidden layer, and 2000 units in the third layer. It obtains 2.5% on the 2000 samples test set used in the previous sections:

For a `eps`

format of this figure: dbn_hid1_gauss_subset_corrected.eps

Here is the same figure, wiht `a`

equal to 3:

For a `eps`

format of this figure: dbn_hid1_gauss_subset_scale2_corrected.eps

Here, `a`

is equal to 3. The SAA had 700 units in the first hidden layer, 1000 in the second and 2000 in the third. It obtains 2.2% on the 2000 samples test set used in the previous sections:

For a `eps`

format of this figure: saa_hid1_lnd_0_subset_corrected.eps

Here, `a`

is equal to 3. The SAA had 1500 units in the first and second hidden layer, which correspond to 100 concatenated multinomial units of size 15. The third layer corresponds to 2000 binomial units. It obtains 2.8% on the 2000 samples test set used in the previous sections:

For a `eps`

format of this figure: saa_hid1_softmax_15_subset_corrected.eps

Here, `a`

is equal to 3. The neural net had 500 units in the first and second hidden layer and 2000 units in the third layer. The hidden activation functions are sigmoid functions,
just like in a DBN or SAA. A large learning rate (0.1) has been used so that the gradient can propagate well to the first hidden layer. It obtains 3.95% on the 2000 samples test set used in the previous sections:

For a `eps`

format of this figure: dbn_hid1_no_greedy_3lay_subset_corrected.eps

-- HugoLarochelle - 18 June 2007

Show attachmentsHide attachments

Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.

Ideas, requests, problems regarding TWiki? Send feedback

Ideas, requests, problems regarding TWiki? Send feedback