Neuron Filters Gallery
Here are some images of the filters (weights) learned by the neurons of the first hidden layer of different deep networks. These features were learned from 10000 MNIST digit images, randomly drawn from the original MNIST with replacement (don't ask why

).
These images where obtained using Matlab's image function. Given a matrix
M where each row corresponds to the input weights of a neuron, I used the following command to obtain the following images:
i=begin; for j=1:128; subplot(8,16,j); axis off; box on; image(reshape((M(i+j,:)+a)/(2*a)*64,d1,d2)'); colormap('gray');axis off; box on; end;
which takes the neurons at position
begin + 1 up to
begin + 128 and plots their input weights. Black means a value of the weight that is smaller or equal to
-a, white
means a weight that is greater or equal to
a, and gray means a weight 0. The weights are drawn in
d1 by
d2 pixel images (for the MNIST images,
d1 and
d2 are 28).
Some test errors are also reported in the following sections, but should be taken with a grain of salt. Indeed, very few hyper-parameters were tested for the different architectures. Also the test error was evaluated on only 2000 samples drawn from the MNIST data and, again, the sampling was done with replacements. This makes the estimated error optimistic, since there are some samples in the test set that are also present in the training set (again, don't ask why this was done...).
DBN with binomial input units
Here,
a is equal to 3, as usually used by Hinton. The DBN had 500 units in the first and second hidden layer, and 2000 units in the third layer. It obtains 1.8% classification error on a 2000 samples drawn from the MNIST data (again, with replacement, so this error is optimistic):
For a
eps format of this figure:
dbn_hid1_subset_corrected.eps
DBN with truncated exponential units
Here,
a is equal to 21. This value was chosen because it is such that

. The DBN had 500 units in the first and second hidden layer, and 2000 units in the third layer. It obtains 2.25% on the 2000 samples test set used in the previous section:
For a
eps format of this figure:
dbn_hid1_trunc_exp_subset_corrected.eps
Here is the same figure, wiht
a equal to 3:
For a
eps format of this figure:
dbn_hid1_trunc_exp_subset_scale2_corrected.eps
DBN with Gaussian units
Here,
a is equal to 23. This value was chosen because it roughly corresponds to the maximum absolute value over all weights. The DBN had 500 units in the first and second hidden layer, and 2000 units in the third layer. It obtains 2.5% on the 2000 samples test set used in the previous sections:
For a
eps format of this figure:
dbn_hid1_gauss_subset_corrected.eps
Here is the same figure, wiht
a equal to 3:
For a
eps format of this figure:
dbn_hid1_gauss_subset_scale2_corrected.eps
Stacked Autoassociator with binomial units
Here,
a is equal to 3. The SAA had 700 units in the first hidden layer, 1000 in the second and 2000 in the third. It obtains 2.2% on the 2000 samples test set used in the previous sections:
For a
eps format of this figure:
saa_hid1_lnd_0_subset_corrected.eps
Stacked Autoassociator with multinomial (softmax) units
Here,
a is equal to 3. The SAA had 1500 units in the first and second hidden layer, which correspond to 100 concatenated multinomial units of size 15. The third layer corresponds to 2000 binomial units. It obtains 2.8% on the 2000 samples test set used in the previous sections:
For a
eps format of this figure:
saa_hid1_softmax_15_subset_corrected.eps
Standard Neural Network with 3 hidden layers
Here,
a is equal to 3. The neural net had 500 units in the first and second hidden layer and 2000 units in the third layer. The hidden activation functions are sigmoid functions,
just like in a DBN or SAA. A large learning rate (0.1) has been used so that the gradient can propagate well to the first hidden layer. It obtains 3.95% on the 2000 samples test set used in the previous sections:
For a
eps format of this figure:
dbn_hid1_no_greedy_3lay_subset_corrected.eps
--
HugoLarochelle - 18 June 2007