PLearn 0.1
|
Deep Belief Net where the stacked RBMs also use a discriminative criteria. More...
#include <DiscriminativeDeepBeliefNet.h>
Public Member Functions | |
DiscriminativeDeepBeliefNet () | |
Default constructor. | |
virtual int | outputsize () const |
Returns the size of this learner's output, (which typically may depend on its inputsize(), targetsize() and set options). | |
virtual void | forget () |
(Re-)initializes the PLearner in its fresh state (that state may depend on the 'seed' option) and sets 'stage' back to 0 (this is the stage of a fresh learner!). | |
virtual void | train () |
The role of the train method is to bring the learner up to stage==nstages, updating the train_stats collector with training costs measured on-line in the process. | |
virtual void | computeOutput (const Vec &input, Vec &output) const |
Computes the output from the input. | |
virtual void | computeCostsFromOutputs (const Vec &input, const Vec &output, const Vec &target, Vec &costs) const |
Computes the costs from already computed output. | |
virtual TVec< std::string > | getTestCostNames () const |
Returns the names of the costs computed by computeCostsFromOutpus (and thus the test method). | |
virtual TVec< std::string > | getTrainCostNames () const |
Returns the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats. | |
virtual void | setTrainingSet (VMat training_set, bool call_forget=true) |
Declares the training set. | |
void | updateNearestNeighbors () |
void | greedyStep (const Vec &input, const Vec &target, int index, Vec train_costs, int stage, Vec dissimilar_example) |
void | fineTuningStep (const Vec &input, const Vec &target, Vec &train_costs) |
void | computeRepresentation (const Vec &input, Vec &representation, int layer) const |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual DiscriminativeDeepBeliefNet * | deepCopy (CopiesMap &copies) const |
virtual void | build () |
Finish building the object; just call inherited::build followed by build_() | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transforms a shallow copy into a deep copy. | |
Static Public Member Functions | |
static string | _classname_ () |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
real | cd_learning_rate |
Contrastive divergence learning rate. | |
real | cd_decrease_ct |
Contrastive divergence decrease constant. | |
real | fine_tuning_learning_rate |
The learning rate used during the fine tuning gradient descent. | |
real | fine_tuning_decrease_ct |
The decrease constant of the learning rate used during fine tuning gradient descent. | |
TVec< int > | training_schedule |
Number of examples to use during each phase of greedy pre-training. | |
TVec< PP< RBMLayer > > | layers |
The layers of units in the network. | |
TVec< PP< RBMConnection > > | connections |
The weights of the connections between the layers. | |
TVec< PP< RBMLayer > > | unsupervised_layers |
Additional units for greedy unsupervised learning. | |
TVec< PP< RBMConnection > > | unsupervised_connections |
Additional connections for greedy unsupervised learning. | |
int | k_neighbors |
Number of good nearest neighbors to attract and bad nearest neighbors to repel. | |
int | n_classes |
Number of classes. | |
real | discriminative_criteria_weight |
Weight of the discriminative criteria. | |
real | output_weights_l1_penalty_factor |
Output weights l1_penalty_factor. | |
real | output_weights_l2_penalty_factor |
Output weights l2_penalty_factor. | |
bool | compare_joint_in_discriminative_criteria |
Indication that the discriminative criteria should use the joint over the input and the hidden units, instead of the conditional over the hidden units given the input units. | |
bool | do_not_use_generative_criteria |
Indication that the generative criteria should not be used during learning (does not work with compare_joint_in_discriminative_criteria = true). | |
int | n_layers |
Number of layers. | |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declares the class options. | |
Protected Attributes | |
bool | nearest_neighbors_are_up_to_date |
Indication that nearest_neighbors_indices is up to date. | |
TVec< Vec > | activations |
Stores the activations of the input and hidden layers (at the input of the layers) | |
TVec< Vec > | expectations |
Stores the expectations of the input and hidden layers (at the output of the layers) | |
TVec< Vec > | activation_gradients |
Stores the gradient of the cost wrt the activations of the input and hidden layers (at the input of the layers) | |
TVec< Vec > | expectation_gradients |
Stores the gradient of the cost wrt the expectations of the input and hidden layers (at the output of the layers) | |
TVec< PP< RBMLayer > > | greedy_layers |
Layers used for greedy learning. | |
TVec< PP< RBMConnection > > | greedy_connections |
Connections used for greedy learning. | |
Vec | dissimilar_example_representation |
Dissimilar example representation. | |
Vec | input_representation |
Example representation. | |
Vec | pos_down_val |
Positive down statistic. | |
Vec | pos_up_val |
Positive up statistic. | |
Vec | neg_down_val |
Negative down statistic. | |
Vec | neg_up_val |
Negative up statistic. | |
Vec | disc_pos_down_val1 |
First discriminative positive down statistic. | |
Vec | disc_pos_up_val1 |
First discriminative positive up statistic. | |
Vec | disc_pos_down_val2 |
Second discriminative positive down statistic. | |
Vec | disc_pos_up_val2 |
Second discriminative positive up statistic. | |
Vec | disc_neg_down_val |
Negative down statistic. | |
Vec | disc_neg_up_val |
Negative up statistic. | |
Vec | final_cost_input |
Input of cost function. | |
Vec | final_cost_value |
Cost value. | |
Vec | final_cost_gradient |
Cost gradient on output layer. | |
TVec< PP< ClassSubsetVMatrix > > | other_class_datasets |
Datasets for each class. | |
TMat< int > | nearest_neighbors_indices |
Nearest neighbors for each training example. | |
TVec< int > | greedy_stages |
Stages of the different greedy phases. | |
int | currently_trained_layer |
Currently trained layer (1 means the first hidden layer, n_layers means the output layer) | |
PP< OnlineLearningModule > | final_module |
Output layer of neural net. | |
PP< CostModule > | final_cost |
Cost on output layer of neural net. | |
Private Types | |
typedef PLearner | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. | |
void | build_layers_and_connections () |
void | build_output_layer_and_cost () |
void | setLearningRate (real the_learning_rate) |
Deep Belief Net where the stacked RBMs also use a discriminative criteria.
Definition at line 64 of file DiscriminativeDeepBeliefNet.h.
typedef PLearner PLearn::DiscriminativeDeepBeliefNet::inherited [private] |
Reimplemented from PLearn::PLearner.
Definition at line 66 of file DiscriminativeDeepBeliefNet.h.
PLearn::DiscriminativeDeepBeliefNet::DiscriminativeDeepBeliefNet | ( | ) |
Default constructor.
Definition at line 59 of file DiscriminativeDeepBeliefNet.cc.
References PLearn::PLearner::nstages, and PLearn::PLearner::random_gen.
: cd_learning_rate( 0. ), cd_decrease_ct( 0. ), fine_tuning_learning_rate( 0. ), fine_tuning_decrease_ct( 0. ), k_neighbors( 1 ), n_classes( -1 ), discriminative_criteria_weight( 0. ), output_weights_l1_penalty_factor(0), output_weights_l2_penalty_factor(0), compare_joint_in_discriminative_criteria( false ), do_not_use_generative_criteria( false ), // cancel_normalization_terms( false ), n_layers( 0 ), nearest_neighbors_are_up_to_date( false ), currently_trained_layer( 0 ) { // random_gen will be initialized in PLearner::build_() random_gen = new PRandom(); nstages = 0; }
string PLearn::DiscriminativeDeepBeliefNet::_classname_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 57 of file DiscriminativeDeepBeliefNet.cc.
OptionList & PLearn::DiscriminativeDeepBeliefNet::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 57 of file DiscriminativeDeepBeliefNet.cc.
RemoteMethodMap & PLearn::DiscriminativeDeepBeliefNet::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 57 of file DiscriminativeDeepBeliefNet.cc.
Reimplemented from PLearn::PLearner.
Definition at line 57 of file DiscriminativeDeepBeliefNet.cc.
Object * PLearn::DiscriminativeDeepBeliefNet::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 57 of file DiscriminativeDeepBeliefNet.cc.
StaticInitializer DiscriminativeDeepBeliefNet::_static_initializer_ & PLearn::DiscriminativeDeepBeliefNet::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 57 of file DiscriminativeDeepBeliefNet.cc.
void PLearn::DiscriminativeDeepBeliefNet::build | ( | ) | [virtual] |
Finish building the object; just call inherited::build followed by build_()
Reimplemented from PLearn::PLearner.
Definition at line 470 of file DiscriminativeDeepBeliefNet.cc.
References PLearn::PLearner::build(), and build_().
{ inherited::build(); build_(); }
void PLearn::DiscriminativeDeepBeliefNet::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::PLearner.
Definition at line 213 of file DiscriminativeDeepBeliefNet.cc.
References build_layers_and_connections(), build_output_layer_and_cost(), PLearn::TVec< T >::clear(), compare_joint_in_discriminative_criteria, currently_trained_layer, do_not_use_generative_criteria, PLearn::endl(), final_cost, final_module, greedy_stages, PLearn::PLearner::inputsize_, k_neighbors, layers, PLearn::TVec< T >::length(), n_classes, n_layers, PLERROR, PLearn::TVec< T >::resize(), PLearn::PLearner::stage, PLearn::PLearner::targetsize_, training_schedule, and PLearn::PLearner::weightsize_.
Referenced by build().
{ // ### This method should do the real building of the object, // ### according to set 'options', in *any* situation. // ### Typical situations include: // ### - Initial building of an object from a few user-specified options // ### - Building of a "reloaded" object: i.e. from the complete set of // ### all serialised options. // ### - Updating or "re-building" of an object after a few "tuning" // ### options have been modified. // ### You should assume that the parent class' build_() has already been // ### called. MODULE_LOG << "build_() called" << endl; if(inputsize_ > 0 && targetsize_ > 0) { // Initialize some learnt variables n_layers = layers.length(); if( n_classes <= 0 ) PLERROR("DiscriminativeDeepBeliefNet::build_() - \n" "n_classes should be > 0.\n"); if( k_neighbors <= 0 ) PLERROR("DiscriminativeDeepBeliefNet::build_() - \n" "k_neighbors should be > 0.\n"); if( weightsize_ > 0 ) PLERROR("DiscriminativeDeepBeliefNet::build_() - \n" "usage of weighted samples (weight size > 0) is not\n" "implemented yet.\n"); if( training_schedule.length() != n_layers-1 ) PLERROR("DiscriminativeDeepBeliefNet::build_() - \n" "training_schedule should have %d elements.\n", n_layers-1); if( compare_joint_in_discriminative_criteria && do_not_use_generative_criteria) PLERROR("DiscriminativeDeepBeliefNet::build_() - \n" "compare_joint_in_discriminative_criteria can't be used with\n" "do_not_use_generative_criteria.\n"); // if( (!compare_joint_in_discriminative_criteria || do_not_use_generative_criteria) // && cancel_normalization_terms ) // PLERROR("DiscriminativeDeepBeliefNet::build_() - \n" // "cancel_normalization_terms should be used with\n" // "compare_joint_in_discriminative_criteria and \n" // "do_not_use_generative_criteria without .\n"); if(greedy_stages.length() == 0) { greedy_stages.resize(n_layers-1); greedy_stages.clear(); } if(stage > 0) currently_trained_layer = n_layers; else { currently_trained_layer = n_layers-1; while(currently_trained_layer>1 && greedy_stages[currently_trained_layer-1] <= 0) currently_trained_layer--; } build_layers_and_connections(); if( !final_module || !final_cost ) build_output_layer_and_cost(); } }
void PLearn::DiscriminativeDeepBeliefNet::build_layers_and_connections | ( | ) | [private] |
Definition at line 334 of file DiscriminativeDeepBeliefNet.cc.
References activation_gradients, activations, connections, PLearn::endl(), expectation_gradients, expectations, greedy_connections, greedy_layers, i, PLearn::PLearner::inputsize_, layers, PLearn::TVec< T >::length(), n_layers, PLERROR, PLearn::PLearner::random_gen, PLearn::TVec< T >::resize(), PLearn::TVec< T >::size(), unsupervised_connections, and unsupervised_layers.
Referenced by build_().
{ MODULE_LOG << "build_layers_and_connections() called" << endl; if( connections.length() != n_layers-1 ) PLERROR("DiscriminativeDeepBeliefNet::build_layers_and_connections() - \n" "there should be %d connections.\n", n_layers-1); if(unsupervised_layers.length() != n_layers-1 && unsupervised_layers.length() != 0) PLERROR("DiscriminativeDeepBeliefNet::build_layers_and_connections() - \n" "there should be either 0 of %d unsupervised_layers.\n", n_layers-1); if(unsupervised_connections.length() != n_layers-1 && unsupervised_connections.length() != 0) PLERROR("DiscriminativeDeepBeliefNet::build_layers_and_connections() - \n" "there should be either 0 of %d unsupervised_connections.\n", n_layers-1); if(unsupervised_connections.length() != unsupervised_layers.length()) PLERROR("DiscriminativeDeepBeliefNet::build_layers_and_connections() - \n" "there should be as many unsupervised_connections and " "unsupervised_layers.\n"); if(layers[0]->size != inputsize_) PLERROR("DiscriminativeDeepBeliefNet::build_layers_and_connections() - \n" "layers[0] should have a size of %d.\n", inputsize_); activations.resize( n_layers ); expectations.resize( n_layers ); activation_gradients.resize( n_layers ); expectation_gradients.resize( n_layers ); greedy_layers.resize(n_layers-1); greedy_connections.resize(n_layers-1); for( int i=0 ; i<n_layers-1 ; i++ ) { if( layers[i]->size != connections[i]->down_size ) PLERROR("DiscriminativeDeepBeliefNet::build_layers_and_connections() " "- \n" "connections[%i] should have a down_size of %d.\n", i, layers[i]->size); if( connections[i]->up_size != layers[i+1]->size ) PLERROR("DiscriminativeDeepBeliefNet::build_layers_and_connections() " "- \n" "connections[%i] should have a up_size of %d.\n", i, layers[i+1]->size); if(unsupervised_layers.length() != 0 && unsupervised_connections.length() != 0 && unsupervised_layers[i] && unsupervised_connections[i]) { if( layers[i]->size != unsupervised_connections[i]->down_size ) PLERROR("DiscriminativeDeepBeliefNet::build_layers_and_connections() " "- \n" "connections[%i] should have a down_size of %d.\n", i, unsupervised_layers[i]->size); if( unsupervised_connections[i]->up_size != unsupervised_layers[i]->size ) PLERROR("DiscriminativeDeepBeliefNet::build_layers_and_connections() " "- \n" "connections[%i] should have a up_size of %d.\n", i, unsupervised_layers[i+1]->size); if( !(unsupervised_layers[i]->random_gen) ) { unsupervised_layers[i]->random_gen = random_gen; unsupervised_layers[i]->forget(); } if( !(unsupervised_connections[i]->random_gen) ) { unsupervised_connections[i]->random_gen = random_gen; unsupervised_connections[i]->forget(); } PP<RBMMixedLayer> greedy_layer = new RBMMixedLayer(); greedy_layer->sub_layers.resize(2); greedy_layer->sub_layers[0] = layers[i+1]; greedy_layer->sub_layers[1] = unsupervised_layers[i]; greedy_layer->size = layers[i+1]->size + unsupervised_layers[i]->size; greedy_layer->build(); PP<RBMMixedConnection> greedy_connection = new RBMMixedConnection(); greedy_connection->sub_connections.resize(2,1); greedy_connection->sub_connections(0,0) = connections[i]; greedy_connection->sub_connections(1,0) = unsupervised_connections[i]; greedy_connection->build(); greedy_layers[i] = greedy_layer; greedy_connections[i] = greedy_connection; } else { greedy_layers[i] = layers[i+1]; greedy_connections[i] = connections[i]; } if( !(layers[i]->random_gen) ) { layers[i]->random_gen = random_gen; layers[i]->forget(); } if( !(connections[i]->random_gen) ) { connections[i]->random_gen = random_gen; connections[i]->forget(); } activations[i].resize( layers[i]->size ); expectations[i].resize( layers[i]->size ); activation_gradients[i].resize( layers[i]->size ); expectation_gradients[i].resize( layers[i]->size ); } if( !(layers[n_layers-1]->random_gen) ) { layers[n_layers-1]->random_gen = random_gen; layers[n_layers-1]->forget(); } activations[n_layers-1].resize( layers[n_layers-1]->size ); expectations[n_layers-1].resize( layers[n_layers-1]->size ); activation_gradients[n_layers-1].resize( layers[n_layers-1]->size ); expectation_gradients[n_layers-1].resize( layers[n_layers-1]->size ); }
void PLearn::DiscriminativeDeepBeliefNet::build_output_layer_and_cost | ( | ) | [private] |
Definition at line 286 of file DiscriminativeDeepBeliefNet.cc.
References PLearn::GradNNetLayerModule::build(), PLearn::SoftmaxModule::build(), PLearn::ModuleStackModule::build(), PLearn::ClassErrorCostModule::build(), PLearn::CombiningCostsModule::build(), PLearn::NLLCostModule::build(), PLearn::class_error(), PLearn::CombiningCostsModule::cost_weights, final_cost, final_module, PLearn::OnlineLearningModule::input_size, PLearn::GradNNetLayerModule::L1_penalty_factor, PLearn::GradNNetLayerModule::L2_penalty_factor, layers, PLearn::ModuleStackModule::modules, n_classes, n_layers, PLearn::OnlineLearningModule::output_size, output_weights_l1_penalty_factor, output_weights_l2_penalty_factor, PLearn::OnlineLearningModule::random_gen, PLearn::PLearner::random_gen, PLearn::TVec< T >::resize(), PLearn::TVec< T >::size(), and PLearn::CombiningCostsModule::sub_costs.
Referenced by build_(), and forget().
{ GradNNetLayerModule* gnl = new GradNNetLayerModule(); gnl->input_size = layers[n_layers-1]->size; gnl->output_size = n_classes; gnl->L1_penalty_factor = output_weights_l1_penalty_factor; gnl->L2_penalty_factor = output_weights_l2_penalty_factor; gnl->random_gen = random_gen; gnl->build(); SoftmaxModule* sm = new SoftmaxModule(); sm->input_size = n_classes; sm->random_gen = random_gen; sm->build(); ModuleStackModule* msm = new ModuleStackModule(); msm->modules.resize(2); msm->modules[0] = gnl; msm->modules[1] = sm; msm->random_gen = random_gen; msm->build(); final_module = msm; final_module->forget(); NLLCostModule* nll = new NLLCostModule(); nll->input_size = n_classes; nll->random_gen = random_gen; nll->build(); ClassErrorCostModule* class_error = new ClassErrorCostModule(); class_error->input_size = n_classes; class_error->random_gen = random_gen; class_error->build(); CombiningCostsModule* comb_costs = new CombiningCostsModule(); comb_costs->cost_weights.resize(2); comb_costs->cost_weights[0] = 1; comb_costs->cost_weights[1] = 0; comb_costs->sub_costs.resize(2); comb_costs->sub_costs[0] = nll; comb_costs->sub_costs[1] = class_error; comb_costs->build(); final_cost = comb_costs; final_cost->forget(); }
string PLearn::DiscriminativeDeepBeliefNet::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 57 of file DiscriminativeDeepBeliefNet.cc.
Referenced by train().
void PLearn::DiscriminativeDeepBeliefNet::computeCostsFromOutputs | ( | const Vec & | input, |
const Vec & | output, | ||
const Vec & | target, | ||
Vec & | costs | ||
) | const [virtual] |
Computes the costs from already computed output.
Implements PLearn::PLearner.
Definition at line 976 of file DiscriminativeDeepBeliefNet.cc.
References currently_trained_layer, PLearn::TVec< T >::fill(), getTestCostNames(), PLearn::TVec< T >::last(), MISSING_VALUE, n_layers, and PLearn::TVec< T >::resize().
{ //Assumes that computeOutput has been called costs.resize( getTestCostNames().length() ); costs.fill( MISSING_VALUE ); if( currently_trained_layer>n_layers-1 ) if( ((int)round(output[0])) == ((int)round(target[0])) ) costs.last() = 0; else costs.last() = 1; }
void PLearn::DiscriminativeDeepBeliefNet::computeOutput | ( | const Vec & | input, |
Vec & | output | ||
) | const [virtual] |
Computes the output from the input.
Reimplemented from PLearn::PLearner.
Definition at line 960 of file DiscriminativeDeepBeliefNet.cc.
References PLearn::argmax(), computeRepresentation(), currently_trained_layer, final_cost_input, final_module, input_representation, and n_layers.
{ if( currently_trained_layer>n_layers-1 ) { computeRepresentation(input,input_representation, n_layers-1); final_module->fprop( input_representation, final_cost_input ); output[0] = argmax(final_cost_input); } else { computeRepresentation(input, output, currently_trained_layer); } }
void PLearn::DiscriminativeDeepBeliefNet::computeRepresentation | ( | const Vec & | input, |
Vec & | representation, | ||
int | layer | ||
) | const |
Definition at line 938 of file DiscriminativeDeepBeliefNet.cc.
References activations, connections, expectations, i, layers, PLearn::TVec< T >::length(), and PLearn::TVec< T >::resize().
Referenced by computeOutput(), fineTuningStep(), and greedyStep().
{ if(layer == 0) { representation.resize(input.length()); expectations[0] << input; representation << input; return; } expectations[0] << input; for( int i=0 ; i<layer; i++ ) { connections[i]->fprop( expectations[i], activations[i+1] ); layers[i+1]->fprop(activations[i+1],expectations[i+1]); } representation.resize(expectations[layer].length()); representation << expectations[layer]; }
void PLearn::DiscriminativeDeepBeliefNet::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declares the class options.
Reimplemented from PLearn::PLearner.
Definition at line 81 of file DiscriminativeDeepBeliefNet.cc.
References PLearn::OptionBase::buildoption, cd_decrease_ct, cd_learning_rate, compare_joint_in_discriminative_criteria, connections, PLearn::declareOption(), PLearn::PLearner::declareOptions(), discriminative_criteria_weight, do_not_use_generative_criteria, final_cost, final_module, fine_tuning_decrease_ct, fine_tuning_learning_rate, greedy_stages, k_neighbors, layers, PLearn::OptionBase::learntoption, n_classes, n_layers, output_weights_l1_penalty_factor, output_weights_l2_penalty_factor, training_schedule, unsupervised_connections, and unsupervised_layers.
{ declareOption(ol, "cd_learning_rate", &DiscriminativeDeepBeliefNet::cd_learning_rate, OptionBase::buildoption, "The learning rate used during the RBM " "contrastive divergence training.\n"); declareOption(ol, "cd_decrease_ct", &DiscriminativeDeepBeliefNet::cd_decrease_ct, OptionBase::buildoption, "The decrease constant of the learning rate used during " "the RBMs contrastive\n" "divergence training. When a hidden layer has finished " "its training,\n" "the learning rate is reset to it's initial value.\n"); declareOption(ol, "fine_tuning_learning_rate", &DiscriminativeDeepBeliefNet::fine_tuning_learning_rate, OptionBase::buildoption, "The learning rate used during the fine tuning gradient descent.\n"); declareOption(ol, "fine_tuning_decrease_ct", &DiscriminativeDeepBeliefNet::fine_tuning_decrease_ct, OptionBase::buildoption, "The decrease constant of the learning rate used during " "fine tuning\n" "gradient descent.\n"); declareOption(ol, "training_schedule", &DiscriminativeDeepBeliefNet::training_schedule, OptionBase::buildoption, "Number of examples to use during each phase of greedy pre-training.\n" "The number of fine-tunig steps is defined by nstages.\n" ); declareOption(ol, "layers", &DiscriminativeDeepBeliefNet::layers, OptionBase::buildoption, "The layers of units in the network. The first element\n" "of this vector should be the input layer and the\n" "subsequent elements should be the hidden layers. The\n" "output layer should not be included in layers.\n"); declareOption(ol, "connections", &DiscriminativeDeepBeliefNet::connections, OptionBase::buildoption, "The weights of the connections between the layers.\n"); declareOption(ol, "unsupervised_layers", &DiscriminativeDeepBeliefNet::unsupervised_layers, OptionBase::buildoption, "Additional units for greedy unsupervised learning.\n"); declareOption(ol, "unsupervised_connections", &DiscriminativeDeepBeliefNet::unsupervised_connections, OptionBase::buildoption, "Additional connections for greedy unsupervised learning.\n"); declareOption(ol, "k_neighbors", &DiscriminativeDeepBeliefNet::k_neighbors, OptionBase::buildoption, "Number of good nearest neighbors to attract and bad nearest " "neighbors to repel.\n"); declareOption(ol, "n_classes", &DiscriminativeDeepBeliefNet::n_classes, OptionBase::buildoption, "Number of classes.\n"); declareOption(ol, "discriminative_criteria_weight", &DiscriminativeDeepBeliefNet::discriminative_criteria_weight, OptionBase::buildoption, "Weight of the discriminative criteria.\n"); declareOption(ol, "output_weights_l1_penalty_factor", &DiscriminativeDeepBeliefNet::output_weights_l1_penalty_factor, OptionBase::buildoption, "Output weights l1_penalty_factor.\n"); declareOption(ol, "output_weights_l2_penalty_factor", &DiscriminativeDeepBeliefNet::output_weights_l2_penalty_factor, OptionBase::buildoption, "Output weights l2_penalty_factor.\n"); declareOption(ol, "compare_joint_in_discriminative_criteria", &DiscriminativeDeepBeliefNet::compare_joint_in_discriminative_criteria, OptionBase::buildoption, "Indication that the discriminative criteria should use the joint\n" "over the input and the hidden units, instead of the conditional\n" "over the hidden units given the input units.\n"); declareOption(ol, "do_not_use_generative_criteria", &DiscriminativeDeepBeliefNet::do_not_use_generative_criteria, OptionBase::buildoption, "Indication that the generative criteria should not be used during learning\n" "(does not work with compare_joint_in_discriminative_criteria = true).\n"); // declareOption(ol, "cancel_normalization_terms", // &DiscriminativeDeepBeliefNet::cancel_normalization_terms, // OptionBase::buildoption, // "Indication that the discriminative and generative criteria should cancel\n" // "their normalization terms. This is for the " // "compare_joint_in_discriminative_criteria\n" // "option, and this option ignores the value of discriminative_criteria_weight.\n"); declareOption(ol, "greedy_stages", &DiscriminativeDeepBeliefNet::greedy_stages, OptionBase::learntoption, "Number of training samples seen in the different greedy " "phases.\n" ); declareOption(ol, "n_layers", &DiscriminativeDeepBeliefNet::n_layers, OptionBase::learntoption, "Number of layers.\n" ); declareOption(ol, "final_module", &DiscriminativeDeepBeliefNet::final_module, OptionBase::learntoption, "Output layer of neural net.\n" ); declareOption(ol, "final_cost", &DiscriminativeDeepBeliefNet::final_cost, OptionBase::learntoption, "Cost on output layer of neural net.\n" ); // Now call the parent class' declareOptions inherited::declareOptions(ol); }
static const PPath& PLearn::DiscriminativeDeepBeliefNet::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::PLearner.
Definition at line 197 of file DiscriminativeDeepBeliefNet.h.
:
//##### Not Options #####################################################
DiscriminativeDeepBeliefNet * PLearn::DiscriminativeDeepBeliefNet::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::PLearner.
Definition at line 57 of file DiscriminativeDeepBeliefNet.cc.
void PLearn::DiscriminativeDeepBeliefNet::fineTuningStep | ( | const Vec & | input, |
const Vec & | target, | ||
Vec & | train_costs | ||
) |
Definition at line 902 of file DiscriminativeDeepBeliefNet.cc.
References activation_gradients, activations, computeRepresentation(), connections, expectation_gradients, expectations, final_cost, final_cost_gradient, final_cost_input, final_cost_value, final_module, i, input_representation, layers, and n_layers.
Referenced by train().
{ // Get example representation computeRepresentation(input, input_representation, n_layers-1); // Compute supervised gradient final_module->fprop( input_representation, final_cost_input ); final_cost->fprop( final_cost_input, target, final_cost_value ); final_cost->bpropUpdate( final_cost_input, target, final_cost_value[0], final_cost_gradient ); final_module->bpropUpdate( input_representation, final_cost_input, expectation_gradients[ n_layers-1 ], final_cost_gradient ); for( int i=n_layers-1 ; i>0 ; i-- ) { layers[i]->bpropUpdate( activations[i], expectations[i], activation_gradients[i], expectation_gradients[i] ); connections[i-1]->bpropUpdate( expectations[i-1], activations[i], expectation_gradients[i-1], activation_gradients[i] ); } }
void PLearn::DiscriminativeDeepBeliefNet::forget | ( | ) | [virtual] |
(Re-)initializes the PLearner in its fresh state (that state may depend on the 'seed' option) and sets 'stage' back to 0 (this is the stage of a fresh learner!).
(Re-)initialize the PLearner in its fresh state (that state may depend on the 'seed' option) and sets 'stage' back to 0 (this is the stage of a fresh learner!)
A typical forget() method should do the following:
Reimplemented from PLearn::PLearner.
Definition at line 528 of file DiscriminativeDeepBeliefNet.cc.
References build_output_layer_and_cost(), PLearn::TVec< T >::clear(), connections, PLearn::PLearner::forget(), greedy_stages, i, layers, PLearn::TVec< T >::length(), n_layers, PLearn::PLearner::stage, unsupervised_connections, and unsupervised_layers.
{ inherited::forget(); for( int i=0 ; i<n_layers ; i++ ) layers[i]->forget(); for( int i=0 ; i<n_layers-1 ; i++ ) connections[i]->forget(); if(unsupervised_layers.length() != 0) for( int i=0 ; i<n_layers-1 ; i++ ) unsupervised_layers[i]->forget(); if(unsupervised_connections.length() != 0) for( int i=0 ; i<n_layers-1 ; i++ ) unsupervised_connections[i]->forget(); build_output_layer_and_cost(); stage = 0; greedy_stages.clear(); }
OptionList & PLearn::DiscriminativeDeepBeliefNet::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 57 of file DiscriminativeDeepBeliefNet.cc.
OptionMap & PLearn::DiscriminativeDeepBeliefNet::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 57 of file DiscriminativeDeepBeliefNet.cc.
RemoteMethodMap & PLearn::DiscriminativeDeepBeliefNet::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 57 of file DiscriminativeDeepBeliefNet.cc.
TVec< string > PLearn::DiscriminativeDeepBeliefNet::getTestCostNames | ( | ) | const [virtual] |
Returns the names of the costs computed by computeCostsFromOutpus (and thus the test method).
Implements PLearn::PLearner.
Definition at line 991 of file DiscriminativeDeepBeliefNet.cc.
References PLearn::TVec< T >::append().
Referenced by computeCostsFromOutputs(), and getTrainCostNames().
{ // Return the names of the costs computed by computeCostsFromOutputs // (these may or may not be exactly the same as what's returned by // getTrainCostNames). TVec<string> cost_names(0); cost_names.append( "class_error" ); return cost_names; }
TVec< string > PLearn::DiscriminativeDeepBeliefNet::getTrainCostNames | ( | ) | const [virtual] |
Returns the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats.
Implements PLearn::PLearner.
Definition at line 1004 of file DiscriminativeDeepBeliefNet.cc.
References getTestCostNames().
Referenced by train().
{ return getTestCostNames(); }
void PLearn::DiscriminativeDeepBeliefNet::greedyStep | ( | const Vec & | input, |
const Vec & | target, | ||
int | index, | ||
Vec | train_costs, | ||
int | stage, | ||
Vec | dissimilar_example | ||
) |
Definition at line 708 of file DiscriminativeDeepBeliefNet.cc.
References cd_decrease_ct, cd_learning_rate, compare_joint_in_discriminative_criteria, computeRepresentation(), connections, disc_neg_down_val, disc_neg_up_val, disc_pos_down_val1, disc_pos_down_val2, disc_pos_up_val1, disc_pos_up_val2, discriminative_criteria_weight, dissimilar_example_representation, do_not_use_generative_criteria, greedy_connections, greedy_layers, input_representation, layers, n_layers, neg_down_val, neg_up_val, PLASSERT, pos_down_val, pos_up_val, and PLearn::sample().
Referenced by train().
{ PLASSERT( index < n_layers ); real lr; // Get dissimilar example representation computeRepresentation(dissimilar_example, dissimilar_example_representation, index); // Get example representation computeRepresentation(input, input_representation, index); if( !do_not_use_generative_criteria ) { // CD generative learning stats // Positive phase greedy_connections[index]->setAsDownInput( input_representation ); greedy_layers[index]->getAllActivations( greedy_connections[index] ); greedy_layers[index]->computeExpectation(); greedy_layers[index]->generateSample(); pos_down_val << input_representation; pos_up_val << greedy_layers[index]->expectation; if( !compare_joint_in_discriminative_criteria ) { disc_pos_down_val1 << input_representation; disc_pos_up_val1 << layers[index+1]->expectation; } // if( !cancel_normalization_terms ) // { // Negative phase greedy_connections[index]->setAsUpInput( greedy_layers[index]->sample ); layers[index]->getAllActivations( greedy_connections[index] ); layers[index]->computeExpectation(); layers[index]->generateSample(); greedy_connections[index]->setAsDownInput( layers[index]->sample ); greedy_layers[index]->getAllActivations( greedy_connections[index] ); greedy_layers[index]->computeExpectation(); neg_down_val << layers[index]->sample; neg_up_val << greedy_layers[index]->expectation; // } } else if( !compare_joint_in_discriminative_criteria ) { connections[index]->setAsDownInput( input_representation ); layers[index+1]->getAllActivations( connections[index] ); layers[index+1]->computeExpectation(); disc_pos_down_val1 << input_representation; disc_pos_up_val1 << layers[index+1]->expectation; } // CD discriminative criteria stats if( !compare_joint_in_discriminative_criteria ) { // Positive phase connections[index]->setAsDownInput( dissimilar_example_representation ); layers[index+1]->getAllActivations( connections[index] ); layers[index+1]->computeExpectation(); disc_pos_down_val2 << dissimilar_example_representation; disc_pos_up_val2 << layers[index+1]->expectation; } // Negative phase disc_neg_down_val << input_representation; disc_neg_down_val += dissimilar_example_representation; disc_neg_down_val /= 2; connections[index]->setAsDownInput( disc_neg_down_val ); layers[index+1]->getAllActivations( connections[index] ); layers[index+1]->computeExpectation(); disc_neg_up_val << layers[index+1]->expectation; if( compare_joint_in_discriminative_criteria ) //&& !cancel_normalization_terms) { layers[index+1]->generateSample(); connections[index]->setAsUpInput( layers[index+1]->sample ); layers[index]->getAllActivations( connections[index] ); layers[index]->computeExpectation(); layers[index]->generateSample(); connections[index]->setAsDownInput( layers[index]->sample ); layers[index+1]->getAllActivations( connections[index] ); layers[index+1]->computeExpectation(); disc_pos_down_val1 << layers[index]->sample; disc_pos_up_val1 << layers[index+1]->expectation; } // RBM updates if( !do_not_use_generative_criteria ) //&& !cancel_normalization_terms ) { lr = cd_learning_rate/(1 + cd_decrease_ct * this_stage); layers[index]->setLearningRate( lr ); greedy_connections[index]->setLearningRate( lr ); greedy_layers[index]->setLearningRate( lr ); layers[index]->update( pos_down_val, neg_down_val ); greedy_connections[index]->update( pos_down_val, pos_up_val, neg_down_val, neg_up_val ); greedy_layers[index]->update( pos_up_val, neg_up_val ); } if( //cancel_normalization_terms || discriminative_criteria_weight != 0 ) { lr = discriminative_criteria_weight * cd_learning_rate/(1 + cd_decrease_ct * this_stage); if( !compare_joint_in_discriminative_criteria ) { layers[index]->setLearningRate( lr ); connections[index]->setLearningRate( lr ); layers[index+1]->setLearningRate( lr ); layers[index]->accumulatePosStats( disc_pos_down_val1 ); layers[index]->accumulatePosStats( disc_pos_down_val2 ); layers[index]->accumulateNegStats( disc_neg_down_val ); layers[index]->update(); connections[index]->accumulatePosStats( disc_pos_down_val1, disc_pos_up_val1 ); connections[index]->accumulatePosStats( disc_pos_down_val2, disc_pos_up_val2 ); connections[index]->accumulateNegStats( disc_neg_down_val, disc_neg_up_val ); connections[index]->update(); layers[index+1]->accumulatePosStats( disc_pos_up_val1 ); layers[index+1]->accumulatePosStats( disc_pos_up_val2 ); layers[index+1]->accumulateNegStats( disc_neg_up_val ); layers[index+1]->update(); } else //if( !cancel_normalization_terms ) { layers[index]->setLearningRate( lr ); connections[index]->setLearningRate( lr ); layers[index+1]->setLearningRate( lr ); layers[index]->accumulatePosStats( disc_pos_down_val1 ); layers[index]->accumulateNegStats( disc_neg_down_val ); layers[index]->update(); connections[index]->accumulatePosStats( disc_pos_down_val1, disc_pos_up_val1 ); connections[index]->accumulateNegStats( disc_neg_down_val, disc_neg_up_val ); connections[index]->update(); layers[index+1]->accumulatePosStats( disc_pos_up_val1 ); layers[index+1]->accumulateNegStats( disc_neg_up_val ); layers[index+1]->update(); } // else // { // lr = cd_learning_rate/(1 + cd_decrease_ct // * this_stage); // layers[index]->setLearningRate( lr ); // connections[index]->setLearningRate( lr ); // layers[index+1]->setLearningRate( lr ); // // layers[index]->accumulatePosStats( pos_down_val ); // layers[index]->accumulateNegStats( disc_neg_down_val ); // layers[index]->update(); // // connections[index]->accumulatePosStats( pos_down_val, // pos_up_val ); // connections[index]->accumulateNegStats( disc_neg_down_val, // disc_neg_up_val ); // connections[index]->update(); // // layers[index+1]->accumulatePosStats( pos_up_val ); // layers[index+1]->accumulateNegStats( disc_neg_up_val ); // layers[index+1]->update(); // } } }
void PLearn::DiscriminativeDeepBeliefNet::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::PLearner.
Definition at line 477 of file DiscriminativeDeepBeliefNet.cc.
References activation_gradients, activations, connections, PLearn::deepCopyField(), disc_neg_down_val, disc_neg_up_val, disc_pos_down_val1, disc_pos_down_val2, disc_pos_up_val1, disc_pos_up_val2, dissimilar_example_representation, expectation_gradients, expectations, final_cost, final_cost_gradient, final_cost_input, final_cost_value, final_module, greedy_connections, greedy_layers, greedy_stages, input_representation, layers, PLearn::PLearner::makeDeepCopyFromShallowCopy(), nearest_neighbors_indices, neg_down_val, neg_up_val, other_class_datasets, pos_down_val, pos_up_val, training_schedule, unsupervised_connections, and unsupervised_layers.
{ inherited::makeDeepCopyFromShallowCopy(copies); // deepCopyField(, copies); // Public options deepCopyField(training_schedule, copies); deepCopyField(layers, copies); deepCopyField(connections, copies); deepCopyField(unsupervised_layers, copies); deepCopyField(unsupervised_connections, copies); // Protected options deepCopyField(activations, copies); deepCopyField(expectations, copies); deepCopyField(activation_gradients, copies); deepCopyField(expectation_gradients, copies); deepCopyField(greedy_layers, copies); deepCopyField(greedy_connections, copies); deepCopyField(dissimilar_example_representation, copies); deepCopyField(input_representation, copies); deepCopyField(pos_down_val, copies); deepCopyField(pos_up_val, copies); deepCopyField(neg_down_val, copies); deepCopyField(neg_up_val, copies); deepCopyField(disc_pos_down_val1, copies); deepCopyField(disc_pos_up_val1, copies); deepCopyField(disc_pos_down_val2, copies); deepCopyField(disc_pos_up_val2, copies); deepCopyField(disc_neg_down_val, copies); deepCopyField(disc_neg_up_val, copies); deepCopyField(final_cost_input, copies); deepCopyField(final_cost_value, copies); deepCopyField(final_cost_gradient, copies); deepCopyField(other_class_datasets, copies); deepCopyField(nearest_neighbors_indices, copies); deepCopyField(greedy_stages, copies); deepCopyField(final_module, copies); deepCopyField(final_cost, copies); }
int PLearn::DiscriminativeDeepBeliefNet::outputsize | ( | ) | const [virtual] |
Returns the size of this learner's output, (which typically may depend on its inputsize(), targetsize() and set options).
Implements PLearn::PLearner.
Definition at line 520 of file DiscriminativeDeepBeliefNet.cc.
References currently_trained_layer, layers, n_layers, and PLearn::TVec< T >::size().
{ if( currently_trained_layer>n_layers-1 ) return 1; else return layers[currently_trained_layer]->size; }
void PLearn::DiscriminativeDeepBeliefNet::setLearningRate | ( | real | the_learning_rate | ) | [private] |
Definition at line 1060 of file DiscriminativeDeepBeliefNet.cc.
References final_cost, final_module, greedy_connections, greedy_layers, i, layers, and n_layers.
Referenced by train().
{ layers[0]->setLearningRate( the_learning_rate ); for( int i=0 ; i<n_layers-1 ; i++ ) { greedy_layers[i]->setLearningRate( the_learning_rate ); greedy_connections[i]->setLearningRate( the_learning_rate ); } final_module->setLearningRate( the_learning_rate ); final_cost->setLearningRate( the_learning_rate ); }
void PLearn::DiscriminativeDeepBeliefNet::setTrainingSet | ( | VMat | training_set, |
bool | call_forget = true |
||
) | [virtual] |
Declares the training set.
Then calls build() and forget() if necessary. Also sets this learner's inputsize_ targetsize_ weightsize_ from those of the training_set. Note: You shouldn't have to override this in subclasses, except in maybe to forward the call to an underlying learner.
Reimplemented from PLearn::PLearner.
Definition at line 1009 of file DiscriminativeDeepBeliefNet.cc.
References nearest_neighbors_are_up_to_date, and PLearn::PLearner::setTrainingSet().
{ inherited::setTrainingSet(training_set,call_forget); nearest_neighbors_are_up_to_date = false; }
void PLearn::DiscriminativeDeepBeliefNet::train | ( | ) | [virtual] |
The role of the train method is to bring the learner up to stage==nstages, updating the train_stats collector with training costs measured on-line in the process.
Implements PLearn::PLearner.
Definition at line 562 of file DiscriminativeDeepBeliefNet.cc.
References classname(), currently_trained_layer, PLearn::TVec< T >::data(), disc_neg_down_val, disc_neg_up_val, disc_pos_down_val1, disc_pos_down_val2, disc_pos_up_val1, disc_pos_up_val2, dissimilar_example_representation, PLearn::endl(), PLearn::fast_exact_is_equal(), PLearn::TVec< T >::fill(), final_cost_gradient, final_cost_input, final_cost_value, fine_tuning_decrease_ct, fine_tuning_learning_rate, fineTuningStep(), PLearn::VMat::getExample(), getTrainCostNames(), greedy_layers, greedy_stages, greedyStep(), i, input_representation, PLearn::PLearner::inputsize(), k_neighbors, layers, PLearn::VMat::length(), PLearn::TVec< T >::length(), MISSING_VALUE, n_classes, n_layers, nearest_neighbors_indices, neg_down_val, neg_up_val, PLearn::PLearner::nstages, other_class_datasets, PLERROR, pos_down_val, pos_up_val, PLearn::PLearner::random_gen, PLearn::PLearner::report_progress, PLearn::TVec< T >::resize(), PLearn::sample(), setLearningRate(), PLearn::PLearner::stage, PLearn::TVec< T >::subVec(), PLearn::PLearner::targetsize(), PLearn::tostring(), PLearn::PLearner::train_set, PLearn::PLearner::train_stats, training_schedule, and updateNearestNeighbors().
{ MODULE_LOG << "train() called " << endl; MODULE_LOG << " training_schedule = " << training_schedule << endl; Vec input( inputsize() ); Vec dissimilar_example( inputsize() ); Vec target( targetsize() ); Vec target2( targetsize() ); real weight; // unused real weight2; // unused TVec<string> train_cost_names = getTrainCostNames() ; Vec train_costs( train_cost_names.length() ); train_costs.fill(MISSING_VALUE) ; int nsamples = train_set->length(); int sample; PP<ProgressBar> pb; // clear stats of previous epoch train_stats->forget(); int init_stage; /***** initial greedy training *****/ for( int i=0 ; i<n_layers-1 ; i++ ) { updateNearestNeighbors(); MODULE_LOG << "Training connection weights between layers " << i << " and " << i+1 << endl; int end_stage = training_schedule[i]; int* this_stage = greedy_stages.subVec(i,1).data(); init_stage = *this_stage; MODULE_LOG << " stage = " << *this_stage << endl; MODULE_LOG << " end_stage = " << end_stage << endl; if( report_progress && *this_stage < end_stage ) pb = new ProgressBar( "Training layer "+tostring(i) +" of "+classname(), end_stage - init_stage ); train_costs.fill(MISSING_VALUE); dissimilar_example_representation.resize(layers[i]->size); input_representation.resize(layers[i]->size); pos_down_val.resize(layers[i]->size); pos_up_val.resize(greedy_layers[i]->size); neg_down_val.resize(layers[i]->size); neg_up_val.resize(greedy_layers[i]->size); disc_pos_down_val1.resize(layers[i]->size); disc_pos_up_val1.resize(layers[i+1]->size); disc_pos_down_val2.resize(layers[i]->size); disc_pos_up_val2.resize(layers[i+1]->size); disc_neg_down_val.resize(layers[i]->size); disc_neg_up_val.resize(layers[i+1]->size); for( ; *this_stage<end_stage ; (*this_stage)++ ) { sample = *this_stage % nsamples; train_set->getExample(sample, input, target, weight); // Find dissimilar example int dissim_index = nearest_neighbors_indices( sample,random_gen->uniform_multinomial_sample(k_neighbors)); other_class_datasets[(int)round(target[0])]->getExample(dissim_index, dissimilar_example, target2, weight2); if(((int)round(target[0])) == ((int)round(target2[0]))) PLERROR("DiscriminativeDeepBeliefNet::train(): dissimilar" " example is from same class!"); greedyStep( input, target, i, train_costs, *this_stage, dissimilar_example); train_stats->update( train_costs ); if( pb ) pb->update( *this_stage - init_stage + 1 ); } } /***** fine-tuning by gradient descent *****/ if( stage < nstages ) { MODULE_LOG << "Fine-tuning all parameters, by gradient descent" << endl; MODULE_LOG << " stage = " << stage << endl; MODULE_LOG << " nstages = " << nstages << endl; MODULE_LOG << " fine_tuning_learning_rate = " << fine_tuning_learning_rate << endl; init_stage = stage; if( report_progress && stage < nstages ) pb = new ProgressBar( "Fine-tuning parameters of all layers of " + classname(), nstages - init_stage ); setLearningRate( fine_tuning_learning_rate ); train_costs.fill(MISSING_VALUE); final_cost_input.resize(n_classes); final_cost_value.resize(2); // Should be resized anyways final_cost_gradient.resize(n_classes); for( ; stage<nstages ; stage++ ) { sample = stage % nsamples; if( !fast_exact_is_equal( fine_tuning_decrease_ct, 0. ) ) setLearningRate( fine_tuning_learning_rate / (1. + fine_tuning_decrease_ct * stage ) ); train_set->getExample( sample, input, target, weight ); fineTuningStep( input, target, train_costs ); train_stats->update( train_costs ); if( pb ) pb->update( stage - init_stage + 1 ); } } train_stats->finalize(); MODULE_LOG << " train costs = " << train_stats->getMean() << endl; // Update currently_trained_layer if(stage > 0) currently_trained_layer = n_layers; else { currently_trained_layer = n_layers-1; while(currently_trained_layer>1 && greedy_stages[currently_trained_layer-1] <= 0) currently_trained_layer--; } }
void PLearn::DiscriminativeDeepBeliefNet::updateNearestNeighbors | ( | ) |
Definition at line 1015 of file DiscriminativeDeepBeliefNet.cc.
References PLearn::TVec< T >::append(), PLearn::computeNearestNeighbors(), PLearn::endl(), PLearn::VMat::getExample(), i, PLearn::PLearner::inputsize(), k_neighbors, PLearn::VMat::length(), n_classes, nearest_neighbors_are_up_to_date, nearest_neighbors_indices, other_class_datasets, PLearn::TMat< T >::resize(), PLearn::TVec< T >::resize(), PLearn::PLearner::targetsize(), and PLearn::PLearner::train_set.
Referenced by train().
{ if( !nearest_neighbors_are_up_to_date ) { MODULE_LOG << "Computing nearest neighbors" << endl; Vec input( inputsize() ); Vec target( targetsize() ); real weight; // unused other_class_datasets.resize(n_classes); for(int k=0; k<n_classes; k++) { other_class_datasets[k] = new ClassSubsetVMatrix(); other_class_datasets[k]->classes.resize(0); for(int l=0; l<n_classes; l++) if( l != k ) other_class_datasets[k]->classes.append(l); other_class_datasets[k]->source = train_set; other_class_datasets[k]->build(); } // Find training nearest neighbors input.resize(train_set->inputsize()); target.resize(train_set->targetsize()); nearest_neighbors_indices.resize(train_set->length(), k_neighbors); TVec<int> nearest_neighbors_indices_row; for(int i=0; i<train_set.length(); i++) { train_set->getExample(i,input,target,weight); nearest_neighbors_indices_row = nearest_neighbors_indices(i); computeNearestNeighbors( new GetInputVMatrix((VMatrix *) other_class_datasets[(int)round(target[0])]), input, nearest_neighbors_indices_row, -1); } } nearest_neighbors_are_up_to_date = true; }
Reimplemented from PLearn::PLearner.
Definition at line 197 of file DiscriminativeDeepBeliefNet.h.
TVec<Vec> PLearn::DiscriminativeDeepBeliefNet::activation_gradients [mutable, protected] |
Stores the gradient of the cost wrt the activations of the input and hidden layers (at the input of the layers)
Definition at line 223 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_layers_and_connections(), fineTuningStep(), and makeDeepCopyFromShallowCopy().
TVec<Vec> PLearn::DiscriminativeDeepBeliefNet::activations [mutable, protected] |
Stores the activations of the input and hidden layers (at the input of the layers)
Definition at line 214 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_layers_and_connections(), computeRepresentation(), fineTuningStep(), and makeDeepCopyFromShallowCopy().
Contrastive divergence decrease constant.
Definition at line 75 of file DiscriminativeDeepBeliefNet.h.
Referenced by declareOptions(), and greedyStep().
Contrastive divergence learning rate.
Definition at line 72 of file DiscriminativeDeepBeliefNet.h.
Referenced by declareOptions(), and greedyStep().
Indication that the discriminative criteria should use the joint over the input and the hidden units, instead of the conditional over the hidden units given the input units.
Definition at line 119 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_(), declareOptions(), and greedyStep().
The weights of the connections between the layers.
Definition at line 92 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_layers_and_connections(), computeRepresentation(), declareOptions(), fineTuningStep(), forget(), greedyStep(), and makeDeepCopyFromShallowCopy().
Currently trained layer (1 means the first hidden layer, n_layers means the output layer)
Definition at line 282 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_(), computeCostsFromOutputs(), computeOutput(), outputsize(), and train().
Negative down statistic.
Definition at line 260 of file DiscriminativeDeepBeliefNet.h.
Referenced by greedyStep(), makeDeepCopyFromShallowCopy(), and train().
Negative up statistic.
Definition at line 262 of file DiscriminativeDeepBeliefNet.h.
Referenced by greedyStep(), makeDeepCopyFromShallowCopy(), and train().
First discriminative positive down statistic.
Definition at line 252 of file DiscriminativeDeepBeliefNet.h.
Referenced by greedyStep(), makeDeepCopyFromShallowCopy(), and train().
Second discriminative positive down statistic.
Definition at line 256 of file DiscriminativeDeepBeliefNet.h.
Referenced by greedyStep(), makeDeepCopyFromShallowCopy(), and train().
First discriminative positive up statistic.
Definition at line 254 of file DiscriminativeDeepBeliefNet.h.
Referenced by greedyStep(), makeDeepCopyFromShallowCopy(), and train().
Second discriminative positive up statistic.
Definition at line 258 of file DiscriminativeDeepBeliefNet.h.
Referenced by greedyStep(), makeDeepCopyFromShallowCopy(), and train().
Weight of the discriminative criteria.
Definition at line 108 of file DiscriminativeDeepBeliefNet.h.
Referenced by declareOptions(), and greedyStep().
Dissimilar example representation.
Definition at line 237 of file DiscriminativeDeepBeliefNet.h.
Referenced by greedyStep(), makeDeepCopyFromShallowCopy(), and train().
Indication that the generative criteria should not be used during learning (does not work with compare_joint_in_discriminative_criteria = true).
Definition at line 123 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_(), declareOptions(), and greedyStep().
TVec<Vec> PLearn::DiscriminativeDeepBeliefNet::expectation_gradients [mutable, protected] |
Stores the gradient of the cost wrt the expectations of the input and hidden layers (at the output of the layers)
Definition at line 228 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_layers_and_connections(), fineTuningStep(), and makeDeepCopyFromShallowCopy().
TVec<Vec> PLearn::DiscriminativeDeepBeliefNet::expectations [mutable, protected] |
Stores the expectations of the input and hidden layers (at the output of the layers)
Definition at line 218 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_layers_and_connections(), computeRepresentation(), fineTuningStep(), and makeDeepCopyFromShallowCopy().
PP<CostModule> PLearn::DiscriminativeDeepBeliefNet::final_cost [protected] |
Cost on output layer of neural net.
Definition at line 288 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_(), build_output_layer_and_cost(), declareOptions(), fineTuningStep(), makeDeepCopyFromShallowCopy(), and setLearningRate().
Vec PLearn::DiscriminativeDeepBeliefNet::final_cost_gradient [mutable, protected] |
Cost gradient on output layer.
Definition at line 269 of file DiscriminativeDeepBeliefNet.h.
Referenced by fineTuningStep(), makeDeepCopyFromShallowCopy(), and train().
Vec PLearn::DiscriminativeDeepBeliefNet::final_cost_input [mutable, protected] |
Input of cost function.
Definition at line 265 of file DiscriminativeDeepBeliefNet.h.
Referenced by computeOutput(), fineTuningStep(), makeDeepCopyFromShallowCopy(), and train().
Vec PLearn::DiscriminativeDeepBeliefNet::final_cost_value [mutable, protected] |
Cost value.
Definition at line 267 of file DiscriminativeDeepBeliefNet.h.
Referenced by fineTuningStep(), makeDeepCopyFromShallowCopy(), and train().
Output layer of neural net.
Definition at line 285 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_(), build_output_layer_and_cost(), computeOutput(), declareOptions(), fineTuningStep(), makeDeepCopyFromShallowCopy(), and setLearningRate().
The decrease constant of the learning rate used during fine tuning gradient descent.
Definition at line 82 of file DiscriminativeDeepBeliefNet.h.
Referenced by declareOptions(), and train().
The learning rate used during the fine tuning gradient descent.
Definition at line 78 of file DiscriminativeDeepBeliefNet.h.
Referenced by declareOptions(), and train().
TVec< PP<RBMConnection> > PLearn::DiscriminativeDeepBeliefNet::greedy_connections [protected] |
Connections used for greedy learning.
Definition at line 234 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_layers_and_connections(), greedyStep(), makeDeepCopyFromShallowCopy(), and setLearningRate().
TVec< PP<RBMLayer> > PLearn::DiscriminativeDeepBeliefNet::greedy_layers [protected] |
Layers used for greedy learning.
Definition at line 231 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_layers_and_connections(), greedyStep(), makeDeepCopyFromShallowCopy(), setLearningRate(), and train().
TVec<int> PLearn::DiscriminativeDeepBeliefNet::greedy_stages [protected] |
Stages of the different greedy phases.
Definition at line 278 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_(), declareOptions(), forget(), makeDeepCopyFromShallowCopy(), and train().
Vec PLearn::DiscriminativeDeepBeliefNet::input_representation [mutable, protected] |
Example representation.
Definition at line 240 of file DiscriminativeDeepBeliefNet.h.
Referenced by computeOutput(), fineTuningStep(), greedyStep(), makeDeepCopyFromShallowCopy(), and train().
Number of good nearest neighbors to attract and bad nearest neighbors to repel.
Definition at line 102 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_(), declareOptions(), train(), and updateNearestNeighbors().
The layers of units in the network.
Definition at line 89 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_(), build_layers_and_connections(), build_output_layer_and_cost(), computeRepresentation(), declareOptions(), fineTuningStep(), forget(), greedyStep(), makeDeepCopyFromShallowCopy(), outputsize(), setLearningRate(), and train().
Number of classes.
Definition at line 105 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_(), build_output_layer_and_cost(), declareOptions(), train(), and updateNearestNeighbors().
Number of layers.
Definition at line 133 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_(), build_layers_and_connections(), build_output_layer_and_cost(), computeCostsFromOutputs(), computeOutput(), declareOptions(), fineTuningStep(), forget(), greedyStep(), outputsize(), setLearningRate(), and train().
Indication that nearest_neighbors_indices is up to date.
Definition at line 210 of file DiscriminativeDeepBeliefNet.h.
Referenced by setTrainingSet(), and updateNearestNeighbors().
Nearest neighbors for each training example.
Definition at line 275 of file DiscriminativeDeepBeliefNet.h.
Referenced by makeDeepCopyFromShallowCopy(), train(), and updateNearestNeighbors().
Vec PLearn::DiscriminativeDeepBeliefNet::neg_down_val [protected] |
Negative down statistic.
Definition at line 247 of file DiscriminativeDeepBeliefNet.h.
Referenced by greedyStep(), makeDeepCopyFromShallowCopy(), and train().
Vec PLearn::DiscriminativeDeepBeliefNet::neg_up_val [protected] |
Negative up statistic.
Definition at line 249 of file DiscriminativeDeepBeliefNet.h.
Referenced by greedyStep(), makeDeepCopyFromShallowCopy(), and train().
TVec< PP<ClassSubsetVMatrix> > PLearn::DiscriminativeDeepBeliefNet::other_class_datasets [protected] |
Datasets for each class.
Definition at line 272 of file DiscriminativeDeepBeliefNet.h.
Referenced by makeDeepCopyFromShallowCopy(), train(), and updateNearestNeighbors().
Output weights l1_penalty_factor.
Definition at line 111 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_output_layer_and_cost(), and declareOptions().
Output weights l2_penalty_factor.
Definition at line 114 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_output_layer_and_cost(), and declareOptions().
Vec PLearn::DiscriminativeDeepBeliefNet::pos_down_val [protected] |
Positive down statistic.
Definition at line 243 of file DiscriminativeDeepBeliefNet.h.
Referenced by greedyStep(), makeDeepCopyFromShallowCopy(), and train().
Vec PLearn::DiscriminativeDeepBeliefNet::pos_up_val [protected] |
Positive up statistic.
Definition at line 245 of file DiscriminativeDeepBeliefNet.h.
Referenced by greedyStep(), makeDeepCopyFromShallowCopy(), and train().
Number of examples to use during each phase of greedy pre-training.
The number of fine-tunig steps is defined by nstages.
Definition at line 86 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_(), declareOptions(), makeDeepCopyFromShallowCopy(), and train().
Additional connections for greedy unsupervised learning.
Definition at line 98 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_layers_and_connections(), declareOptions(), forget(), and makeDeepCopyFromShallowCopy().
Additional units for greedy unsupervised learning.
Definition at line 95 of file DiscriminativeDeepBeliefNet.h.
Referenced by build_layers_and_connections(), declareOptions(), forget(), and makeDeepCopyFromShallowCopy().