PLearn 0.1
|
Does the same thing as Hinton's deep belief nets. More...
#include <GaussianDBNRegression.h>
Public Member Functions | |
GaussianDBNRegression () | |
Default constructor. | |
virtual real | density (const Vec &y) const |
Return probability density p(y | x) | |
virtual real | log_density (const Vec &y) const |
Return log of probability density log(p(y | x)). | |
virtual real | survival_fn (const Vec &y) const |
Return survival function: P(Y>y | x). | |
virtual real | cdf (const Vec &y) const |
Return cdf: P(Y<y | x). | |
virtual void | expectation (Vec &mu) const |
Return E[Y | x]. | |
virtual void | variance (Mat &cov) const |
Return Var[Y | x]. | |
virtual void | generate (Vec &y) const |
Return a pseudo-random sample generated from the conditional distribution, of density p(y | x). | |
virtual bool | setPredictorPredictedSizes (int the_predictor_size, int the_predicted_size, bool call_parent=true) |
Generates a pseudo-random sample x from the reversed conditional distribution, of density p(x | y) (and NOT p(y | x)). | |
virtual void | setPredictor (const Vec &predictor, bool call_parent=true) const |
Set the value for the predictor part of a conditional probability. | |
virtual void | forget () |
(Re-)initializes the PDistribution in its fresh state (that state may depend on the 'seed' option). | |
virtual void | train () |
The role of the train method is to bring the learner up to stage == nstages, updating the train_stats collector with training costs measured on-line in the process. | |
virtual void | computeCostsFromOutputs (const Vec &input, const Vec &output, const Vec &target, Vec &costs) const |
Compute a cost, depending on the type of the first output : if it is the density or the log-density: NLL if it is the expectation: NLL and class error. | |
virtual TVec< string > | getTestCostNames () const |
Return [ "NLL" ] (the only cost computed by a PDistribution). | |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual GaussianDBNRegression * | deepCopy (CopiesMap &copies) const |
virtual void | build () |
Simply calls inherited::build() then build_(). | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transforms a shallow copy into a deep copy. | |
Static Public Member Functions | |
static string | _classname_ () |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
real | learning_rate |
### declare public option fields (such as build options) here Start your comments with Doxygen-compatible comments such as //! | |
real | weight_decay |
The weight decay. | |
string | initialization_method |
The method used to initialize the weights: | |
int | n_layers |
Number of layers, including input layer and last layer, but not target layer. | |
TVec< PP< RBMLayer > > | layers |
Layers that learn representations of the input, layers[0] is input layer, layers[n_layers-1] is last layer. | |
PP< RBMLayer > | last_layer |
last_layer is layer[n_layers-1] | |
PP< RBMLayer > | target_layer |
Target (or label) layer. | |
TVec< PP< RBMLLParameters > > | params |
RBMParameters linking the unsupervised layers. | |
PP< RBMQLParameters > | input_params |
Parameters linking input layer[0] and layer[1]. | |
PP< RBMLQParameters > | target_params |
Parameters linking target_layer and last_layer. | |
TVec< int > | training_schedule |
Number of examples to use during each of the different greedy steps of the training phase. | |
string | fine_tuning_method |
Method for fine-tuning the whole network after greedy learning. | |
bool | use_sample_rather_than_expectation_in_positive_phase_statistics |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Protected Member Functions | |
virtual void | greedyStep (const Vec &predictor, int params_index) |
virtual void | fineTuneByGradientDescent (const Vec &input) |
void | fineTuneByGradientDescentLastLayer (const Vec &input) |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declares the class options. | |
Protected Attributes | |
TVec< Vec > | activation_gradients |
gradients of cost wrt the activations (output of params) | |
TVec< Vec > | expectation_gradients |
gradients of cost wrt the expectations (output of layers) | |
Vec | output_gradient |
gradient wrt output activations | |
Private Types | |
typedef PDistribution | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. | |
void | build_layers () |
Build the layers. | |
void | build_params () |
Build the parameters if needed. |
Does the same thing as Hinton's deep belief nets.
Definition at line 60 of file GaussianDBNRegression.h.
typedef PDistribution PLearn::GaussianDBNRegression::inherited [private] |
Reimplemented from PLearn::PDistribution.
Definition at line 62 of file GaussianDBNRegression.h.
PLearn::GaussianDBNRegression::GaussianDBNRegression | ( | ) |
Default constructor.
Definition at line 64 of file GaussianDBNRegression.cc.
References PLearn::PLearner::random_gen.
: learning_rate(0.), weight_decay(0.), use_sample_rather_than_expectation_in_positive_phase_statistics(false) { random_gen = new PRandom(); }
string PLearn::GaussianDBNRegression::_classname_ | ( | ) | [static] |
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
OptionList & PLearn::GaussianDBNRegression::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
RemoteMethodMap & PLearn::GaussianDBNRegression::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
Object * PLearn::GaussianDBNRegression::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
StaticInitializer GaussianDBNRegression::_static_initializer_ & PLearn::GaussianDBNRegression::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
void PLearn::GaussianDBNRegression::build | ( | ) | [virtual] |
Simply calls inherited::build() then build_().
Reimplemented from PLearn::PDistribution.
Definition at line 156 of file GaussianDBNRegression.cc.
References PLearn::PDistribution::build(), and build_().
{ // ### Nothing to add here, simply calls build_(). inherited::build(); build_(); }
void PLearn::GaussianDBNRegression::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::PDistribution.
Definition at line 166 of file GaussianDBNRegression.cc.
References build_layers(), build_params(), PLearn::endl(), fine_tuning_method, initialization_method, layers, PLearn::TVec< T >::length(), PLearn::lowerstring(), n_layers, PLERROR, and training_schedule.
Referenced by build().
{ MODULE_LOG << "build_() called" << endl; n_layers = layers.length(); if( n_layers <= 1 ) return; // check value of initialization_method string im = lowerstring( initialization_method ); if( im == "" || im == "uniform_sqrt" ) initialization_method = "uniform_sqrt"; else if( im == "uniform_linear" ) initialization_method = im; else if( im == "zero" ) initialization_method = im; else PLERROR( "RBMParameters::build_ - initialization_method\n" "\"%s\" unknown.\n", initialization_method.c_str() ); MODULE_LOG << " initialization_method = \"" << initialization_method << "\"" << endl; // check value of fine_tuning_method string ftm = lowerstring( fine_tuning_method ); if( ftm == "" | ftm == "none" ) fine_tuning_method = ""; else if( ftm == "cd" | ftm == "contrastive_divergence" ) fine_tuning_method = "CD"; else if( ftm == "egd" | ftm == "error_gradient_descent" ) fine_tuning_method = "EGD"; else if( ftm == "ws" | ftm == "wake_sleep" ) fine_tuning_method = "WS"; else PLERROR( "GaussianDBNRegression::build_ - fine_tuning_method \"%s\"\n" "is unknown.\n", fine_tuning_method.c_str() ); MODULE_LOG << " fine_tuning_method = \"" << fine_tuning_method << "\"" << endl; //TODO: build structure to store gradients during gradient descent if( training_schedule.length() != n_layers ) training_schedule = TVec<int>( n_layers, 1000000 ); MODULE_LOG << " training_schedule = " << training_schedule << endl; MODULE_LOG << endl; build_layers(); build_params(); }
void PLearn::GaussianDBNRegression::build_layers | ( | ) | [private] |
Build the layers.
Definition at line 213 of file GaussianDBNRegression.cc.
References PLearn::endl(), i, PLearn::PLearner::inputsize(), PLearn::PLearner::inputsize_, last_layer, layers, n_layers, PLearn::PDistribution::n_predicted, PLearn::PDistribution::n_predictor, PLASSERT, PLearn::PLearner::random_gen, setPredictorPredictedSizes(), and target_layer.
Referenced by build_().
{ MODULE_LOG << "build_layers() called" << endl; if( inputsize_ >= 0 ) { PLASSERT( layers[0]->size + target_layer->size == inputsize() ); setPredictorPredictedSizes( layers[0]->size, target_layer->size, false ); MODULE_LOG << " n_predictor = " << n_predictor << endl; MODULE_LOG << " n_predicted = " << n_predicted << endl; } for( int i=0 ; i<n_layers ; i++ ) layers[i]->random_gen = random_gen; target_layer->random_gen = random_gen; last_layer = layers[n_layers-1]; }
void PLearn::GaussianDBNRegression::build_params | ( | ) | [private] |
Build the parameters if needed.
Definition at line 233 of file GaussianDBNRegression.cc.
References activation_gradients, PLearn::endl(), expectation_gradients, i, initialization_method, input_params, last_layer, layers, learning_rate, PLearn::TVec< T >::length(), n_layers, PLearn::PDistribution::n_predicted, output_gradient, params, PLERROR, PLearn::PLearner::random_gen, PLearn::TVec< T >::resize(), target_layer, and target_params.
Referenced by build_().
{ MODULE_LOG << "build_params() called" << endl; if( params.length() == 0 ) { input_params = new RBMQLParameters() ; params.resize( n_layers-1 ); for( int i=1 ; i<n_layers-1 ; i++ ) params[i] = new RBMLLParameters(); // params[0] is not being using, it is not being created } else if( params.length() != n_layers-1 ) PLERROR( "GaussianDBNRegression::build_params - params.length() should\n" "be equal to layers.length()-1 (%d != %d).\n", params.length(), n_layers-1 ); activation_gradients.resize( n_layers+1 ); expectation_gradients.resize( n_layers+1 ); output_gradient.resize( n_predicted ); input_params->down_units_types = layers[0]->units_types; input_params->up_units_types = layers[1]->units_types; input_params->learning_rate = learning_rate; input_params->initialization_method = initialization_method; input_params->random_gen = random_gen; input_params->build(); activation_gradients[0].resize( input_params->down_layer_size ); expectation_gradients[0].resize( input_params->down_layer_size ); for( int i=1 ; i<n_layers-1 ; i++ ) { //TODO: call changeOptions instead params[i]->down_units_types = layers[i]->units_types; params[i]->up_units_types = layers[i+1]->units_types; params[i]->learning_rate = learning_rate; params[i]->initialization_method = initialization_method; params[i]->random_gen = random_gen; params[i]->build(); activation_gradients[i].resize( params[i]->down_layer_size ); expectation_gradients[i].resize( params[i]->down_layer_size ); } if( target_layer && !target_params ) target_params = new RBMLQParameters(); //TODO: call changeOptions instead target_params->down_units_types = last_layer->units_types; target_params->up_units_types = target_layer->units_types; target_params->learning_rate = learning_rate; target_params->initialization_method = initialization_method; target_params->random_gen = random_gen; target_params->build(); }
Return cdf: P(Y<y | x).
Reimplemented from PLearn::PDistribution.
Definition at line 331 of file GaussianDBNRegression.cc.
References PLERROR.
{ PLERROR("cdf not implemented for GaussianDBNRegression"); return 0; }
string PLearn::GaussianDBNRegression::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
Referenced by train().
void PLearn::GaussianDBNRegression::computeCostsFromOutputs | ( | const Vec & | input, |
const Vec & | output, | ||
const Vec & | target, | ||
Vec & | costs | ||
) | const [virtual] |
Compute a cost, depending on the type of the first output : if it is the density or the log-density: NLL if it is the expectation: NLL and class error.
Reimplemented from PLearn::PDistribution.
Definition at line 837 of file GaussianDBNRegression.cc.
References c, PLearn::PDistribution::computeCostsFromOutputs(), i, PLearn::TVec< T >::length(), PLearn::PDistribution::outputs_def, PLearn::PDistribution::predicted_part, PLearn::TVec< T >::resize(), PLearn::PDistribution::splitCond(), and PLearn::square().
{ char c = outputs_def[0]; if( c == 'l' || c == 'd' ) inherited::computeCostsFromOutputs(input, output, target, costs); else if( c == 'e' ) { costs.resize( 1 ); costs[0] = .0 ; splitCond(input); int output_size = output.length(); for(int i=0 ; i<output_size ; ++i) { costs[0] += square(output[i] - predicted_part[i]) ; } costs[0] /= output_size ; } }
void PLearn::GaussianDBNRegression::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declares the class options.
Reimplemented from PLearn::PDistribution.
Definition at line 75 of file GaussianDBNRegression.cc.
References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::PDistribution::declareOptions(), fine_tuning_method, initialization_method, input_params, layers, learning_rate, PLearn::OptionBase::learntoption, n_layers, params, target_layer, target_params, training_schedule, use_sample_rather_than_expectation_in_positive_phase_statistics, and weight_decay.
{ declareOption(ol, "learning_rate", &GaussianDBNRegression::learning_rate, OptionBase::buildoption, "Learning rate"); declareOption(ol, "weight_decay", &GaussianDBNRegression::weight_decay, OptionBase::buildoption, "Weight decay"); declareOption(ol, "initialization_method", &GaussianDBNRegression::initialization_method, OptionBase::buildoption, "The method used to initialize the weights:\n" " - \"uniform_linear\" = a uniform law in [-1/d, 1/d]\n" " - \"uniform_sqrt\" = a uniform law in [-1/sqrt(d)," " 1/sqrt(d)]\n" " - \"zero\" = all weights are set to 0,\n" "where d = max( up_layer_size, down_layer_size ).\n"); declareOption(ol, "training_schedule", &GaussianDBNRegression::training_schedule, OptionBase::buildoption, "Number of examples to use during each of the different" " greedy\n" "steps of the training phase.\n"); declareOption(ol, "fine_tuning_method", &GaussianDBNRegression::fine_tuning_method, OptionBase::buildoption, "Method for fine-tuning the whole network after greedy" " learning.\n" "One of:\n" " - \"none\"\n" " - \"CD\" or \"contrastive_divergence\"\n" " - \"EGD\" or \"error_gradient_descent\"\n" " - \"WS\" or \"wake_sleep\".\n"); declareOption(ol, "layers", &GaussianDBNRegression::layers, OptionBase::buildoption, "Layers that learn representations of the input," " unsupervisedly.\n" "layers[0] is input layer.\n"); declareOption(ol, "target_layer", &GaussianDBNRegression::target_layer, OptionBase::buildoption, "Target (or label) layer"); declareOption(ol, "params", &GaussianDBNRegression::params, OptionBase::buildoption, "RBMParameters linking the unsupervised layers.\n" "params[i] links layers[i] and layers[i+1], except for" "params[n_layers-1],\n" "that links layers[n_layers-1] and last_layer.\n"); declareOption(ol, "target_params", &GaussianDBNRegression::target_params, OptionBase::buildoption, "Parameters linking target_layer and last_layer"); declareOption(ol, "input_params", &GaussianDBNRegression::input_params, OptionBase::buildoption, "Parameters linking layer[0] and layer[1]"); declareOption(ol, "use_sample_rather_than_expectation_in_positive_phase_statistics", &GaussianDBNRegression::use_sample_rather_than_expectation_in_positive_phase_statistics, OptionBase::buildoption, "In positive phase statistics use output->sample * input\n" "rather than output->expectation * input.\n"); declareOption(ol, "n_layers", &GaussianDBNRegression::n_layers, OptionBase::learntoption, "Number of unsupervised layers, including input layer"); // Now call the parent class' declareOptions(). inherited::declareOptions(ol); }
static const PPath& PLearn::GaussianDBNRegression::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::PDistribution.
Definition at line 232 of file GaussianDBNRegression.h.
:
//##### Protected Options ###############################################
GaussianDBNRegression * PLearn::GaussianDBNRegression::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
Return probability density p(y | x)
Reimplemented from PLearn::PDistribution.
Definition at line 367 of file GaussianDBNRegression.cc.
References PLearn::argmax(), expectation(), i, PLearn::is_equal(), PLearn::PDistribution::n_predicted, PLASSERT, PLearn::TVec< T >::size(), and PLearn::PDistribution::store_expect.
Referenced by log_density().
{ PLASSERT( y.size() == n_predicted ); // TODO: 'y'[0] devrait plutot etre l'entier "index" lui-meme! int index = argmax( y ); // If y != onehot( index ), then density is 0 if( !is_equal( y[index], 1. ) ) return 0; for( int i=0 ; i<n_predicted ; i++ ) if( !is_equal( y[i], 0 ) && i != index ) return 0; expectation( store_expect ); return store_expect[index]; }
void PLearn::GaussianDBNRegression::expectation | ( | Vec & | mu | ) | const [virtual] |
Return E[Y | x].
Reimplemented from PLearn::PDistribution.
Definition at line 339 of file GaussianDBNRegression.cc.
References i, input_params, last_layer, layers, n_layers, params, PLearn::PDistribution::predicted_size, PLearn::PDistribution::predictor_part, PLearn::TVec< T >::resize(), target_layer, and target_params.
Referenced by density(), fineTuneByGradientDescent(), fineTuneByGradientDescentLastLayer(), greedyStep(), and train().
{ mu.resize( predicted_size ); // Propagate input (predictor_part) until penultimate layer layers[0]->expectation << predictor_part; input_params->setAsDownInput(layers[0]->expectation) ; layers[1]->getAllActivations( (RBMQLParameters*) input_params ); layers[1]->computeExpectation(); for( int i=1 ; i<n_layers-1 ; i++ ) { params[i]->setAsDownInput( layers[i]->expectation ); layers[i+1]->getAllActivations( (RBMLLParameters*) params[i] ); layers[i+1]->computeExpectation(); } target_params->setAsDownInput( last_layer->expectation ); target_layer->getAllActivations( (RBMLQParameters*) target_params ); target_layer->computeExpectation(); mu << target_layer->expectation; }
void PLearn::GaussianDBNRegression::fineTuneByGradientDescent | ( | const Vec & | input | ) | [protected, virtual] |
Definition at line 785 of file GaussianDBNRegression.cc.
References activation_gradients, expectation(), expectation_gradients, i, input_params, layers, n_layers, output_gradient, params, PLearn::PDistribution::predicted_part, PLearn::TVec< T >::resize(), PLearn::TVec< T >::size(), PLearn::PDistribution::splitCond(), target_layer, and target_params.
{ // split input in predictor_part and predicted_part splitCond(input); // compute predicted_part expectation, conditioned on predictor_part // (forward pass) expectation( output_gradient ); int target_size = predicted_part.size() ; expectation_gradients[n_layers].resize(target_size) ; for(int i=0 ; i < target_size ; ++i) { expectation_gradients[n_layers][i] = 2 * (output_gradient[i] - predicted_part[i]) ; } target_layer->bpropUpdate( target_layer->activations, target_layer->expectation, activation_gradients[n_layers] , expectation_gradients[n_layers]) ; target_params->bpropUpdate( layers[n_layers-1]->expectation, target_layer->activations, expectation_gradients[n_layers-1], activation_gradients[n_layers] ); for( int i=n_layers-1 ; i>1 ; i-- ) { layers[i]->bpropUpdate( layers[i]->activations, layers[i]->expectation, activation_gradients[i], expectation_gradients[i] ); params[i-1]->bpropUpdate( layers[i-1]->expectation, layers[i]->activations, expectation_gradients[i-1], activation_gradients[i] ); } layers[1]->bpropUpdate( layers[1]->activations, layers[1]->expectation, activation_gradients[1], expectation_gradients[1] ); input_params->bpropUpdate( layers[0]->expectation, layers[1]->activations, expectation_gradients[0], activation_gradients[1] ); }
void PLearn::GaussianDBNRegression::fineTuneByGradientDescentLastLayer | ( | const Vec & | input | ) | [protected] |
Definition at line 756 of file GaussianDBNRegression.cc.
References activation_gradients, expectation(), expectation_gradients, i, layers, n_layers, output_gradient, PLearn::PDistribution::predicted_part, PLearn::TVec< T >::resize(), PLearn::TVec< T >::size(), PLearn::PDistribution::splitCond(), target_layer, and target_params.
{ // split input in predictor_part and predicted_part splitCond(input); // compute predicted_part expectation, conditioned on predictor_part // (forward pass) expectation( output_gradient ); int target_size = predicted_part.size() ; expectation_gradients[n_layers].resize(target_size) ; for(int i=0 ; i < target_size ; ++i) { expectation_gradients[n_layers][i] = 2 * (output_gradient[i] - predicted_part[i]) ; } target_layer->bpropUpdate( target_layer->activations, target_layer->expectation, activation_gradients[n_layers] , expectation_gradients[n_layers]) ; target_params->bpropUpdate( layers[n_layers-1]->expectation, target_layer->activations, expectation_gradients[n_layers-1], activation_gradients[n_layers] ); }
void PLearn::GaussianDBNRegression::forget | ( | ) | [virtual] |
(Re-)initializes the PDistribution in its fresh state (that state may depend on the 'seed' option).
And sets 'stage' back to 0 (this is the stage of a fresh learner!). ### You may remove this method if your distribution does not ### implement it.
A typical forget() method should do the following:
Reimplemented from PLearn::PDistribution.
Definition at line 297 of file GaussianDBNRegression.cc.
References PLearn::endl(), i, input_params, layers, n_layers, params, PLearn::PDistribution::resetGenerator(), PLearn::PLearner::seed_, PLearn::PLearner::stage, target_layer, and target_params.
{ MODULE_LOG << "forget() called" << endl; resetGenerator(seed_); input_params->forget() ; for( int i=1 ; i<n_layers-1 ; i++ ) params[i]->forget(); for( int i=0 ; i<n_layers ; i++ ) layers[i]->reset(); target_params->forget(); target_layer->reset(); stage = 0; }
void PLearn::GaussianDBNRegression::generate | ( | Vec & | y | ) | const [virtual] |
Return a pseudo-random sample generated from the conditional distribution, of density p(y | x).
Reimplemented from PLearn::PDistribution.
Definition at line 323 of file GaussianDBNRegression.cc.
References PLERROR.
{ PLERROR("generate not implemented for GaussianDBNRegression"); }
OptionList & PLearn::GaussianDBNRegression::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
OptionMap & PLearn::GaussianDBNRegression::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
RemoteMethodMap & PLearn::GaussianDBNRegression::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
TVec< string > PLearn::GaussianDBNRegression::getTestCostNames | ( | ) | const [virtual] |
Return [ "NLL" ] (the only cost computed by a PDistribution).
Reimplemented from PLearn::PDistribution.
Definition at line 861 of file GaussianDBNRegression.cc.
References PLearn::TVec< T >::append(), c, and PLearn::PDistribution::outputs_def.
{ char c = outputs_def[0]; TVec<string> result; if( c == 'l' || c == 'd' ) result.append( "NLL" ); else if( c == 'e' ) { result.append( "MSE" ); } return result; }
void PLearn::GaussianDBNRegression::greedyStep | ( | const Vec & | predictor, |
int | params_index | ||
) | [protected, virtual] |
Definition at line 677 of file GaussianDBNRegression.cc.
References PLearn::RBMQLParameters::accumulateNegStats(), PLearn::RBMQLParameters::accumulatePosStats(), expectation(), i, input_params, layers, params, PLearn::sample(), PLearn::RBMParameters::setAsDownInput(), PLearn::RBMParameters::setAsUpInput(), PLearn::RBMQLParameters::update(), and use_sample_rather_than_expectation_in_positive_phase_statistics.
Referenced by train().
{ // deterministic propagation until we reach index layers[0]->expectation << predictor; input_params->setAsDownInput( layers[0]->expectation ); layers[1]->getAllActivations( (RBMQLParameters*) input_params ); layers[1]->computeExpectation(); for( int i=1 ; i<index ; i++ ) { params[i]->setAsDownInput( layers[i]->expectation ); layers[i+1]->getAllActivations( (RBMLLParameters*) params[i] ); layers[i+1]->computeExpectation(); } // positive phase if (index == 0) { input_params->setAsDownInput( layers[index]->expectation ); layers[index+1]->getAllActivations((RBMQLParameters*) input_params); layers[index+1]->computeExpectation(); layers[index+1]->generateSample(); if (use_sample_rather_than_expectation_in_positive_phase_statistics) input_params->accumulatePosStats(layers[index]->expectation, layers[index+1]->sample ); else input_params->accumulatePosStats(layers[index]->expectation, layers[index+1]->expectation ); // down propagation input_params->setAsUpInput( layers[index+1]->sample ); layers[index]->getAllActivations( (RBMQLParameters*) input_params ); // negative phase layers[index]->generateSample(); input_params->setAsDownInput( layers[index]->sample ); layers[index+1]->getAllActivations((RBMQLParameters*) input_params); layers[index+1]->computeExpectation(); input_params->accumulateNegStats( layers[index]->sample, layers[index+1]->expectation ); // update input_params->update(); } else { params[index]->setAsDownInput( layers[index]->expectation ); layers[index+1]->getAllActivations((RBMLLParameters*) params[index]); layers[index+1]->computeExpectation(); layers[index+1]->generateSample(); if (use_sample_rather_than_expectation_in_positive_phase_statistics) params[index]->accumulatePosStats(layers[index]->expectation, layers[index+1]->sample ); else params[index]->accumulatePosStats(layers[index]->expectation, layers[index+1]->expectation ); // down propagation params[index]->setAsUpInput( layers[index+1]->sample ); layers[index]->getAllActivations( (RBMLLParameters*) params[index] ); // negative phase layers[index]->generateSample(); params[index]->setAsDownInput( layers[index]->sample ); layers[index+1]->getAllActivations((RBMLLParameters*) params[index]); layers[index+1]->computeExpectation(); params[index]->accumulateNegStats( layers[index]->sample, layers[index+1]->expectation ); // update params[index]->update(); } }
Return log of probability density log(p(y | x)).
Reimplemented from PLearn::PDistribution.
Definition at line 389 of file GaussianDBNRegression.cc.
References density(), and pl_log.
void PLearn::GaussianDBNRegression::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::PDistribution.
Definition at line 413 of file GaussianDBNRegression.cc.
References PLearn::deepCopyField(), input_params, last_layer, layers, PLearn::PDistribution::makeDeepCopyFromShallowCopy(), params, target_layer, target_params, and training_schedule.
{ inherited::makeDeepCopyFromShallowCopy(copies); deepCopyField(layers, copies); deepCopyField(last_layer, copies); deepCopyField(target_layer, copies); deepCopyField(params, copies); deepCopyField(input_params, copies); deepCopyField(target_params, copies); deepCopyField(training_schedule, copies); }
void PLearn::GaussianDBNRegression::setPredictor | ( | const Vec & | predictor, |
bool | call_parent = true |
||
) | const [virtual] |
Set the value for the predictor part of a conditional probability.
Reimplemented from PLearn::PDistribution.
Definition at line 429 of file GaussianDBNRegression.cc.
References PLearn::PDistribution::setPredictor().
{ if (call_parent) inherited::setPredictor(predictor, true); // ### Add here any specific code required by your subclass. }
bool PLearn::GaussianDBNRegression::setPredictorPredictedSizes | ( | int | the_predictor_size, |
int | the_predicted_size, | ||
bool | call_parent = true |
||
) | [virtual] |
Generates a pseudo-random sample x from the reversed conditional distribution, of density p(x | y) (and NOT p(y | x)).
i.e., generates a "predictor" part given a "predicted" part, regardless of any previously set predictor. Set the 'predictor' and 'predicted' sizes for this distribution.
Reimplemented from PLearn::PDistribution.
Definition at line 440 of file GaussianDBNRegression.cc.
References layers, PLearn::PDistribution::n_predicted, PLearn::PDistribution::n_predictor, PLERROR, PLearn::PDistribution::setPredictorPredictedSizes(), PLearn::TVec< T >::size(), and target_layer.
Referenced by build_layers().
{ bool sizes_have_changed = false; if (call_parent) sizes_have_changed = inherited::setPredictorPredictedSizes( the_predictor_size, the_predicted_size, true); // ### Add here any specific code required by your subclass. if( the_predictor_size >= 0 && the_predictor_size != layers[0]->size || the_predicted_size >= 0 && the_predicted_size != target_layer->size ) PLERROR( "GaussianDBNRegression::setPredictorPredictedSizes - \n" "n_predictor should be equal to layer[0]->size (%d)\n" "n_predicted should be equal to target_layer->size (%d).\n", layers[0]->size, target_layer->size ); n_predictor = layers[0]->size; n_predicted = target_layer->size; // Returned value. return sizes_have_changed; }
Return survival function: P(Y>y | x).
Reimplemented from PLearn::PDistribution.
Definition at line 397 of file GaussianDBNRegression.cc.
References PLERROR.
{ PLERROR("survival_fn not implemented for GaussianDBNRegression"); return 0; }
void PLearn::GaussianDBNRegression::train | ( | ) | [virtual] |
The role of the train method is to bring the learner up to stage == nstages, updating the train_stats collector with training costs measured on-line in the process.
Reimplemented from PLearn::PDistribution.
Definition at line 468 of file GaussianDBNRegression.cc.
References classname(), PLearn::endl(), expectation(), PLearn::VMat::getExample(), greedyStep(), i, PLearn::PLearner::initTrain(), PLearn::PLearner::inputsize(), j, last_layer, PLearn::VMat::length(), PLearn::linearRegression(), PLearn::min(), n_layers, PLearn::PDistribution::n_predicted, PLearn::PDistribution::n_predictor, PLearn::PLearner::nstages, PLearn::PLearner::report_progress, PLearn::sample(), PLearn::PDistribution::splitCond(), PLearn::PLearner::stage, PLearn::TVec< T >::subVec(), target_params, PLearn::PLearner::targetsize(), PLearn::tostring(), PLearn::PLearner::train_set, PLearn::PLearner::train_stats, training_schedule, and PLearn::ProgressBar::update().
{ MODULE_LOG << "train() called" << endl; // The role of the train method is to bring the learner up to // stage==nstages, updating train_stats with training costs measured // on-line in the process. /* TYPICAL CODE: static Vec input; // static so we don't reallocate memory each time... static Vec target; // (but be careful that static means shared!) input.resize(inputsize()); // the train_set's inputsize() target.resize(targetsize()); // the train_set's targetsize() real weight; // This generic PLearner method does a number of standard stuff useful for // (almost) any learner, and return 'false' if no training should take // place. See PLearner.h for more details. if (!initTrain()) return; while(stage<nstages) { // clear statistics of previous epoch train_stats->forget(); //... train for 1 stage, and update train_stats, // using train_set->getExample(input, target, weight) // and train_stats->update(train_costs) ++stage; train_stats->finalize(); // finalize statistics for this epoch } */ Vec input( inputsize() ); Vec target( targetsize() ); // unused real weight; // unused if( !initTrain() ) { MODULE_LOG << "train() aborted" << endl; return; } int nsamples = train_set->length(); MODULE_LOG << " nsamples = " << nsamples << endl; // Let's define stage and nstages: // - 0: fresh state, nothing is done // - 1..n_layers-2: params[stage-1] is trained // - n_layers-1: joint_params is trained (including params[n_layers-2]) // - n_layers: after the fine tuning MODULE_LOG << "initial stage = " << stage << endl; MODULE_LOG << "objective: nstages = " << nstages << endl; // clear stats of previous epoch train_stats->forget(); for(int layer=0 ; layer<n_layers-1 ; ++layer) { MODULE_LOG << "Training parameters between layers " << layer << " and " << layer+1 << endl; // this progress bar shows the number of loops through the whole // training set ProgressBar* pb = 0; int end_stage = min( training_schedule[layer], nstages ); if( report_progress && stage < end_stage ) { pb = new ProgressBar( "Training layer "+tostring(layer)+ "of" + classname(), end_stage - stage ); } for( ; stage < end_stage ; stage++ ) { // sample is the index in the training set int sample = stage % train_set->length(); train_set->getExample(sample, input, target, weight); greedyStep( input.subVec(0, n_predictor), layer ); if( pb ) { if( layer == 0 ) pb->update(stage + 1); else pb->update(stage - training_schedule[layer-1] + 1); } Mat inputs(train_set.length() , n_predictor) ; Mat outputs(train_set.length() , n_predicted); Mat theta(1 + n_predictor , n_predicted) ; Vec output_value(n_predicted) ; for(int i=0 ; i<train_set.length() ; ++i) { train_set->getExample(i, input, target, weight); // split input in predictor_part and predicted_part splitCond(input); // compute predicted_part expectation, conditioned on predictor_part // (forward pass) expectation( output_value ); for(int j=0 ; j<n_predictor ; ++j) { inputs[i][j] = last_layer->expectation[j] ; // cout << last_layer->expectation[j] << " " ; } for(int j=0 ; j<n_predicted ; ++j) { outputs[i][j] = input[j+n_predictor] ; } } // pout << "inputs " << endl << inputs << endl ; // pout << "outputs " << endl << outputs << endl ; linearRegression(inputs,outputs,0.0,theta); // init the a_i term target_params->up_units_params[1].fill(1) ; // pout << "Theta" << theta << endl ; // set the bias (b_i) for(int i=0 ; i<n_predicted ; ++i) { target_params->up_units_params[0][i] = - 2.0 * theta[i][0] ; } for(int i=0 ; i<n_predicted ; ++i) { for(int j=0 ; j<n_predictor ; ++j) { target_params->weights[i][j] = -2.0 * theta[j][i+1] ; } } } } /* MODULE_LOG << "Fine-tuning all parameters, using method " << fine_tuning_method << endl; if( fine_tuning_method == "" ) // do nothing sample += n_samples_to_see; else if( fine_tuning_method == "EGD" ) { if( report_progress ) pb = new ProgressBar( "Training all " + classname() + " parameters by fine tuning", n_samples_to_see ); */ /* pout << "==================" << endl << "Before update:" << endl << "up: " << joint_params->up_units_params << endl << "weights: " << endl << joint_params->weights << endl << "down: " << joint_params->down_units_params << endl << endl; // */ // linear regression for last weights /* int begin_sample = sample; int end_sample = begin_sample + n_samples_to_see; for( ; sample < end_sample ; sample++ ) { // sample is the index in the training set int i = sample % train_set->length(); train_set->getExample(i, input, target, weight); fineTuneByGradientDescentLastLayer( input ); if( pb ) pb->update( sample - begin_sample + 1 ); } sample = begin_sample ; for( ; sample < 100 ; sample++ ) { // sample is the index in the training set int i = sample % train_set->length(); train_set->getExample(i, input, target, weight); fineTuneByGradientDescent( input ); if( pb ) pb->update( sample - begin_sample + 1 ); } */ /* pout << "-------" << endl << "After update:" << endl << "up: " << joint_params->up_units_params << endl << "weights: " << endl << joint_params->weights << endl << "down: " << joint_params->down_units_params << endl << endl; // */ train_stats->finalize(); // finalize statistics for this epoch MODULE_LOG << endl; }
void PLearn::GaussianDBNRegression::variance | ( | Mat & | cov | ) | const [virtual] |
Reimplemented from PLearn::PDistribution.
Definition at line 405 of file GaussianDBNRegression.cc.
References PLERROR.
{ PLERROR("variance not implemented for GaussianDBNRegression"); }
Reimplemented from PLearn::PDistribution.
Definition at line 232 of file GaussianDBNRegression.h.
TVec< Vec > PLearn::GaussianDBNRegression::activation_gradients [mutable, protected] |
gradients of cost wrt the activations (output of params)
Definition at line 248 of file GaussianDBNRegression.h.
Referenced by build_params(), fineTuneByGradientDescent(), and fineTuneByGradientDescentLastLayer().
TVec< Vec > PLearn::GaussianDBNRegression::expectation_gradients [mutable, protected] |
gradients of cost wrt the expectations (output of layers)
Definition at line 251 of file GaussianDBNRegression.h.
Referenced by build_params(), fineTuneByGradientDescent(), and fineTuneByGradientDescentLastLayer().
Method for fine-tuning the whole network after greedy learning.
One of:
Definition at line 117 of file GaussianDBNRegression.h.
Referenced by build_(), and declareOptions().
The method used to initialize the weights:
Definition at line 81 of file GaussianDBNRegression.h.
Referenced by build_(), build_params(), and declareOptions().
Parameters linking input layer[0] and layer[1].
Definition at line 102 of file GaussianDBNRegression.h.
Referenced by build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), forget(), greedyStep(), and makeDeepCopyFromShallowCopy().
last_layer is layer[n_layers-1]
Definition at line 92 of file GaussianDBNRegression.h.
Referenced by build_layers(), build_params(), expectation(), makeDeepCopyFromShallowCopy(), and train().
Layers that learn representations of the input, layers[0] is input layer, layers[n_layers-1] is last layer.
Definition at line 89 of file GaussianDBNRegression.h.
Referenced by build_(), build_layers(), build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), fineTuneByGradientDescentLastLayer(), forget(), greedyStep(), makeDeepCopyFromShallowCopy(), and setPredictorPredictedSizes().
### declare public option fields (such as build options) here Start your comments with Doxygen-compatible comments such as //!
The learning rate
Definition at line 71 of file GaussianDBNRegression.h.
Referenced by build_params(), and declareOptions().
Number of layers, including input layer and last layer, but not target layer.
Definition at line 85 of file GaussianDBNRegression.h.
Referenced by build_(), build_layers(), build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), fineTuneByGradientDescentLastLayer(), forget(), and train().
Vec PLearn::GaussianDBNRegression::output_gradient [mutable, protected] |
gradient wrt output activations
Definition at line 254 of file GaussianDBNRegression.h.
Referenced by build_params(), fineTuneByGradientDescent(), and fineTuneByGradientDescentLastLayer().
RBMParameters linking the unsupervised layers.
params[i] links layers[i] and layers[i+1], i>0
Definition at line 99 of file GaussianDBNRegression.h.
Referenced by build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), forget(), greedyStep(), and makeDeepCopyFromShallowCopy().
Target (or label) layer.
Definition at line 95 of file GaussianDBNRegression.h.
Referenced by build_layers(), build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), fineTuneByGradientDescentLastLayer(), forget(), makeDeepCopyFromShallowCopy(), and setPredictorPredictedSizes().
Parameters linking target_layer and last_layer.
Definition at line 105 of file GaussianDBNRegression.h.
Referenced by build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), fineTuneByGradientDescentLastLayer(), forget(), makeDeepCopyFromShallowCopy(), and train().
Number of examples to use during each of the different greedy steps of the training phase.
Definition at line 109 of file GaussianDBNRegression.h.
Referenced by build_(), declareOptions(), makeDeepCopyFromShallowCopy(), and train().
Definition at line 119 of file GaussianDBNRegression.h.
Referenced by declareOptions(), and greedyStep().
The weight decay.
Definition at line 74 of file GaussianDBNRegression.h.
Referenced by declareOptions().