PLearn 0.1
|
Trains a stack of OnlineLearningModule, which are layers. More...
#include <StackedModulesLearner.h>
Public Member Functions | |
StackedModulesLearner () | |
Default constructor. | |
virtual int | outputsize () const |
Returns the size of this learner's output, (which typically may depend on its inputsize(), targetsize() and set options). | |
virtual void | forget () |
(Re-)initializes the PLearner in its fresh state (that state may depend on the 'seed' option) and sets 'stage' back to 0 (this is the stage of a fresh learner!). | |
virtual void | train () |
The role of the train method is to bring the learner up to stage==nstages, updating the train_stats collector with training costs measured on-line in the process. | |
virtual void | computeOutput (const Vec &input, Vec &output) const |
Computes the output from the input. | |
virtual void | computeCostsFromOutputs (const Vec &input, const Vec &output, const Vec &target, Vec &costs) const |
Computes the costs from already computed output. | |
virtual TVec< std::string > | getTestCostNames () const |
Returns the names of the costs computed by computeCostsFromOutpus (and thus the test method). | |
virtual TVec< std::string > | getTrainCostNames () const |
Returns the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats. | |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual StackedModulesLearner * | deepCopy (CopiesMap &copies) const |
virtual void | build () |
Finish building the object; just call inherited::build followed by build_() | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transforms a shallow copy into a deep copy. | |
Static Public Member Functions | |
static string | _classname_ () |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
TVec< PP< OnlineLearningModule > > | modules |
Layers of the learner. | |
TVec< string > | cost_funcs |
Names of the cost functions to apply on output. | |
string | hessian_estimation |
Estimation of the second-order terms. | |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declares the class options. | |
Protected Attributes | |
int | nmodules |
Number of module layers. | |
TVec< PP< OnlineLearningModule > > | cost_modules |
Modules that will compute the costs. | |
int | ncosts |
Number of cost modules. | |
TVec< Vec > | values |
stores the input and output values of the functions | |
TVec< Vec > | gradients |
stores the gradients | |
TVec< Vec > | diag_hessians |
stores the diagonal of Hessians | |
Private Types | |
typedef PLearner | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. | |
void | buildOptions () |
void | buildCostFunctions () |
void | buildLayers () |
Trains a stack of OnlineLearningModule, which are layers.
The OnlineLearningModule's are disposed like superposed layers: outputs of module i are the inputs of module (i+1), the last layer is the output layer. Another TVec of modules contains the cost modules. The first one is used during the training phase as the cost to minimize, the other ones are only measured.
Definition at line 63 of file StackedModulesLearner.h.
typedef PLearner PLearn::StackedModulesLearner::inherited [private] |
Reimplemented from PLearn::PLearner.
Definition at line 65 of file StackedModulesLearner.h.
PLearn::StackedModulesLearner::StackedModulesLearner | ( | ) |
Default constructor.
Definition at line 61 of file StackedModulesLearner.cc.
References PLearn::PLearner::random_gen.
: cost_funcs( 1, "mse" ), hessian_estimation( "none" ), nmodules( 0 ), ncosts( 1 ) { random_gen = new PRandom(); }
string PLearn::StackedModulesLearner::_classname_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 59 of file StackedModulesLearner.cc.
OptionList & PLearn::StackedModulesLearner::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 59 of file StackedModulesLearner.cc.
RemoteMethodMap & PLearn::StackedModulesLearner::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 59 of file StackedModulesLearner.cc.
Reimplemented from PLearn::PLearner.
Definition at line 59 of file StackedModulesLearner.cc.
Object * PLearn::StackedModulesLearner::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 59 of file StackedModulesLearner.cc.
StaticInitializer StackedModulesLearner::_static_initializer_ & PLearn::StackedModulesLearner::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 59 of file StackedModulesLearner.cc.
void PLearn::StackedModulesLearner::build | ( | ) | [virtual] |
Finish building the object; just call inherited::build followed by build_()
Reimplemented from PLearn::PLearner.
Definition at line 266 of file StackedModulesLearner.cc.
References PLearn::PLearner::build(), and build_().
{ inherited::build(); build_(); }
void PLearn::StackedModulesLearner::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::PLearner.
Definition at line 115 of file StackedModulesLearner.cc.
References buildCostFunctions(), buildLayers(), buildOptions(), PLearn::PLearner::inputsize_, PLearn::PLearner::random_gen, and PLearn::PLearner::seed_.
Referenced by build().
{ // ### This method should do the real building of the object, // ### according to set 'options', in *any* situation. // ### Typical situations include: // ### - Initial building of an object from a few user-specified options // ### - Building of a "reloaded" object: i.e. from the complete set of // ### all serialised options. // ### - Updating or "re-building" of an object after a few "tuning" // ### options have been modified. // ### You should assume that the parent class' build_() has already been // ### called. // build the layers buildOptions(); // initialize random generator from seed random_gen->manual_seed( seed_ ); // if train_set is not set, we don't know inputsize nor targetsize if( inputsize_ >= 0 ) // we don't use inputsize() because it crashes if <0 buildLayers(); // build cost functions buildCostFunctions(); }
void PLearn::StackedModulesLearner::buildCostFunctions | ( | ) | [private] |
Definition at line 183 of file StackedModulesLearner.cc.
References cost_funcs, cost_modules, hessian_estimation, i, ncosts, and outputsize().
Referenced by build_().
{ // build cost functions for( int i=0 ; i<ncosts ; i++ ) { string cf = cost_funcs[i]; if( cf == "mse" ) { PP<SquaredErrModule> p_mse; // if the first module is not already a SquaredErrModule, // allocate a new one if( !(p_mse = dynamic_cast<SquaredErrModule*>( (OnlineLearningModule*) cost_modules[i] )) ) { p_mse = new SquaredErrModule(); cost_modules[i] = p_mse; } } else if( cf == "NLL" ) { PP<NLLErrModule> p_nll; // if the first module is not already a NLLErrModule, // allocate a new one if( !(p_nll = dynamic_cast<NLLErrModule*>( (OnlineLearningModule*) cost_modules[i] )) ) { p_nll = new NLLErrModule(); cost_modules[i] = p_nll; } } cost_modules[i]->input_size = outputsize(); if( hessian_estimation == "diag" ) cost_modules[i]->estimate_simpler_diag_hessian = false; else cost_modules[i]->estimate_simpler_diag_hessian = true; cost_modules[i]->build(); } }
void PLearn::StackedModulesLearner::buildLayers | ( | ) | [private] |
Definition at line 223 of file StackedModulesLearner.cc.
References diag_hessians, gradients, hessian_estimation, i, PLearn::PLearner::inputsize(), modules, nmodules, PLWARNING, PLearn::TVec< T >::resize(), and values.
Referenced by build_().
{ // first values will be "input" values int size = inputsize(); values.resize( nmodules+1 ); values[0].resize( size ); gradients.resize( nmodules+1 ); gradients[0].resize( size ); if( hessian_estimation != "none" ) { diag_hessians.resize( nmodules+1 ); diag_hessians[0].resize( size ); } for( int i=0 ; i<nmodules ; i++ ) { PP<OnlineLearningModule> p_module = modules[i]; if( p_module->input_size != size ) { PLWARNING( "StackedModulesLearner::buildLayers(): module '%d'\n" "has an input size of '%d', but previous layer's output" " size\n" "is '%d'. Resizing module '%d'.\n", i, p_module->input_size, size, i); p_module->input_size = size; } if( hessian_estimation == "diag" ) p_module->estimate_simpler_diag_hessian = false; else p_module->estimate_simpler_diag_hessian = true; p_module->build(); size = p_module->output_size; values[i+1].resize( size ); gradients[i+1].resize( size ); if( hessian_estimation != "none" ) diag_hessians[i+1].resize( size ); } }
void PLearn::StackedModulesLearner::buildOptions | ( | ) | [private] |
Definition at line 142 of file StackedModulesLearner.cc.
References cost_funcs, cost_modules, hessian_estimation, i, PLearn::TVec< T >::length(), PLearn::lowerstring(), modules, ncosts, nmodules, PLERROR, PLWARNING, and PLearn::TVec< T >::resize().
Referenced by build_().
{ nmodules = modules.length(); ncosts = cost_funcs.length(); // check length of cost_modules int ncm = cost_modules.length(); if( ncm != 0 && ncm != ncosts ) PLWARNING( "StackedModulesLearner::buildOptions(): 'cost_modules' is" " set,\n" "but its length differ from 'cost_funcs.length()'" " (%d != %d).\n", ncm, ncosts); cost_modules.resize( ncosts ); // check string's values for( int i=0 ; i<ncosts ; i++ ) { string cf = lowerstring( cost_funcs[i] ); if( cf == "nll" ) cost_funcs[i] = "NLL"; else if( cf == "mse" || cf == "" ) cost_funcs[i] = "mse"; else PLERROR( "StackedModulesLearner::buildOptions(): cost function\n" "'%s' is unknown.\n", cost_funcs[i].c_str() ); } // hessian estimation string h_est = lowerstring( hessian_estimation ); if( h_est == "none" || h_est == "" ) hessian_estimation = "none"; else if( h_est == "diag" ) hessian_estimation = h_est; else if( h_est == "simpler_diag" ) hessian_estimation = h_est; else PLERROR( "StackedModulesLearner::buildOptions(): hessian_estimation\n" "value '%s' is unknown.\n", hessian_estimation.c_str() ); }
string PLearn::StackedModulesLearner::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 59 of file StackedModulesLearner.cc.
void PLearn::StackedModulesLearner::computeCostsFromOutputs | ( | const Vec & | input, |
const Vec & | output, | ||
const Vec & | target, | ||
Vec & | costs | ||
) | const [virtual] |
Computes the costs from already computed output.
Implements PLearn::PLearner.
Definition at line 442 of file StackedModulesLearner.cc.
References PLearn::TVec< T >::append(), PLearn::TVec< T >::copy(), cost_modules, i, and ncosts.
{ Vec out_tgt = output.copy(); out_tgt.append( target ); for( int i=0 ; i<ncosts ; i++ ) { Vec cost(1); cost_modules[i]->fprop( out_tgt, cost ); costs[i] = cost[0]; } }
void PLearn::StackedModulesLearner::computeOutput | ( | const Vec & | input, |
Vec & | output | ||
) | const [virtual] |
Computes the output from the input.
Reimplemented from PLearn::PLearner.
Definition at line 430 of file StackedModulesLearner.cc.
References i, modules, nmodules, outputsize(), PLearn::TVec< T >::resize(), and values.
{ values[0] << input; // fprop for( int i=0 ; i<nmodules ; i++ ) modules[i]->fprop( values[i], values[i+1] ); output.resize( outputsize() ); output << values[ nmodules ]; }
void PLearn::StackedModulesLearner::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declares the class options.
Reimplemented from PLearn::PLearner.
Definition at line 70 of file StackedModulesLearner.cc.
References PLearn::OptionBase::buildoption, cost_funcs, cost_modules, PLearn::declareOption(), PLearn::PLearner::declareOptions(), hessian_estimation, PLearn::OptionBase::learntoption, modules, ncosts, nmodules, and PLearn::PLearner::random_gen.
{ declareOption(ol, "modules", &StackedModulesLearner::modules, OptionBase::buildoption, "Layers of the learner"); declareOption(ol, "cost_funcs", &StackedModulesLearner::cost_funcs, OptionBase::buildoption, "Names of the cost functions to apply on output. First one\n" "will be used ass the cost function to optimize. For the" " moment,\n" "supported value are:\n" " - \"mse\" (default)\n" " - \"NLL\"\n"); declareOption(ol, "hessian_estimation", &StackedModulesLearner::hessian_estimation, OptionBase::buildoption, "Estimation of the second-order terms. One of:\n" " - \"none\": using only first-order derivative for" " update,\n" " - \"diag\": estimating the diagonal of the hessian,\n" " - \"simpler_diag\": positive estimation of the diagonal\n" ); declareOption(ol, "random_gen", &StackedModulesLearner::random_gen, OptionBase::buildoption, "Random numbers generator."); declareOption(ol, "nmodules", &StackedModulesLearner::nmodules, OptionBase::learntoption, "Number of module layers"); declareOption(ol, "cost_modules", &StackedModulesLearner::cost_modules, OptionBase::learntoption, "Modules that will compute the costs"); declareOption(ol, "ncosts", &StackedModulesLearner::ncosts, OptionBase::learntoption, "Number of cost modules"); // Now call the parent class' declareOptions inherited::declareOptions(ol); }
static const PPath& PLearn::StackedModulesLearner::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::PLearner.
Definition at line 155 of file StackedModulesLearner.h.
:
//##### Protected Options ###############################################
StackedModulesLearner * PLearn::StackedModulesLearner::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::PLearner.
Definition at line 59 of file StackedModulesLearner.cc.
void PLearn::StackedModulesLearner::forget | ( | ) | [virtual] |
(Re-)initializes the PLearner in its fresh state (that state may depend on the 'seed' option) and sets 'stage' back to 0 (this is the stage of a fresh learner!).
(Re-)initialize the PLearner in its fresh state (that state may depend on the 'seed' option) and sets 'stage' back to 0 (this is the stage of a fresh learner!)
A typical forget() method should do the following:
Reimplemented from PLearn::PLearner.
Definition at line 303 of file StackedModulesLearner.cc.
References PLearn::TVec< T >::clear(), diag_hessians, gradients, hessian_estimation, i, modules, nmodules, PLearn::PLearner::random_gen, PLearn::PLearner::seed_, PLearn::PLearner::stage, and values.
{ random_gen->manual_seed( seed_ ); // reset inputs values[0].clear(); gradients[0].clear(); if( hessian_estimation != "none" ) diag_hessians[0].clear(); // reset modules and outputs for( int i=0 ; i<nmodules ; i++ ) { modules[i]->forget(); values[i+1].clear(); gradients[i+1].clear(); if( hessian_estimation != "none" ) diag_hessians[i+1].clear(); } stage = 0; }
OptionList & PLearn::StackedModulesLearner::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 59 of file StackedModulesLearner.cc.
OptionMap & PLearn::StackedModulesLearner::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 59 of file StackedModulesLearner.cc.
RemoteMethodMap & PLearn::StackedModulesLearner::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 59 of file StackedModulesLearner.cc.
TVec< string > PLearn::StackedModulesLearner::getTestCostNames | ( | ) | const [virtual] |
Returns the names of the costs computed by computeCostsFromOutpus (and thus the test method).
Implements PLearn::PLearner.
Definition at line 457 of file StackedModulesLearner.cc.
References cost_funcs.
{ // Return the names of the costs computed by computeCostsFromOutputs return cost_funcs; }
TVec< string > PLearn::StackedModulesLearner::getTrainCostNames | ( | ) | const [virtual] |
Returns the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats.
Implements PLearn::PLearner.
Definition at line 463 of file StackedModulesLearner.cc.
References cost_funcs.
{ // Return the names of the objective costs that the train method computes // and for which it updates the VecStatsCollector train_stats return cost_funcs; }
void PLearn::StackedModulesLearner::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::PLearner.
Definition at line 273 of file StackedModulesLearner.cc.
References cost_funcs, cost_modules, PLearn::deepCopyField(), diag_hessians, gradients, PLearn::PLearner::makeDeepCopyFromShallowCopy(), modules, PLearn::PLearner::random_gen, and values.
{ inherited::makeDeepCopyFromShallowCopy(copies); // ### Call deepCopyField on all "pointer-like" fields // ### that you wish to be deepCopied rather than // ### shallow-copied. // ### ex: // deepCopyField(trainvec, copies); deepCopyField(modules, copies); deepCopyField(cost_funcs, copies); deepCopyField(random_gen, copies); deepCopyField(cost_modules, copies); deepCopyField(values, copies); deepCopyField(gradients, copies); deepCopyField(diag_hessians, copies); }
int PLearn::StackedModulesLearner::outputsize | ( | ) | const [virtual] |
Returns the size of this learner's output, (which typically may depend on its inputsize(), targetsize() and set options).
Implements PLearn::PLearner.
Definition at line 293 of file StackedModulesLearner.cc.
References PLearn::TVec< T >::length(), nmodules, and values.
Referenced by buildCostFunctions(), computeOutput(), and train().
{ // Compute and return the size of this learner's output (which typically // may depend on its inputsize(), targetsize() and set options). if( nmodules < 0 || values.length() <= nmodules ) return -1; else return values[ nmodules ].length(); }
void PLearn::StackedModulesLearner::train | ( | ) | [virtual] |
The role of the train method is to bring the learner up to stage==nstages, updating the train_stats collector with training costs measured on-line in the process.
Implements PLearn::PLearner.
Definition at line 336 of file StackedModulesLearner.cc.
References PLearn::TVec< T >::append(), PLearn::PLearner::computeOutputAndCosts(), cost_modules, diag_hessians, PLearn::VMat::getExample(), gradients, hessian_estimation, i, PLearn::PLearner::initTrain(), PLearn::PLearner::inputsize(), PLearn::VMat::length(), modules, ncosts, nmodules, PLearn::PLearner::nstages, outputsize(), PLearn::sample(), PLearn::PLearner::stage, PLearn::TVec< T >::subVec(), PLearn::PLearner::targetsize(), PLearn::PLearner::train_set, PLearn::PLearner::train_stats, and values.
{ // The role of the train method is to bring the learner up to // stage==nstages, updating train_stats with training costs measured // on-line in the process. /* TYPICAL CODE: static Vec input; // static so we don't reallocate memory each time... static Vec target; // (but be careful that static means shared!) input.resize(inputsize()); // the train_set's inputsize() target.resize(targetsize()); // the train_set's targetsize() real weight; // This generic PLearner method does a number of standard stuff useful for // (almost) any learner, and return 'false' if no training should take // place. See PLearner.h for more details. if (!initTrain()) return; while(stage<nstages) { // clear statistics of previous epoch train_stats->forget(); //... train for 1 stage, and update train_stats, // using train_set->getExample(input, target, weight) // and train_stats->update(train_costs) ++stage; train_stats->finalize(); // finalize statistics for this epoch } */ Vec input( inputsize() ); Vec target( targetsize() ); real weight; Vec train_costs( ncosts ); Vec output( outputsize() ); int nsamples = train_set->length(); if( !initTrain() ) return; for( ; stage < nstages ; stage++ ) { // clear stats of previous epoch train_stats->forget(); for( int sample=0 ; sample < nsamples ; sample++ ) { train_set->getExample( sample, input, target, weight ); // fprop computeOutputAndCosts(input, target, output, train_costs); output.append( target ); // bprop Vec out_gradient(1,1); // the gradient wrt the cost is '1' Vec out_dh(1); // the hessian wrt the cost is '0' if( hessian_estimation != "none" ) // bbpropUpdate { cost_modules[0]->bbpropUpdate( output, train_costs.subVec(0,1), gradients[ nmodules ], out_gradient, diag_hessians[ nmodules ], out_dh ); for( int i=nmodules-1 ; i>=0 ; i-- ) modules[i]->bbpropUpdate( values[i], values[i+1], gradients[i], gradients[i+1], diag_hessians[i], diag_hessians[i+1] ); } else // bpropUpdate { cost_modules[0]->bpropUpdate( output, train_costs.subVec(0,1), gradients[ nmodules ], out_gradient ); for( int i=nmodules-1 ; i>=0 ; i-- ) modules[i]->bpropUpdate( values[i], values[i+1], gradients[i], gradients[i+1] ); } train_stats->update( train_costs ); } train_stats->finalize(); // finalize statistics for this epoch } }
Reimplemented from PLearn::PLearner.
Definition at line 155 of file StackedModulesLearner.h.
Names of the cost functions to apply on output.
First one will be used as the cost function to optimize. For the moment, supported value are:
Definition at line 77 of file StackedModulesLearner.h.
Referenced by buildCostFunctions(), buildOptions(), declareOptions(), getTestCostNames(), getTrainCostNames(), and makeDeepCopyFromShallowCopy().
TVec< PP<OnlineLearningModule> > PLearn::StackedModulesLearner::cost_modules [protected] |
Modules that will compute the costs.
Definition at line 173 of file StackedModulesLearner.h.
Referenced by buildCostFunctions(), buildOptions(), computeCostsFromOutputs(), declareOptions(), makeDeepCopyFromShallowCopy(), and train().
TVec<Vec> PLearn::StackedModulesLearner::diag_hessians [protected] |
stores the diagonal of Hessians
Definition at line 187 of file StackedModulesLearner.h.
Referenced by buildLayers(), forget(), makeDeepCopyFromShallowCopy(), and train().
TVec<Vec> PLearn::StackedModulesLearner::gradients [protected] |
stores the gradients
Definition at line 184 of file StackedModulesLearner.h.
Referenced by buildLayers(), forget(), makeDeepCopyFromShallowCopy(), and train().
Estimation of the second-order terms.
One of:
Definition at line 83 of file StackedModulesLearner.h.
Referenced by buildCostFunctions(), buildLayers(), buildOptions(), declareOptions(), forget(), and train().
Layers of the learner.
Definition at line 71 of file StackedModulesLearner.h.
Referenced by buildLayers(), buildOptions(), computeOutput(), declareOptions(), forget(), makeDeepCopyFromShallowCopy(), and train().
int PLearn::StackedModulesLearner::ncosts [protected] |
Number of cost modules.
Definition at line 176 of file StackedModulesLearner.h.
Referenced by buildCostFunctions(), buildOptions(), computeCostsFromOutputs(), declareOptions(), and train().
int PLearn::StackedModulesLearner::nmodules [protected] |
Number of module layers.
Definition at line 170 of file StackedModulesLearner.h.
Referenced by buildLayers(), buildOptions(), computeOutput(), declareOptions(), forget(), outputsize(), and train().
TVec<Vec> PLearn::StackedModulesLearner::values [protected] |
stores the input and output values of the functions
Definition at line 181 of file StackedModulesLearner.h.
Referenced by buildLayers(), computeOutput(), forget(), makeDeepCopyFromShallowCopy(), outputsize(), and train().