PLearn 0.1
|
#include <SequentialModelSelector.h>
Public Member Functions | |
SequentialModelSelector () | |
Constructor. | |
real | sequenceCost (const Vec &sequence_errors) |
compute the cost of the given sequence of errors (based on the cost_type) | |
real | paired_t_test (const int &m1, const int &m2, int cc=0) const |
Computes a paired t-test between common_cost[cc] series of models m1 and m2. | |
virtual void | build () |
simply calls inherited::build() then build_() | |
virtual void | setExperimentDirectory (const PPath &_expdir) |
Redefines so that it ALSO calls the method on all the learners in the TVec models. | |
virtual void | train () |
virtual void | test (VMat testset, PP< VecStatsCollector > test_stats, VMat testoutputs=0, VMat testcosts=0) const |
virtual void | computeOutput (const Vec &input, Vec &output) const |
*** SUBCLASS WRITING: *** | |
virtual void | computeCostsFromOutputs (const Vec &input, const Vec &output, const Vec &target, Vec &costs) const |
*** SUBCLASS WRITING: *** | |
virtual void | computeOutputAndCosts (const Vec &input, const Vec &target, Vec &output, Vec &costs) const |
Default calls computeOutput and computeCostsFromOutputs. | |
virtual void | computeCostsOnly (const Vec &input, const Vec &target, Vec &costs) const |
Default calls computeOutputAndCosts. | |
virtual void | matlabSave (const string &matlab_subdir) |
virtual TVec< string > | getTestCostNames () const |
This should return the names of the costs computed by computeCostsFromOutputs. | |
virtual TVec< string > | getTrainCostNames () const |
This should return the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats. | |
virtual void | forget () |
*** SUBCLASS WRITING: *** | |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual SequentialModelSelector * | deepCopy (CopiesMap &copies) const |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transforms a shallow copy into a deep copy. | |
Static Public Member Functions | |
static string | _classname_ () |
Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be. | |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
bool | stepwise_save |
TVec< PP< SequentialLearner > > | models |
List of all the models. | |
TVec< string > | model_names |
If the user desires to provide a name for each model instead of model_i. | |
TVec< string > | common_costs |
int | comparison_type |
int | comparison_window |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declare this class' options. | |
Protected Attributes | |
TMat< int > | common_cost_indices |
See common_costs options for details. Row index: model. Column index: common_cost. | |
TVec< int > | best_model |
Vec | sequence_costs |
bool | report_paired_T_tests |
Private Types | |
typedef SequentialLearner | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. | |
void | checkModelNames () const |
Definition at line 49 of file SequentialModelSelector.h.
typedef SequentialLearner PLearn::SequentialModelSelector::inherited [private] |
Reimplemented from PLearn::SequentialLearner.
Definition at line 55 of file SequentialModelSelector.h.
PLearn::SequentialModelSelector::SequentialModelSelector | ( | ) |
Constructor.
Definition at line 48 of file SequentialModelSelector.cc.
: report_paired_T_tests(false), stepwise_save(true), comparison_type(1), comparison_window(-1) {}
string PLearn::SequentialModelSelector::_classname_ | ( | ) | [static] |
Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be.
Reimplemented from PLearn::SequentialLearner.
Definition at line 46 of file SequentialModelSelector.cc.
OptionList & PLearn::SequentialModelSelector::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::SequentialLearner.
Definition at line 46 of file SequentialModelSelector.cc.
RemoteMethodMap & PLearn::SequentialModelSelector::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::SequentialLearner.
Definition at line 46 of file SequentialModelSelector.cc.
Reimplemented from PLearn::SequentialLearner.
Definition at line 46 of file SequentialModelSelector.cc.
Object * PLearn::SequentialModelSelector::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 46 of file SequentialModelSelector.cc.
StaticInitializer SequentialModelSelector::_static_initializer_ & PLearn::SequentialModelSelector::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::SequentialLearner.
Definition at line 46 of file SequentialModelSelector.cc.
void PLearn::SequentialModelSelector::build | ( | ) | [virtual] |
simply calls inherited::build() then build_()
Reimplemented from PLearn::SequentialLearner.
Definition at line 106 of file SequentialModelSelector.cc.
References PLearn::SequentialLearner::build(), and build_().
{ inherited::build(); build_(); }
void PLearn::SequentialModelSelector::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::SequentialLearner.
Definition at line 71 of file SequentialModelSelector.cc.
References best_model, c, common_cost_indices, common_costs, forget(), PLearn::SequentialLearner::horizon, PLearn::SequentialLearner::init_train_size, PLearn::TVec< T >::length(), m, PLearn::SequentialLearner::max_seq_len, models, PLERROR, PLWARNING, PLearn::TVec< T >::resize(), PLearn::TMat< T >::resize(), and sequence_costs.
Referenced by build().
{ // Precondition check on init_train_size if(init_train_size < horizon) { PLWARNING("The init_train_size provided by the user was %d. However since the horizon is %d\n" "the init_train_size will be modified to be %d, the horizon value. This modification is\n" "necessary because, within a model selection context, a first internal test is needed to\n" "determine the first best model (see the SequentialModelSelector::train method\n" "implementation). If you want to avoid getting this warning, you should set init_train_size\n" "to horizon (%d) directly.", init_train_size, horizon, horizon, horizon); init_train_size = horizon; } int nb_models = models.length(); if(nb_models < 1) PLERROR("The 'models' option is mandatory in a SequentialModelSelector."); int nb_common_costs = common_costs.length(); if(nb_common_costs < 1) PLERROR("The 'common_costs' requires at least 1 cost from which models will\n" "be compared to choose the best model."); common_cost_indices.resize(nb_models, nb_common_costs); for(int m=0; m < nb_models; m++) for(int c=0; c < nb_common_costs; c++) common_cost_indices(m, c) = models[m]->getTestCostIndex( common_costs[c] ); best_model.resize(max_seq_len); sequence_costs.resize(nb_models); forget(); }
void PLearn::SequentialModelSelector::checkModelNames | ( | ) | const [private] |
Definition at line 460 of file SequentialModelSelector.cc.
References PLearn::TVec< T >::length(), m, model_names, models, PLERROR, PLearn::TVec< T >::resize(), and PLearn::tostring().
Referenced by getTestCostNames(), and setExperimentDirectory().
{ int nb_models = models.length(); int nb_names = model_names.length(); if(nb_names == 0) { model_names.resize(nb_models); for(int m=0; m < nb_models; m++) model_names[m] = "Model_"+tostring(m); } else if(nb_names != nb_models) PLERROR("Names must be provided for all models (%d = nb_names != nb_models = %d).", nb_names, nb_models); }
string PLearn::SequentialModelSelector::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 46 of file SequentialModelSelector.cc.
void PLearn::SequentialModelSelector::computeCostsFromOutputs | ( | const Vec & | input, |
const Vec & | output, | ||
const Vec & | target, | ||
Vec & | costs | ||
) | const [virtual] |
*** SUBCLASS WRITING: ***
This should be defined in subclasses to compute the weighted costs from already computed output. The costs should correspond to the cost names returned by getTestCostNames().
NOTE: In exotic cases, the cost may also depend on some info in the input, that's why the method also gets so see it.
Reimplemented from PLearn::SequentialLearner.
Definition at line 442 of file SequentialModelSelector.cc.
References best_model, PLearn::SequentialLearner::last_train_t, and models.
{ models[best_model[last_train_t]]->computeCostsFromOutputs(input, output, target, costs); }
void PLearn::SequentialModelSelector::computeCostsOnly | ( | const Vec & | input, |
const Vec & | target, | ||
Vec & | costs | ||
) | const [virtual] |
Default calls computeOutputAndCosts.
This may be overridden if there is a more efficient way to compute the costs directly, without computing the whole output vector.
Reimplemented from PLearn::SequentialLearner.
Definition at line 454 of file SequentialModelSelector.cc.
References best_model, PLearn::SequentialLearner::last_train_t, and models.
{ models[best_model[last_train_t]]->computeCostsOnly(input, target, costs); }
void PLearn::SequentialModelSelector::computeOutput | ( | const Vec & | input, |
Vec & | output | ||
) | const [virtual] |
*** SUBCLASS WRITING: ***
This should be defined in subclasses to compute the output from the input.
Reimplemented from PLearn::SequentialLearner.
Definition at line 437 of file SequentialModelSelector.cc.
References best_model, PLearn::SequentialLearner::last_train_t, and models.
{ models[best_model[last_train_t]]->computeOutput(input, output); }
void PLearn::SequentialModelSelector::computeOutputAndCosts | ( | const Vec & | input, |
const Vec & | target, | ||
Vec & | output, | ||
Vec & | costs | ||
) | const [virtual] |
Default calls computeOutput and computeCostsFromOutputs.
You may override this if you have a more efficient way to compute both output and weighted costs at the same time.
Reimplemented from PLearn::SequentialLearner.
Definition at line 448 of file SequentialModelSelector.cc.
References best_model, PLearn::SequentialLearner::last_train_t, and models.
{ models[best_model[last_train_t]]->computeOutputAndCosts(input, target, output, costs); }
void PLearn::SequentialModelSelector::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declare this class' options.
Reimplemented from PLearn::SequentialLearner.
Definition at line 112 of file SequentialModelSelector.cc.
References PLearn::OptionBase::buildoption, common_costs, comparison_type, comparison_window, PLearn::declareOption(), PLearn::SequentialLearner::declareOptions(), model_names, models, report_paired_T_tests, and stepwise_save.
{ declareOption(ol, "stepwise_save", &SequentialModelSelector::stepwise_save, OptionBase::buildoption, "Does the model selector hass to save errors at each step.\n" "Default: true."); declareOption(ol, "models", &SequentialModelSelector::models, OptionBase::buildoption, "List of all the models.\n"); declareOption(ol, "model_names", &SequentialModelSelector::model_names, OptionBase::buildoption, "If the user desires to provide a name for each model instead of model_i."); declareOption(ol, "common_costs", &SequentialModelSelector::common_costs, OptionBase::buildoption, "The names of costs that are common to all models and that the user wishes the model\n" "selector to keep track of. The first one is considered to be the main cost, the one\n" "from which models will be compared to choose the best model."); declareOption(ol, "report_paired_T_tests", &SequentialModelSelector::report_paired_T_tests, OptionBase::buildoption, "If true, the model selector will report as costs the paired T tests on common_cost_indices[0] for\n" "models[0] against each other model. The model selector will only report T tests once,\n" "at t=(max_seq_len-1)\n" "\n" "Default: false."); declareOption(ol, "comparison_type", &SequentialModelSelector::comparison_type, OptionBase::buildoption, "From the common_costs list, the first cost is the one from which models will be compared\n" "to choose the best model. But should the best model be chosen according to the\n" "\n" "max/min\n" " +/- 1: Mean\n" " +/- 2: Mean / Variance\n" " +/- 3: more to come.\n" "\n" "of the cost realizations. \n" "Default: 1.\n"); declareOption(ol, "comparison_window", &SequentialModelSelector::comparison_window, OptionBase::buildoption, "If positive, the comparison performed on the basis of common_cost[0] will be applyed only\n" "the comparison_window last elements of the cost sequence.\n" "Default: -1. (No window)"); inherited::declareOptions(ol); }
static const PPath& PLearn::SequentialModelSelector::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::SequentialLearner.
Definition at line 182 of file SequentialModelSelector.h.
SequentialModelSelector * PLearn::SequentialModelSelector::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::SequentialLearner.
Definition at line 46 of file SequentialModelSelector.cc.
void PLearn::SequentialModelSelector::forget | ( | ) | [virtual] |
*** SUBCLASS WRITING: ***
(Re-)initializes the PLearner in its fresh state (that state may depend on the 'seed' option) and sets 'stage' back to 0 (this is the stage of a fresh learner!)
A typical forget() method should do the following:
This method is typically called by the build_() method, after it has finished setting up the parameters, and if it deemed useful to set or reset the learner in its fresh state. (remember build may be called after modifying options that do not necessarily require the learner to restart from a fresh state...) forget is also called by the setTrainingSet method, after calling build(), so it will generally be called TWICE during setTrainingSet!
Reimplemented from PLearn::SequentialLearner.
Definition at line 200 of file SequentialModelSelector.cc.
References best_model, PLearn::TVec< T >::fill(), PLearn::SequentialLearner::forget(), i, PLearn::SequentialLearner::max_seq_len, models, PLearn::TVec< T >::resize(), and PLearn::TVec< T >::size().
Referenced by build_().
{ best_model.resize(max_seq_len); best_model.fill(-1); // by default for (int i=0; i<models.size(); i++) models[i]->forget(); inherited::forget(); }
OptionList & PLearn::SequentialModelSelector::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 46 of file SequentialModelSelector.cc.
OptionMap & PLearn::SequentialModelSelector::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 46 of file SequentialModelSelector.cc.
RemoteMethodMap & PLearn::SequentialModelSelector::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 46 of file SequentialModelSelector.cc.
TVec< string > PLearn::SequentialModelSelector::getTestCostNames | ( | ) | const [virtual] |
This should return the names of the costs computed by computeCostsFromOutputs.
Implements PLearn::PLearner.
Definition at line 474 of file SequentialModelSelector.cc.
References PLearn::TVec< T >::append(), c, checkModelNames(), common_costs, PLearn::TVec< T >::length(), m, model_names, models, and report_paired_T_tests.
{ TVec<string> tcnames = common_costs; int nb_models = models.length(); checkModelNames(); for(int m=0; m < nb_models; m++) { TVec<string> tcm = models[m]->getTestCostNames(); for(int c=0; c < tcm.length(); c++) tcnames.append(model_names[m] + "::" + tcm[c]); } if( report_paired_T_tests ) for(int m=1; m < nb_models; m++) tcnames.append("T-test_" + model_names[0] + "_vs_" + model_names[m]); return tcnames; }
TVec< string > PLearn::SequentialModelSelector::getTrainCostNames | ( | ) | const [virtual] |
This should return the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats.
Implements PLearn::PLearner.
Definition at line 495 of file SequentialModelSelector.cc.
References common_costs.
{ return common_costs; }
void PLearn::SequentialModelSelector::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::SequentialLearner.
Definition at line 498 of file SequentialModelSelector.cc.
References PLearn::deepCopyField(), PLearn::SequentialLearner::makeDeepCopyFromShallowCopy(), and models.
{ inherited::makeDeepCopyFromShallowCopy(copies); deepCopyField(models, copies); //deepCopyField(mean_costs, copies); }
void PLearn::SequentialModelSelector::matlabSave | ( | const string & | matlab_subdir | ) | [virtual] |
Any SequentialLearner will have the possiblity to save some data in a matlab 'readable' format. The data will be saved in expdir/matlab_subdir through the global matlabSave function (MatIO.h).
Reimplemented from PLearn::SequentialLearner.
Definition at line 505 of file SequentialModelSelector.cc.
References PLearn::add(), PLearn::TVec< T >::append(), PLearn::append_slash(), common_cost_indices, common_costs, PLearn::concat(), g, PLearn::SequentialLearner::getCostSequence(), PLearn::PLearner::getExperimentDirectory(), PLearn::hconcat(), PLearn::TVec< T >::length(), m, model_names, models, and PLearn::TVec< T >::subVec().
{ string save_dir = append_slash(getExperimentDirectory()) + matlab_subdir; Vec dummy, add(1); add[0] = 0; TVec<string> legend(1, "ModelSelector"); legend.append(model_names); int nb_models = models.length(); int nb_common_costs = common_costs.length(); for(int g=0; g < nb_common_costs; g++) { Vec cs; Array<Mat> columns(nb_models+1); cs = getCostSequence(g); // Since the model selector first runs tests in its train method, the models' sequence // length will probably be to long... int msel_cs_len = cs.length(); columns[0] = Mat(cs.length(), 1, cs); for(int m=0; m < nb_models; m++) { cs = models[m]->getCostSequence( common_cost_indices(m, g) ); cs = cs.subVec(cs.length() - msel_cs_len, msel_cs_len); //... length correction (see the comment above) columns[m+1] = Mat(cs.length(), 1, cs); } Mat concat = hconcat(columns); PLearn::matlabSave( save_dir, common_costs[g], concat, add, dummy, legend); } for(int m=0; m < nb_models; m++) models[m]->matlabSave(matlab_subdir); }
real PLearn::SequentialModelSelector::paired_t_test | ( | const int & | m1, |
const int & | m2, | ||
int | cc = 0 |
||
) | const |
Computes a paired t-test between common_cost[cc] series of models m1 and m2.
Definition at line 182 of file SequentialModelSelector.cc.
References PLearn::TMat< T >::column(), common_cost_indices, PLearn::SequentialLearner::errors, PLearn::TVec< T >::length(), PLearn::mean(), models, n, PLERROR, PLearn::remove_missing(), PLearn::sqrt(), PLearn::sumsquare(), PLearn::TMat< T >::toVecCopy(), and u.
Referenced by test().
{ Vec u = remove_missing(models[m1]->errors.column( common_cost_indices(m1, cc) ).toVecCopy()); Vec v = remove_missing(models[m2]->errors.column( common_cost_indices(m2, cc) ).toVecCopy()); int n = u.length(); if( v.length() != n ) PLERROR("SequentialModelSelector::paired_t_test -- Can't make a paired t-test on to unequally lengthed vectors (%d != %d).", n, v.length()); real ubar = mean(u); real vbar = mean(v); Vec u2 = u - ubar; Vec v2 = v - vbar; return (ubar - vbar) * sqrt( n*(n-1) / sumsquare(u2-v2)); }
compute the cost of the given sequence of errors (based on the cost_type)
Definition at line 157 of file SequentialModelSelector.cc.
References PLearn::abs(), comparison_type, comparison_window, PLearn::TVec< T >::length(), PLearn::mean(), MISSING_VALUE, PLERROR, PLearn::TVec< T >::subVec(), and PLearn::variance().
Referenced by train().
{ int seq_len = sequence_errors.length(); if(seq_len == 0) return MISSING_VALUE; int window_start = seq_len - comparison_window; Vec windowed; if(comparison_window < 1 || window_start < 0) windowed = sequence_errors; // The option is inactive or the sequence is not yet comparison_window long else windowed = sequence_errors.subVec(window_start, comparison_window); int type = abs(comparison_type); real mean_ = mean(windowed); if (type == 1) return mean_; if (type == 2) return mean_ / variance(windowed, mean_); PLERROR("Invalid comparison type %d!", comparison_type); return MISSING_VALUE; }
void PLearn::SequentialModelSelector::setExperimentDirectory | ( | const PPath & | _expdir | ) | [virtual] |
Redefines so that it ALSO calls the method on all the learners in the TVec models.
Reimplemented from PLearn::PLearner.
Definition at line 55 of file SequentialModelSelector.cc.
References PLearn::append_slash(), checkModelNames(), PLearn::TVec< T >::length(), m, model_names, and models.
{ int nb_models = models.length(); checkModelNames(); PLearner::setExperimentDirectory(_expdir); string model_m; for(int m=0; m < nb_models; m++) { model_m = models[m]->getExperimentDirectory(); if(model_m == "") model_m = model_names[m]; models[m]->setExperimentDirectory(append_slash(_expdir)+ model_m); } }
void PLearn::SequentialModelSelector::test | ( | VMat | testset, |
PP< VecStatsCollector > | test_stats, | ||
VMat | testoutputs = 0 , |
||
VMat | testcosts = 0 |
||
) | const [virtual] |
*** SUBCLASS WRITING: *** The method should:
Implements PLearn::SequentialLearner.
Definition at line 346 of file SequentialModelSelector.cc.
References PLearn::TVec< T >::append(), PLearn::append_slash(), best_model, common_cost_indices, PLearn::endl(), PLearn::SequentialLearner::errors, PLearn::PLearner::expdir, i, PLearn::SequentialLearner::last_test_t, PLearn::SequentialLearner::last_train_t, PLearn::TVec< T >::length(), PLearn::VMat::length(), m, MAX, PLearn::SequentialLearner::max_seq_len, MISSING_VALUE, models, paired_t_test(), PLWARNING, PLearn::SequentialLearner::predictions, report_paired_T_tests, PLearn::PLearner::report_progress, PLearn::saveAsciiWithoutSize(), PLearn::TVec< T >::size(), stepwise_save, PLearn::tostring(), and PLearn::update().
{ // Since model selection train method may not train before (init_train_size+horizon) // we have to verify if a first train was actually done prior to entering the // test core if(last_train_t == -1) { PLWARNING("SequentialModelSelector::test -- Skipped because there were no prior train."); return; } int start_t = MAX(last_train_t+1,last_test_t+1); if( test_set.length()-1 < start_t ) return; PP<ProgressBar> pb; if (report_progress) pb = new ProgressBar("Testing SequentialModelSelector learner",test_set.length()); TVec< PP<VecStatsCollector> > dummy_stats(models.length()); for (int t=start_t; t < test_set.length(); t++) { Vec best_model_costs; Vec models_update; for (int i=0; i<models.size(); i++) { if(t == start_t) dummy_stats[i] = new VecStatsCollector(); models[i]->test(test_set, dummy_stats[i]); if(i == best_model[last_train_t]) { best_model_costs = models[i]->errors(t); models_update.append( best_model_costs ); } else models_update.append( models[i]->errors(t) ); } // The update is composed of the common_costs and the modelwise costs Vec update = best_model_costs( common_cost_indices(best_model[last_train_t]) ); update.append(models_update); // Report T test processing: only reports T tests once, at t=(max_seq_len-1) if( report_paired_T_tests ) for(int m=1; m < models.length(); m++) { if( test_set.length() == max_seq_len ) update.append( paired_t_test(0, m) ); else update.append( MISSING_VALUE ); } test_stats->update( update ); predictions(t) << models[best_model[last_train_t]]->predictions(t); errors(t) << update; if (testoutputs) testoutputs->appendRow( predictions(t) ); if (testcosts) testcosts->appendRow(update); #ifdef DEBUG cout << "predictions(" << t << "): " << predictions(t) << endl << "errors(" << t << "): " << errors(t) << endl; #endif if(pb) pb->update(t); } if(pb) pb->close(); last_test_t = test_set.length()-1; #ifdef DEBUG cout << "SequentialModelSelector.last_test_t = " << last_test_t << endl; #endif if(stepwise_save) { string s1 = append_slash(expdir) + "predictions_test_t=" + tostring(last_test_t); saveAsciiWithoutSize(s1, predictions); string s2 = append_slash(expdir) + "errors_test_t=" + tostring(last_test_t); saveAsciiWithoutSize(s2, errors); } }
void PLearn::SequentialModelSelector::train | ( | ) | [virtual] |
*** SUBCLASS WRITING: *** Does the actual training. Subclasses must implement this method. The method should upon entry, call setTrainingSet(training_set); The method should:
First call to train: if the models[i]->init_train_size = init_train_size-horizon; statement was done during the build and if init_train_size was modified afterwards (e.g. by SequentialValidation) the models would not have been updated properly.
Note also that the sublearners are provided init_train_size-horizon since the model selector needs to go horizon time steps back in time.
Implements PLearn::SequentialLearner.
Definition at line 210 of file SequentialModelSelector.cc.
References PLearn::append_slash(), PLearn::argmax(), PLearn::argmin(), best_model, PLearn::TMat< T >::column(), common_cost_indices, comparison_type, PLearn::endl(), PLearn::SequentialLearner::errors, PLearn::PLearner::expdir, PLearn::SequentialLearner::horizon, i, PLearn::SequentialLearner::init_train_size, PLearn::SequentialLearner::last_call_train_t, PLearn::SequentialLearner::last_test_t, PLearn::SequentialLearner::last_train_t, PLearn::TVec< T >::length(), PLearn::VMat::length(), MAX, models, PLERROR, PLWARNING, PLearn::SequentialLearner::predictions, PLearn::remove_missing(), PLearn::PLearner::report_progress, PLearn::saveAsciiWithoutSize(), sequence_costs, sequenceCost(), PLearn::TVec< T >::size(), stepwise_save, PLearn::PLearner::sub_test(), PLearn::TMat< T >::subMat(), PLearn::VMat::subMatRows(), PLearn::tostring(), PLearn::TMat< T >::toVecCopy(), PLearn::PLearner::train_set, PLearn::PLearner::train_stats, PLearn::SequentialLearner::train_step, and PLearn::update().
{ if(last_call_train_t == -1) for (int i=0; i<models.size(); i++) { models[i]->horizon = horizon; models[i]->init_train_size = init_train_size-horizon; } last_call_train_t = train_set.length()-1; // The init_train_size option is the train_set size at which we should start training. // The corresponding time index is (init_train_size-1). See also the build_ precondition // check on init_train_size. int init_train_t = init_train_size-1; int start_t = MAX(init_train_t, last_train_t+train_step); if( start_t < last_test_t ) PLERROR("SequentialModelSelector::train -- start_t = %d < %d = last_test_t", start_t, last_test_t); if( last_call_train_t < start_t ) return; PP<ProgressBar> pb; if (report_progress) pb = new ProgressBar("Training SequentialModelSelector learner",train_set.length()); TVec< PP<VecStatsCollector> > dummy_stats(models.length()); for (int t=start_t; t < train_set.length(); t++) { // The time t index is equal to a set last row index. Its length will therefore be: int set_length = t+1; #ifdef DEBUG cout << "SequentialModelSelector::train() -- sub_train.length = " << set_length-horizon << " et sub_test.length = " << set_length << endl; #endif VMat sub_train = train_set.subMatRows(0,set_length-horizon); // last training pair is (t-2*horizon,t-horizon) sub_train->defineSizes(train_set->inputsize(), train_set->targetsize(), train_set->weightsize()); VMat sub_test = train_set.subMatRows(0,set_length); // last test pair is (t-horizon,t) sub_test->defineSizes(train_set->inputsize(), train_set->targetsize(), train_set->weightsize()); for (int i=0; i < models.size(); i++) { if(t == start_t) dummy_stats[i] = new VecStatsCollector(); models[i]->setTrainingSet(sub_train, false); models[i]->train(); models[i]->test(sub_test, dummy_stats[i]); // last cost computed goes at t-1, last prediction at t-1-horizon Vec sequence_errors = remove_missing(models[i]->errors.column( common_cost_indices(i, 0) ).toVecCopy()); #ifdef DEBUG cout << "models["<<i<<"]->getTestCostNames()[common_cost_indices(i, 0)]: " << models[i]->getTestCostNames()[common_cost_indices(i, 0)] << endl; PLWARNING("models[%d]->errors.subMat(0,%d,%d,1)", i, common_cost_indices(i, 0), t); cout << remove_missing( models[i]->errors.subMat(0,common_cost_indices(i,0),set_length,1).toVecCopy() ) << endl; cout << "---\nOR\n---\n" << sequence_errors << endl; #endif sequence_costs[i] = sequenceCost(sequence_errors); } // we set the best model for this time step if(comparison_type > 0) best_model[t] = argmax(sequence_costs, true); else best_model[t] = argmin(sequence_costs, true); if(best_model[t] == -1) best_model[t] = 0; // All models provided only MISSING_VALUE as common_costs[0] !!! #ifdef DEBUG cout << "sequence_costs: " << sequence_costs << endl; cout << "SequentialModelSelector::train() -- t = " << t << " et best_model = " << best_model[t] << endl; #endif if(predictions(t-horizon).hasMissing()) predictions(t-horizon) << models[best_model[t]]->predictions(t-horizon); #ifdef DEBUG cout << "SequentialModelSelector::train() -- train_set.length = " << set_length << endl; #endif if (pb) pb->update(t); } // Now train with everything that is available -- last training pair is (t-horizon,t) Vec best_model_costs; for (int i=0; i<models.size(); i++) { models[i]->setTrainingSet(train_set, false); models[i]->train(); if(i == best_model[last_call_train_t]) best_model_costs = models[i]->errors(last_call_train_t); } if(train_stats) { Vec update = best_model_costs( common_cost_indices(best_model[last_call_train_t]) ); train_stats->update( update ); } predictions(last_call_train_t) << models[ best_model[last_call_train_t] ]->predictions(last_call_train_t); if(pb) pb->close(); last_train_t = last_call_train_t; #ifdef DEBUG cout << "SequentialModelSelector.last_train_t = " << last_train_t << endl; #endif if(stepwise_save) { string s1 = append_slash(expdir) + "predictions_train_t=" + tostring(last_train_t); //"seq_model/predictions_train_t=" + tostring(last_train_t); saveAsciiWithoutSize(s1, predictions); string s2 = append_slash(expdir) + "errors_train_t=" + tostring(last_train_t); saveAsciiWithoutSize(s2, errors); } }
Reimplemented from PLearn::SequentialLearner.
Definition at line 182 of file SequentialModelSelector.h.
TVec<int> PLearn::SequentialModelSelector::best_model [protected] |
Definition at line 70 of file SequentialModelSelector.h.
Referenced by build_(), computeCostsFromOutputs(), computeCostsOnly(), computeOutput(), computeOutputAndCosts(), forget(), test(), and train().
TMat<int> PLearn::SequentialModelSelector::common_cost_indices [protected] |
See common_costs options for details. Row index: model. Column index: common_cost.
Definition at line 69 of file SequentialModelSelector.h.
Referenced by build_(), matlabSave(), paired_t_test(), test(), and train().
The names of costs that are common to all models and that the user wishes the model selector to keep track of. The first one is considered to be the main cost, the one from which models will be compared to choose the best model.
Definition at line 112 of file SequentialModelSelector.h.
Referenced by build_(), declareOptions(), getTestCostNames(), getTrainCostNames(), and matlabSave().
From the common_costs list, the first cost is the one from which models will be compared to choose the best model. But should the best model be chosen according to the
max/min +/- 1: Mean +/- 2: Mean / Variance +/- 3: more to come.
of the cost realizations. Default: 1.
Definition at line 126 of file SequentialModelSelector.h.
Referenced by declareOptions(), sequenceCost(), and train().
If positive, the comparison performed on the basis of common_cost[0] will be applyed only the comparison_window last elements of the cost sequence. Default: -1. (No window)
Definition at line 133 of file SequentialModelSelector.h.
Referenced by declareOptions(), and sequenceCost().
TVec< string > PLearn::SequentialModelSelector::model_names [mutable] |
If the user desires to provide a name for each model instead of model_i.
Definition at line 105 of file SequentialModelSelector.h.
Referenced by checkModelNames(), declareOptions(), getTestCostNames(), matlabSave(), and setExperimentDirectory().
List of all the models.
Definition at line 102 of file SequentialModelSelector.h.
Referenced by build_(), checkModelNames(), computeCostsFromOutputs(), computeCostsOnly(), computeOutput(), computeOutputAndCosts(), declareOptions(), forget(), getTestCostNames(), makeDeepCopyFromShallowCopy(), matlabSave(), paired_t_test(), setExperimentDirectory(), test(), and train().
If true, the model selector will report as costs the paired T tests on common_cost_indices[0] for models[0] against each other model. The model selector will only report T tests once, at t=(max_seq_len-1)
Default: false.
Definition at line 80 of file SequentialModelSelector.h.
Referenced by declareOptions(), getTestCostNames(), and test().
Vec PLearn::SequentialModelSelector::sequence_costs [protected] |
Definition at line 71 of file SequentialModelSelector.h.
Does the model selector hass to save errors at each step. Default: true.
Definition at line 99 of file SequentialModelSelector.h.
Referenced by declareOptions(), test(), and train().