PLearn 0.1
|
#include <SequentialValidation.h>
Public Member Functions | |
SequentialValidation () | |
Default constructor. | |
virtual void | build () |
Simply calls inherited::build() then build_() | |
virtual void | setExperimentDirectory (const PPath &_expdir) |
virtual void | run () |
The main method; runs the experiment. | |
virtual void | warmupModel (int warmup_size) |
If warmup_size > 0, warmup the learner before running the experiment. | |
virtual void | setTestStartTime (int test_start_time, bool call_build=true) |
Set the test-start time of learner and accessory learners; call resetInternalState() and optionally build() | |
virtual void | createStatCollectors () |
Create the stat collectors. | |
virtual void | createStatSpecs () |
Create the stat specs. | |
virtual void | createStatVMats () |
Create the vmatrix required for saving the statistics. | |
virtual void | trainLearners (VMat training_set) |
Train the main learner (and accessory learners) | |
virtual void | testLearners (VMat test_set) |
Test learner on LAST OBSERVATION of test_set; also call computeOutputAndCosts on accessory learners. | |
void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be. | |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual SequentialValidation * | deepCopy (CopiesMap &copies) const |
Static Public Member Functions | |
static string | _classname_ () |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
int | init_train_size |
Size of first training set (default: 1) | |
int | warmup_size |
If specified, this is a number of time-steps that are taken FROM THE END of init_train_size to start "testing" (i.e. | |
int | train_step |
At how many timesteps must we retrain? (default: 1) If this is zero, train() is never called. | |
int | last_test_time |
The last time-step to use for testing (Default = -1, i.e. use all data) | |
VMat | dataset |
The training/test set. | |
PP< StatefulLearner > | learner |
The StatefulLearner that will be tested. | |
TVec< PP< StatefulLearner > > | accessory_learners |
TVec< string > | statnames |
Global statistics or split statistics to be computed. | |
TVec< string > | timewise_statnames |
Timewise statistics to be computed. | |
PPath | expdir |
the directory where everything will be saved | |
bool | report_stats |
bool | save_final_model |
bool | save_initial_model |
bool | save_initial_seqval |
bool | save_data_sets |
bool | save_test_outputs |
bool | save_test_costs |
bool | save_stat_collectors |
bool | provide_learner_expdir |
bool | save_sequence_stats |
Whether the statistics accumulated at each time step should be saved in the file "sequence_stats.pmat". | |
bool | report_memory_usage |
Whether to report memory usage in a directory expdir/MemoryUsage. | |
TVec< pair< string, string > > | measure_after_train |
List of options to "measure" AFTER training at each timestep, but BEFORE testing. | |
TVec< pair< string, string > > | measure_after_test |
List of options to "measure" AFTER test, in the same format as 'measure_after_test'. | |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Protected Member Functions | |
virtual void | reportStats (const Vec &global_result) |
Utility method to report statistics. | |
virtual void | reportMemoryUsage (int t) |
Utility method to report the amount of memory used at timestep t. | |
virtual bool | shouldTrain (int t) |
Utility method that returns true if train() should be called at timestep t. | |
virtual VMat | trainVMat (int t) |
Utility method to return the training VMatrix at timestep t (i.e. | |
virtual VMat | testVMat (int t) |
Utility method to return the test VMatrix at timestep t (i.e. | |
virtual int | maxTimeStep () const |
Utility method to return the highest possible timestep plus 1. | |
virtual void | measureOptions (const TVec< pair< string, string > > &options, PPath where_to_save) |
Utility function to measure a list of options and save them in the specified directory. | |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declare this class' options. | |
Protected Attributes | |
PP< VecStatsCollector > | train_stats |
Training stat collector for main learner. | |
PP< VecStatsCollector > | accessory_train_stats |
Training stat collector for accessory learners. | |
PP< VecStatsCollector > | test_stats |
Test stat collector. | |
PP< VecStatsCollector > | sequence_stats |
Sequence stat collector. | |
PP< VecStatsCollector > | timewise_stats |
Timewise stat collector. | |
TVec< PP< VecStatsCollector > > | stcol |
TVec< StatSpec > | statspecs |
Statspec corresponding to statnames. | |
TVec< StatSpec > | timewise_statspecs |
Statspec corresponding to timewise_statnames. | |
VMat | global_stats_vm |
vmat where to save global result stats specified in statnames | |
VMat | split_stats_vm |
vmat where to save per split result stats | |
VMat | timewise_stats_vm |
vmat where to save timewise statistics | |
Vec | input |
Buffers. | |
Vec | target |
Vec | dummy_output |
for accessory_learners | |
Vec | dummy_costs |
for accessory_learners | |
Vec | output |
Vec | costs |
Private Types | |
typedef Object | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. |
Definition at line 53 of file SequentialValidation.h.
typedef Object PLearn::SequentialValidation::inherited [private] |
Reimplemented from PLearn::Object.
Definition at line 55 of file SequentialValidation.h.
PLearn::SequentialValidation::SequentialValidation | ( | ) |
Default constructor.
Definition at line 58 of file SequentialValidation.cc.
: init_train_size(1), warmup_size(0), train_step(1), last_test_time(-1), expdir(""), report_stats(true), save_final_model(true), save_initial_model(false), save_initial_seqval(true), save_data_sets(false), save_test_outputs(false), save_test_costs(false), save_stat_collectors(false), provide_learner_expdir(true), save_sequence_stats(true), report_memory_usage(false) {}
string PLearn::SequentialValidation::_classname_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 56 of file SequentialValidation.cc.
OptionList & PLearn::SequentialValidation::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 56 of file SequentialValidation.cc.
RemoteMethodMap & PLearn::SequentialValidation::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 56 of file SequentialValidation.cc.
Reimplemented from PLearn::Object.
Definition at line 56 of file SequentialValidation.cc.
Object * PLearn::SequentialValidation::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 56 of file SequentialValidation.cc.
StaticInitializer SequentialValidation::_static_initializer_ & PLearn::SequentialValidation::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 56 of file SequentialValidation.cc.
void PLearn::SequentialValidation::build | ( | ) | [virtual] |
Simply calls inherited::build() then build_()
Reimplemented from PLearn::Object.
Definition at line 83 of file SequentialValidation.cc.
References PLearn::Object::build(), and build_().
{ inherited::build(); build_(); }
void PLearn::SequentialValidation::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::Object.
Definition at line 77 of file SequentialValidation.cc.
References dataset, and PLearn::VMat::width().
Referenced by build().
string PLearn::SequentialValidation::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 56 of file SequentialValidation.cc.
void PLearn::SequentialValidation::createStatCollectors | ( | ) | [virtual] |
Create the stat collectors.
Definition at line 533 of file SequentialValidation.cc.
References a, accessory_learners, accessory_train_stats, learner, PLearn::TVec< T >::length(), n, PLearn::TVec< T >::resize(), sequence_stats, stcol, test_stats, timewise_stats, and train_stats.
Referenced by run().
{ // Always manage the accessory_learners first since they may be used // within the main trader. accessory_train_stats = new VecStatsCollector(); for (int a=0, n=accessory_learners.length() ; a<n ; ++a) accessory_learners[a]->setTrainStatsCollector( accessory_train_stats ); // stats for a train on one split stcol.resize(2); train_stats = new VecStatsCollector(); train_stats->setFieldNames(learner->getTrainCostNames()); learner->setTrainStatsCollector(train_stats); stcol[0] = train_stats; // stats for a test on one split test_stats = new VecStatsCollector(); test_stats->setFieldNames(learner->getTestCostNames()); stcol[1] = test_stats; // stats over all sequence sequence_stats = new VecStatsCollector(); // timewise stats (may not be used) timewise_stats = new VecStatsCollector(); }
void PLearn::SequentialValidation::createStatSpecs | ( | ) | [virtual] |
Create the stat specs.
Definition at line 560 of file SequentialValidation.cc.
References PLearn::TVec< T >::length(), PLearn::TVec< T >::resize(), statnames, statspecs, timewise_statnames, and timewise_statspecs.
Referenced by run().
{ // Stat specs (overall) const int nstats = statnames.length(); statspecs.resize(nstats); for (int k=0; k<nstats; k++) statspecs[k].init(statnames[k]); // Stat specs (timewise) const int timewise_nstats = timewise_statnames.length(); timewise_statspecs.resize(timewise_nstats); for (int k=0; k<timewise_nstats; ++k) timewise_statspecs[k].init(timewise_statnames[k]); }
void PLearn::SequentialValidation::createStatVMats | ( | ) | [virtual] |
Create the vmatrix required for saving the statistics.
Definition at line 575 of file SequentialValidation.cc.
References expdir, global_stats_vm, PLearn::join(), learner, save_sequence_stats, PLearn::saveStringInFile(), PLearn::TVec< T >::size(), split_stats_vm, statnames, statspecs, timewise_statnames, timewise_stats_vm, and timewise_statspecs.
Referenced by run().
{ TVec<string> traincostnames = learner->getTrainCostNames(); TVec<string> testcostnames = learner->getTestCostNames(); const int nstats = statnames.size(); const int timewise_nstats = timewise_statnames.size(); saveStringInFile(expdir / "train_cost_names.txt", join(traincostnames,"\n")+"\n"); saveStringInFile(expdir / "test_cost_names.txt", join(testcostnames,"\n")+"\n"); global_stats_vm = new FileVMatrix(expdir / "global_stats.pmat", 0, nstats); for(int k=0; k<nstats; k++) global_stats_vm->declareField(k,statspecs[k].statName()); global_stats_vm->saveFieldInfos(); if (save_sequence_stats) { split_stats_vm = new FileVMatrix(expdir+"sequence_stats.pmat", 0, 1+nstats); split_stats_vm->declareField(0,"splitnum"); for(int k=0; k<nstats; k++) split_stats_vm->declareField(k+1,statspecs[k].setname + "." + statspecs[k].intstatname); split_stats_vm->saveFieldInfos(); } if (timewise_nstats > 0) { timewise_stats_vm = new FileVMatrix(expdir+"timewise_stats.pmat", 0, timewise_nstats); for (int k=0; k<timewise_nstats; ++k) timewise_stats_vm->declareField(k, timewise_statspecs[k].statName()); timewise_stats_vm->saveFieldInfos(); } }
void PLearn::SequentialValidation::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declare this class' options.
Reimplemented from PLearn::Object.
Definition at line 89 of file SequentialValidation.cc.
References accessory_learners, PLearn::OptionBase::buildoption, dataset, PLearn::declareOption(), PLearn::Object::declareOptions(), expdir, init_train_size, last_test_time, learner, measure_after_train, provide_learner_expdir, report_memory_usage, report_stats, save_data_sets, save_final_model, save_initial_model, save_initial_seqval, save_sequence_stats, save_stat_collectors, save_test_costs, save_test_outputs, statnames, timewise_statnames, train_step, and warmup_size.
{ declareOption( ol, "report_stats", &SequentialValidation::report_stats, OptionBase::buildoption, "If true, the computed global statistics specified in statnames will be saved in global_stats.pmat \n" "and the corresponding per-split statistics will be saved in split_stats.pmat \n" "For reference, all cost names (as given by the learner's getTrainCostNames() and getTestCostNames() ) \n" "will be reported in files train_cost_names.txt and test_cost_names.txt"); declareOption( ol, "statnames", &SequentialValidation::statnames, OptionBase::buildoption, "A list of global statistics we are interested in.\n" "These are strings of the form S1[S2[dataset.cost_name]] where:\n" " - dataset is train or test1 or test2 ... (train being \n" " the first dataset in a split, test1 the second, ...) \n" " - cost_name is one of the training or test cost names (depending on dataset) understood \n" " by the underlying learner (see its getTrainCostNames and getTestCostNames methods) \n" " - S1 and S2 are a statistic, i.e. one of: E (expectation), V(variance), MIN, MAX, STDDEV, ... \n" " S2 is computed over the samples of a given dataset split. S1 is over the splits. \n"); declareOption( ol, "timewise_statnames", &SequentialValidation::timewise_statnames, OptionBase::buildoption, "Statistics to be collected into a VecStatsCollector at each timestep."); declareOption( ol, "expdir", &SequentialValidation::expdir, OptionBase::buildoption, "Path of this experiment's directory in which to save all experiment results (will be created if it does not already exist). \n"); declareOption( ol, "learner", &SequentialValidation::learner, OptionBase::buildoption, "The SequentialLearner to train/test. \n"); declareOption( ol, "accessory_learners", &SequentialValidation::accessory_learners, OptionBase::buildoption, "Accessory learners that must be managed in parallel with the main one." ); declareOption( ol, "dataset", &SequentialValidation::dataset, OptionBase::buildoption, "The dataset to use for training/testing. \n"); declareOption( ol, "init_train_size", &SequentialValidation::init_train_size, OptionBase::buildoption, "Size of the first training set. Before starting the train/test cycle,\n" "the method setTestStartTime() is called on the learner with init_train_size\n" "as argument."); declareOption( ol, "warmup_size", &SequentialValidation::warmup_size, OptionBase::buildoption, "If specified, this is a number of time-steps that are taken FROM THE\n" "END of init_train_size to start \"testing\" (i.e. alternating between\n" "train and test), but WITHOUT ACCUMULATING ANY TEST STATISTICS. In\n" "other words, this is a \"warmup\" period just before the true test.\n" "Before starting the real test period, the setTestStartTime() method is\n" "called on the learner, followed by resetInternalState(). Note that\n" "the very first \"init_train_size\" is REDUCED by the warmup_size.\n"); declareOption( ol, "train_step", &SequentialValidation::train_step, OptionBase::buildoption, "At how many timesteps must we retrain? (default: 1)"); declareOption( ol, "last_test_time", &SequentialValidation::last_test_time, OptionBase::buildoption, "The last time-step to use for testing (Default = -1, i.e. use all data)"); declareOption( ol, "save_final_model", &SequentialValidation::save_final_model, OptionBase::buildoption, "If true, the final model will be saved in model.psave \n"); declareOption( ol, "save_initial_model", &SequentialValidation::save_initial_model, OptionBase::buildoption, "If true, the initial model will be saved in initial_model.psave. \n"); declareOption( ol, "save_initial_seqval", &SequentialValidation::save_initial_seqval, OptionBase::buildoption, "If true, this SequentialValidation object will be saved in sequential_validation.psave. \n"); declareOption( ol, "save_data_sets", &SequentialValidation::save_data_sets, OptionBase::buildoption, "If true, the data sets (train/test) for each split will be saved. \n"); declareOption( ol, "save_test_outputs", &SequentialValidation::save_test_outputs, OptionBase::buildoption, "If true, the outputs of the tests will be saved in test_outputs.pmat \n"); declareOption( ol, "save_test_costs", &SequentialValidation::save_test_costs, OptionBase::buildoption, "If true, the costs of the tests will be saved in test_costs.pmat \n"); declareOption( ol, "save_stat_collectors", &SequentialValidation::save_stat_collectors, OptionBase::buildoption, "If true, stat collectors of each data sets (train/test) will be saved for each split. \n"); declareOption( ol, "provide_learner_expdir", &SequentialValidation::provide_learner_expdir, OptionBase::buildoption, "If true, learning results from the learner will be saved. \n"); declareOption( ol, "save_sequence_stats", &SequentialValidation::save_sequence_stats, OptionBase::buildoption, "Whether the statistics accumulated at each time step should\n" "be saved in the file \"sequence_stats.pmat\". WARNING: this\n" "file can get big! (Default = 1, i.e. true)"); declareOption( ol, "report_memory_usage", &SequentialValidation::report_memory_usage, OptionBase::buildoption, "Whether to report memory usage in a directory expdir/MemoryUsage.\n" "Memory usage is reported AT THE BEGINNING OF EACH time-step, using\n" "both the /proc/PID/status method, and the 'mem_usage PID' method\n" "(if available). This is only supported on Linux at the moment.\n" "(Default = false)"); declareOption( ol, "measure_after_train", &SequentialValidation::measure_after_train, OptionBase::buildoption, "List of options to \"measure\" AFTER training at each timestep, but\n" "BEFORE testing. The options are specified as a list of pairs\n" "'option':'filename', where the option is measured with respect to the\n" "sequential validation object itself. Hence, if the learner contains\n" "an option 'advisor' that you want to save at each time step, you would\n" "write [\"learner.advisor\":\"advisor.psave\"]. The files are saved in the\n" "splitdir directory, which is unique for each timestep."); inherited::declareOptions(ol); }
static const PPath& PLearn::SequentialValidation::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::Object.
Definition at line 261 of file SequentialValidation.h.
SequentialValidation * PLearn::SequentialValidation::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 56 of file SequentialValidation.cc.
OptionList & PLearn::SequentialValidation::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 56 of file SequentialValidation.cc.
OptionMap & PLearn::SequentialValidation::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 56 of file SequentialValidation.cc.
RemoteMethodMap & PLearn::SequentialValidation::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 56 of file SequentialValidation.cc.
void PLearn::SequentialValidation::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be.
Reimplemented from PLearn::Object.
Definition at line 639 of file SequentialValidation.cc.
References accessory_learners, accessory_train_stats, costs, dataset, PLearn::deepCopyField(), dummy_costs, dummy_output, global_stats_vm, input, learner, PLearn::Object::makeDeepCopyFromShallowCopy(), measure_after_test, measure_after_train, output, sequence_stats, split_stats_vm, statnames, statspecs, stcol, target, test_stats, timewise_statnames, timewise_stats, timewise_stats_vm, timewise_statspecs, and train_stats.
{ inherited::makeDeepCopyFromShallowCopy(copies); deepCopyField(train_stats, copies); deepCopyField(accessory_train_stats, copies); deepCopyField(test_stats, copies); deepCopyField(sequence_stats, copies); deepCopyField(timewise_stats, copies); deepCopyField(stcol, copies); deepCopyField(statspecs, copies); deepCopyField(timewise_statspecs, copies); deepCopyField(global_stats_vm, copies); deepCopyField(split_stats_vm, copies); deepCopyField(timewise_stats_vm, copies); deepCopyField(input, copies); deepCopyField(target, copies); deepCopyField(dummy_output, copies); deepCopyField(dummy_costs, copies); deepCopyField(output, copies); deepCopyField(costs, copies); deepCopyField(dataset, copies); deepCopyField(learner, copies); deepCopyField(accessory_learners, copies); deepCopyField(statnames, copies); deepCopyField(timewise_statnames, copies); deepCopyField(measure_after_train, copies); deepCopyField(measure_after_test, copies); }
int PLearn::SequentialValidation::maxTimeStep | ( | ) | const [protected, virtual] |
Utility method to return the highest possible timestep plus 1.
Definition at line 515 of file SequentialValidation.cc.
References dataset, PLearn::VMat::length(), and PLASSERT.
Referenced by run().
void PLearn::SequentialValidation::measureOptions | ( | const TVec< pair< string, string > > & | options, |
PPath | where_to_save | ||
) | [protected, virtual] |
Utility function to measure a list of options and save them in the specified directory.
Definition at line 521 of file SequentialValidation.cc.
References PLearn::Object::getOption(), i, n, PLearn::openFile(), and PLearn::PStream::raw_ascii.
Referenced by run().
{ for (int i=0, n=options.size() ; i<n ; ++i) { const string& optionname = options[i].first; PPath filename = where_to_save / options[i].second; string optvalue = getOption(optionname); PStream out = openFile(filename, PStream::raw_ascii, "w"); out << optvalue; } }
void PLearn::SequentialValidation::reportMemoryUsage | ( | int | t | ) | [protected, virtual] |
Utility method to report the amount of memory used at timestep t.
Definition at line 478 of file SequentialValidation.cc.
References PLearn::append_slash(), expdir, and PLearn::tostring().
Referenced by run().
{ pid_t pid = getpid(); char t_str[100]; sprintf(t_str, "%05d", t); string memdir = append_slash(expdir) + "MemoryUsage"; string method1 = string("cat /proc/")+tostring(pid)+"/status > " + memdir + "/status_" + t_str; string method2 = string("mem_usage ")+tostring(pid)+" > " + memdir + "/mem_usage_" + t_str; system(method1.c_str()); system(method2.c_str()); }
void PLearn::SequentialValidation::reportStats | ( | const Vec & | global_result | ) | [protected, virtual] |
Utility method to report statistics.
Definition at line 468 of file SequentialValidation.cc.
References expdir, report_stats, and PLearn::saveAscii().
Referenced by run().
{ if (!report_stats) return; saveAscii(expdir+"global_result.avec", global_result); // saveAscii(expdir+"predictions.amat", learner->predictions); // saveAscii(expdir+"errors.amat", learner->errors, learner->getTestCostNames()); }
void PLearn::SequentialValidation::run | ( | ) | [virtual] |
The main method; runs the experiment.
Reimplemented from PLearn::Object.
Definition at line 237 of file SequentialValidation.cc.
References a, accessory_learners, PLearn::append_slash(), costs, createStatCollectors(), createStatSpecs(), createStatVMats(), PLearn::endl(), expdir, PLearn::force_mkdir(), global_stats_vm, init_train_size, PLearn::StatSpec::intstatname, last_test_time, learner, PLearn::TVec< T >::length(), maxTimeStep(), measure_after_test, measure_after_train, measureOptions(), output, PLearn::pathexists(), PLERROR, report_memory_usage, report_stats, reportMemoryUsage(), reportStats(), PLearn::TVec< T >::resize(), PLearn::Object::save(), save_data_sets, save_final_model, save_initial_model, save_initial_seqval, save_stat_collectors, save_test_costs, save_test_outputs, sequence_stats, setExperimentDirectory(), PLearn::StatSpec::setnum, setTestStartTime(), shouldTrain(), PLearn::TVec< T >::size(), split_stats_vm, statnames, statspecs, stcol, PLearn::TVec< T >::subVec(), test_stats, testLearners(), testVMat(), timewise_statnames, timewise_stats, timewise_stats_vm, timewise_statspecs, PLearn::tostring(), train_stats, trainLearners(), trainVMat(), warmup_size, and warmupModel().
{ if (expdir=="") PLERROR("No expdir specified for SequentialValidation."); else { if(pathexists(expdir)) PLERROR("Directory (or file) %s already exists. First move it out of the way.", expdir.c_str()); if(!force_mkdir(expdir)) PLERROR("Could not create experiment directory %s", expdir.c_str()); } if (!learner) PLERROR("SequentialValidation::run: learner not specified."); if (warmup_size >= init_train_size) PLERROR("SequentialValidation::run: 'warmup_size' must be strictly smaller than " "'init_train_size'"); if (warmup_size < 0 || init_train_size < 0) PLERROR("SequentialValidation::run: negative warmup_size or init_train_size."); // Get a first dataset to set inputsize() and targetsize() VMat train_vmat = trainVMat(init_train_size); for ( int a=0; a < accessory_learners.length(); a++ ) accessory_learners[a]->setTrainingSet( train_vmat, false ); learner->setTrainingSet( train_vmat, false ); setExperimentDirectory( append_slash(expdir) ); // If we need to report memory usage, create the appropriate directory if (report_memory_usage) force_mkdir( expdir / "MemoryUsage" ); // Save this experiment description in the expdir (buildoptions only) if (save_initial_seqval) PLearn::save(expdir / "sequential_validation.psave", *this); // Create the stat collectors and set them into the learner(s) createStatCollectors(); createStatSpecs(); // Warm up the model before starting the real experiment; this is done // after setting the training stats collectors into everybody... if (warmup_size > 0) warmupModel(warmup_size); // Create all VMatrix related to saving statistics if (report_stats) createStatVMats(); // Final model initialization before the test setTestStartTime(init_train_size, true /* call_build */); VMat test_outputs; VMat test_costs; if (save_test_outputs) test_outputs = new FileVMatrix(expdir / "test_outputs.pmat",0, learner->outputsize()); if (save_test_costs) test_costs = new FileVMatrix(expdir / "test_costs.pmat",0, learner->getTestCostNames()); // Some further initializations int maxt = (last_test_time >= 0? last_test_time : maxTimeStep() - 1); int splitnum = 0; output.resize(learner->outputsize()); costs.resize(learner->nTestCosts()); for (int t=init_train_size; t <= maxt; t++, splitnum++) { #ifdef DEBUG cout << "SequentialValidation::run() -- sub_train.length = " << t << " et sub_test.length = " << t+horizon << endl; #endif if (report_memory_usage) reportMemoryUsage(t); // Create splitdirs PPath splitdir = expdir / "test_t="+tostring(t); if (save_data_sets || save_initial_model || save_stat_collectors || save_final_model || measure_after_train.size() > 0 || measure_after_test.size() > 0 ) force_mkdir(splitdir); // Ensure a first train and, afterwards, train only if we arrive at an allowed // training time-step if ( t == init_train_size || shouldTrain(t)) { // Compute training set. Don't compute test set right away in case // it's a complicated structure that cannot co-exist with an // instantiated training set VMat sub_train = trainVMat(t); if (save_data_sets) PLearn::save(splitdir / "training_set.psave", sub_train); if (save_initial_model) PLearn::save(splitdir / "initial_learner.psave",learner); // Perform train trainLearners(sub_train); // Save post-train stuff if (save_stat_collectors) PLearn::save(splitdir / "train_stats.psave",train_stats); if (save_final_model) PLearn::save(splitdir / "final_learner.psave",learner); measureOptions(measure_after_train, splitdir); } // TEST: simply use computeOutputAndCosts for 1 observation in this // implementation VMat sub_test = testVMat(t); testLearners(sub_test); // Save what is required from the test run if (save_data_sets) PLearn::save(splitdir / "test_set.psave", sub_test); if (test_outputs) test_outputs->appendRow(output); if (test_costs) test_costs->appendRow(costs); if (save_stat_collectors) PLearn::save(splitdir / "test_stats.psave",test_stats); measureOptions(measure_after_test, splitdir); const int nstats = statnames.size(); Vec splitres(1+nstats); splitres[0] = splitnum; // Compute statnames for this split only for(int k=0; k<nstats; k++) { StatSpec& sp = statspecs[k]; if (sp.setnum>=stcol.length()) PLERROR("SequentialValidation::run, trying to access a test set (test%d) beyond the last one (test%d)", sp.setnum, stcol.length()-1); splitres[k+1] = stcol[sp.setnum]->getStat(sp.intstatname); } if (split_stats_vm) split_stats_vm->appendRow(splitres); // Add to overall stats collector sequence_stats->update(splitres.subVec(1,nstats)); // Now compute timewise statnames. First loop is on the inner // statistics; then update the stats collector; then loop on the outer // statistics if (timewise_stats_vm) { const int timewise_nstats = timewise_statnames.size(); Vec timewise_res(timewise_nstats); for (int k=0; k<timewise_nstats; ++k) { StatSpec& sp = timewise_statspecs[k]; if (sp.setnum>=stcol.length()) PLERROR("SequentialValidation::run, trying to access a test set " "(test%d) beyond the last one (test%d)", sp.setnum, stcol.length()-1); timewise_res[k] = stcol[sp.setnum]->getStat(sp.intstatname); } timewise_stats->update(timewise_res); for (int k=0; k<timewise_nstats; ++k) timewise_res[k] = timewise_stats->getStats(k).getStat(timewise_statspecs[k].extstat); timewise_stats_vm->appendRow(timewise_res); } } sequence_stats->finalize(); const int nstats = statnames.size(); Vec global_result(nstats); for (int k=0; k<nstats; k++) global_result[k] = sequence_stats->getStats(k).getStat(statspecs[k].extstat); if (global_stats_vm) global_stats_vm->appendRow(global_result); reportStats(global_result); }
void PLearn::SequentialValidation::setExperimentDirectory | ( | const PPath & | _expdir | ) | [virtual] |
Definition at line 461 of file SequentialValidation.cc.
References expdir, learner, and provide_learner_expdir.
Referenced by run().
{ expdir = _expdir; if(provide_learner_expdir) learner->setExperimentDirectory(expdir / "Model"); }
void PLearn::SequentialValidation::setTestStartTime | ( | int | test_start_time, |
bool | call_build = true |
||
) | [virtual] |
Set the test-start time of learner and accessory learners; call resetInternalState() and optionally build()
Definition at line 431 of file SequentialValidation.cc.
References a, accessory_learners, learner, PLearn::TVec< T >::length(), n, and PLASSERT.
Referenced by run(), and warmupModel().
{ // Ensure correct build of learner and reset internal state. We call // setTestStartTime TWICE, because some learners need it before build, // and because other learners, such as SequentialSelector-types, will not // have finished to construct the complete structure of sub-learners // until AFTER build, and we want the setTestStartTime() message to // propagate to everybody. PLASSERT( test_start_time > 0 ); // Start with the accessory learners for (int a=0, n=accessory_learners.length() ; a<n ; ++a ) { if (call_build) { accessory_learners[a]->setTestStartTime(test_start_time); accessory_learners[a]->build(); } accessory_learners[a]->setTestStartTime(test_start_time); accessory_learners[a]->resetInternalState(); } // And now the main learner if (call_build) { learner->setTestStartTime(test_start_time); learner->build(); } learner->setTestStartTime(test_start_time); learner->resetInternalState(); }
Utility method that returns true if train() should be called at timestep t.
By default, this is determined from the 'train_step' option, but can be redefined in derived classes
Definition at line 494 of file SequentialValidation.cc.
References init_train_size, and train_step.
Referenced by run().
{ if ( train_step <= 0 ) return false; return (t - init_train_size) % train_step == 0; }
void PLearn::SequentialValidation::testLearners | ( | VMat | test_set | ) | [virtual] |
Test learner on LAST OBSERVATION of test_set; also call computeOutputAndCosts on accessory learners.
Definition at line 622 of file SequentialValidation.cc.
References a, accessory_learners, costs, dummy_costs, dummy_output, PLearn::VMat::getExample(), input, learner, PLearn::TVec< T >::length(), PLearn::VMat::length(), n, output, target, and test_stats.
Referenced by run(), and warmupModel().
{ real weight; test_set.getExample(test_set.length()-1, input, target, weight); for (int a=0, n=accessory_learners.length() ; a<n ; ++a ) { accessory_learners[a]->setTestSet(test_set); // temporary hack accessory_learners[a]->computeOutputAndCosts(input, target, dummy_output, dummy_costs); } test_stats->forget(); learner->setTestSet(test_set); // temporary hack learner->computeOutputAndCosts(input, target, output, costs); test_stats->update(costs); test_stats->finalize(); }
Utility method to return the test VMatrix at timestep t (i.e.
all timesteps 0..t, t included)
Definition at line 509 of file SequentialValidation.cc.
References dataset, PLASSERT, and PLearn::VMat::subMatRows().
Referenced by run(), and warmupModel().
void PLearn::SequentialValidation::trainLearners | ( | VMat | training_set | ) | [virtual] |
Train the main learner (and accessory learners)
Definition at line 608 of file SequentialValidation.cc.
References a, accessory_learners, accessory_train_stats, learner, PLearn::TVec< T >::length(), n, and train_stats.
Referenced by run(), and warmupModel().
{ for (int a=0, n=accessory_learners.length(); a<n ; ++a) { accessory_train_stats->forget(); accessory_learners[a]->setTrainingSet(training_set, false); accessory_learners[a]->train(); } train_stats->forget(); learner->setTrainingSet(training_set, false); learner->train(); train_stats->finalize(); }
Utility method to return the training VMatrix at timestep t (i.e.
all timesteps 0..t-1, t excluded)
Definition at line 502 of file SequentialValidation.cc.
References dataset, PLASSERT, and PLearn::VMat::subMatRows().
Referenced by run(), and warmupModel().
{ // exclude t, last training pair is (t-2,t-1) PLASSERT( dataset ); return dataset.subMatRows(0,t); }
void PLearn::SequentialValidation::warmupModel | ( | int | warmup_size | ) | [virtual] |
If warmup_size > 0, warmup the learner before running the experiment.
Definition at line 417 of file SequentialValidation.cc.
References init_train_size, PLASSERT, setTestStartTime(), testLearners(), testVMat(), trainLearners(), and trainVMat().
Referenced by run().
{ PLASSERT( warmup_size < init_train_size ); setTestStartTime(init_train_size - warmup_size, true /* call_build */); for (int t = init_train_size-warmup_size ; t<init_train_size ; ++t) { VMat sub_train = trainVMat(t); // train trainLearners(sub_train); VMat sub_test = testVMat(t); // test testLearners(sub_test); } }
Reimplemented from PLearn::Object.
Definition at line 261 of file SequentialValidation.h.
Accessory learners that must be managed in parallel with the main one.
Definition at line 136 of file SequentialValidation.h.
Referenced by createStatCollectors(), declareOptions(), makeDeepCopyFromShallowCopy(), run(), setTestStartTime(), testLearners(), and trainLearners().
Training stat collector for accessory learners.
Definition at line 64 of file SequentialValidation.h.
Referenced by createStatCollectors(), makeDeepCopyFromShallowCopy(), and trainLearners().
Vec PLearn::SequentialValidation::costs [protected] |
Definition at line 98 of file SequentialValidation.h.
Referenced by makeDeepCopyFromShallowCopy(), run(), and testLearners().
The training/test set.
Definition at line 128 of file SequentialValidation.h.
Referenced by build_(), declareOptions(), makeDeepCopyFromShallowCopy(), maxTimeStep(), testVMat(), and trainVMat().
Vec PLearn::SequentialValidation::dummy_costs [protected] |
for accessory_learners
Definition at line 96 of file SequentialValidation.h.
Referenced by makeDeepCopyFromShallowCopy(), and testLearners().
Vec PLearn::SequentialValidation::dummy_output [protected] |
for accessory_learners
Definition at line 95 of file SequentialValidation.h.
Referenced by makeDeepCopyFromShallowCopy(), and testLearners().
the directory where everything will be saved
Definition at line 145 of file SequentialValidation.h.
Referenced by createStatVMats(), declareOptions(), reportMemoryUsage(), reportStats(), run(), and setExperimentDirectory().
VMat PLearn::SequentialValidation::global_stats_vm [protected] |
vmat where to save global result stats specified in statnames
Definition at line 85 of file SequentialValidation.h.
Referenced by createStatVMats(), makeDeepCopyFromShallowCopy(), and run().
Size of first training set (default: 1)
Definition at line 107 of file SequentialValidation.h.
Referenced by declareOptions(), run(), shouldTrain(), and warmupModel().
Vec PLearn::SequentialValidation::input [protected] |
Buffers.
Definition at line 94 of file SequentialValidation.h.
Referenced by makeDeepCopyFromShallowCopy(), and testLearners().
The last time-step to use for testing (Default = -1, i.e. use all data)
Definition at line 125 of file SequentialValidation.h.
Referenced by declareOptions(), and run().
The StatefulLearner that will be tested.
Definition at line 131 of file SequentialValidation.h.
Referenced by createStatCollectors(), createStatVMats(), declareOptions(), makeDeepCopyFromShallowCopy(), run(), setExperimentDirectory(), setTestStartTime(), testLearners(), and trainLearners().
TVec< pair<string,string> > PLearn::SequentialValidation::measure_after_test |
List of options to "measure" AFTER test, in the same format as 'measure_after_test'.
Definition at line 182 of file SequentialValidation.h.
Referenced by makeDeepCopyFromShallowCopy(), and run().
TVec< pair<string,string> > PLearn::SequentialValidation::measure_after_train |
List of options to "measure" AFTER training at each timestep, but BEFORE testing.
The options are specified as a list of pairs 'option':'filename', where the option is measured with respect to the sequential validation object itself. Hence, if the learner contains an option 'advisor' that you want to save at each time step, you would write ["learner.advisor":"advisor.psave"]. The files are saved in the splitdir directory, which is unique for each timestep.
Definition at line 178 of file SequentialValidation.h.
Referenced by declareOptions(), makeDeepCopyFromShallowCopy(), and run().
Vec PLearn::SequentialValidation::output [protected] |
Definition at line 97 of file SequentialValidation.h.
Referenced by makeDeepCopyFromShallowCopy(), run(), and testLearners().
Definition at line 155 of file SequentialValidation.h.
Referenced by declareOptions(), and setExperimentDirectory().
Whether to report memory usage in a directory expdir/MemoryUsage.
Memory usage is reported AT THE BEGINNING OF EACH time-step, using both the /proc/PID/status method, and the 'mem_usage PID' method (if available). This is only supported on Linux at the moment. (Default = false)
Definition at line 167 of file SequentialValidation.h.
Referenced by declareOptions(), and run().
Definition at line 147 of file SequentialValidation.h.
Referenced by declareOptions(), reportStats(), and run().
Definition at line 151 of file SequentialValidation.h.
Referenced by declareOptions(), and run().
Definition at line 148 of file SequentialValidation.h.
Referenced by declareOptions(), and run().
Definition at line 149 of file SequentialValidation.h.
Referenced by declareOptions(), and run().
Definition at line 150 of file SequentialValidation.h.
Referenced by declareOptions(), and run().
Whether the statistics accumulated at each time step should be saved in the file "sequence_stats.pmat".
WARNING: this file can get big! (Default = 1, i.e. true)
Definition at line 160 of file SequentialValidation.h.
Referenced by createStatVMats(), and declareOptions().
Definition at line 154 of file SequentialValidation.h.
Referenced by declareOptions(), and run().
Definition at line 153 of file SequentialValidation.h.
Referenced by declareOptions(), and run().
Definition at line 152 of file SequentialValidation.h.
Referenced by declareOptions(), and run().
Sequence stat collector.
Definition at line 70 of file SequentialValidation.h.
Referenced by createStatCollectors(), makeDeepCopyFromShallowCopy(), and run().
VMat PLearn::SequentialValidation::split_stats_vm [protected] |
vmat where to save per split result stats
Definition at line 88 of file SequentialValidation.h.
Referenced by createStatVMats(), makeDeepCopyFromShallowCopy(), and run().
Global statistics or split statistics to be computed.
Definition at line 139 of file SequentialValidation.h.
Referenced by createStatSpecs(), createStatVMats(), declareOptions(), makeDeepCopyFromShallowCopy(), and run().
TVec<StatSpec> PLearn::SequentialValidation::statspecs [protected] |
Statspec corresponding to statnames.
Definition at line 79 of file SequentialValidation.h.
Referenced by createStatSpecs(), createStatVMats(), makeDeepCopyFromShallowCopy(), and run().
TVec< PP<VecStatsCollector> > PLearn::SequentialValidation::stcol [protected] |
Definition at line 76 of file SequentialValidation.h.
Referenced by createStatCollectors(), makeDeepCopyFromShallowCopy(), and run().
Vec PLearn::SequentialValidation::target [protected] |
Definition at line 94 of file SequentialValidation.h.
Referenced by makeDeepCopyFromShallowCopy(), and testLearners().
PP<VecStatsCollector> PLearn::SequentialValidation::test_stats [protected] |
Test stat collector.
Definition at line 67 of file SequentialValidation.h.
Referenced by createStatCollectors(), makeDeepCopyFromShallowCopy(), run(), and testLearners().
Timewise statistics to be computed.
Definition at line 142 of file SequentialValidation.h.
Referenced by createStatSpecs(), createStatVMats(), declareOptions(), makeDeepCopyFromShallowCopy(), and run().
Timewise stat collector.
Definition at line 73 of file SequentialValidation.h.
Referenced by createStatCollectors(), makeDeepCopyFromShallowCopy(), and run().
VMat PLearn::SequentialValidation::timewise_stats_vm [protected] |
vmat where to save timewise statistics
Definition at line 91 of file SequentialValidation.h.
Referenced by createStatVMats(), makeDeepCopyFromShallowCopy(), and run().
TVec<StatSpec> PLearn::SequentialValidation::timewise_statspecs [protected] |
Statspec corresponding to timewise_statnames.
Definition at line 82 of file SequentialValidation.h.
Referenced by createStatSpecs(), createStatVMats(), makeDeepCopyFromShallowCopy(), and run().
PP<VecStatsCollector> PLearn::SequentialValidation::train_stats [protected] |
Training stat collector for main learner.
Definition at line 61 of file SequentialValidation.h.
Referenced by createStatCollectors(), makeDeepCopyFromShallowCopy(), run(), and trainLearners().
At how many timesteps must we retrain? (default: 1) If this is zero, train() is never called.
Definition at line 122 of file SequentialValidation.h.
Referenced by declareOptions(), and shouldTrain().
If specified, this is a number of time-steps that are taken FROM THE END of init_train_size to start "testing" (i.e.
alternating between train and test), but WITHOUT ACCUMULATING ANY TEST STATISTICS. In other words, this is a "warmup" period just before the true test. Before starting the real test period, the setTestStartTime() method is called on the learner, followed by resetInternalState(). Note that the very first "init_train_size" is REDUCED by the warmup_size.
Definition at line 118 of file SequentialValidation.h.
Referenced by declareOptions(), and run().