PLearn 0.1
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Protected Member Functions | Static Protected Member Functions | Protected Attributes | Private Types | Private Member Functions
PLearn::SequentialValidation Class Reference

#include <SequentialValidation.h>

Inheritance diagram for PLearn::SequentialValidation:
Inheritance graph
[legend]
Collaboration diagram for PLearn::SequentialValidation:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 SequentialValidation ()
 Default constructor.
virtual void build ()
 Simply calls inherited::build() then build_()
virtual void setExperimentDirectory (const PPath &_expdir)
virtual void run ()
 The main method; runs the experiment.
virtual void warmupModel (int warmup_size)
 If warmup_size > 0, warmup the learner before running the experiment.
virtual void setTestStartTime (int test_start_time, bool call_build=true)
 Set the test-start time of learner and accessory learners; call resetInternalState() and optionally build()
virtual void createStatCollectors ()
 Create the stat collectors.
virtual void createStatSpecs ()
 Create the stat specs.
virtual void createStatVMats ()
 Create the vmatrix required for saving the statistics.
virtual void trainLearners (VMat training_set)
 Train the main learner (and accessory learners)
virtual void testLearners (VMat test_set)
 Test learner on LAST OBSERVATION of test_set; also call computeOutputAndCosts on accessory learners.
void makeDeepCopyFromShallowCopy (CopiesMap &copies)
 Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be.
virtual string classname () const
virtual OptionListgetOptionList () const
virtual OptionMapgetOptionMap () const
virtual RemoteMethodMapgetRemoteMethodMap () const
virtual SequentialValidationdeepCopy (CopiesMap &copies) const

Static Public Member Functions

static string _classname_ ()
static OptionList_getOptionList_ ()
static RemoteMethodMap_getRemoteMethodMap_ ()
static Object_new_instance_for_typemap_ ()
static bool _isa_ (const Object *o)
static void _static_initialize_ ()
static const PPathdeclaringFile ()

Public Attributes

int init_train_size
 Size of first training set (default: 1)
int warmup_size
 If specified, this is a number of time-steps that are taken FROM THE END of init_train_size to start "testing" (i.e.
int train_step
 At how many timesteps must we retrain? (default: 1) If this is zero, train() is never called.
int last_test_time
 The last time-step to use for testing (Default = -1, i.e. use all data)
VMat dataset
 The training/test set.
PP< StatefulLearnerlearner
 The StatefulLearner that will be tested.
TVec< PP< StatefulLearner > > accessory_learners
TVec< string > statnames
 Global statistics or split statistics to be computed.
TVec< string > timewise_statnames
 Timewise statistics to be computed.
PPath expdir
 the directory where everything will be saved
bool report_stats
bool save_final_model
bool save_initial_model
bool save_initial_seqval
bool save_data_sets
bool save_test_outputs
bool save_test_costs
bool save_stat_collectors
bool provide_learner_expdir
bool save_sequence_stats
 Whether the statistics accumulated at each time step should be saved in the file "sequence_stats.pmat".
bool report_memory_usage
 Whether to report memory usage in a directory expdir/MemoryUsage.
TVec< pair< string, string > > measure_after_train
 List of options to "measure" AFTER training at each timestep, but BEFORE testing.
TVec< pair< string, string > > measure_after_test
 List of options to "measure" AFTER test, in the same format as 'measure_after_test'.

Static Public Attributes

static StaticInitializer _static_initializer_

Protected Member Functions

virtual void reportStats (const Vec &global_result)
 Utility method to report statistics.
virtual void reportMemoryUsage (int t)
 Utility method to report the amount of memory used at timestep t.
virtual bool shouldTrain (int t)
 Utility method that returns true if train() should be called at timestep t.
virtual VMat trainVMat (int t)
 Utility method to return the training VMatrix at timestep t (i.e.
virtual VMat testVMat (int t)
 Utility method to return the test VMatrix at timestep t (i.e.
virtual int maxTimeStep () const
 Utility method to return the highest possible timestep plus 1.
virtual void measureOptions (const TVec< pair< string, string > > &options, PPath where_to_save)
 Utility function to measure a list of options and save them in the specified directory.

Static Protected Member Functions

static void declareOptions (OptionList &ol)
 Declare this class' options.

Protected Attributes

PP< VecStatsCollectortrain_stats
 Training stat collector for main learner.
PP< VecStatsCollectoraccessory_train_stats
 Training stat collector for accessory learners.
PP< VecStatsCollectortest_stats
 Test stat collector.
PP< VecStatsCollectorsequence_stats
 Sequence stat collector.
PP< VecStatsCollectortimewise_stats
 Timewise stat collector.
TVec< PP< VecStatsCollector > > stcol
TVec< StatSpecstatspecs
 Statspec corresponding to statnames.
TVec< StatSpectimewise_statspecs
 Statspec corresponding to timewise_statnames.
VMat global_stats_vm
 vmat where to save global result stats specified in statnames
VMat split_stats_vm
 vmat where to save per split result stats
VMat timewise_stats_vm
 vmat where to save timewise statistics
Vec input
 Buffers.
Vec target
Vec dummy_output
 for accessory_learners
Vec dummy_costs
 for accessory_learners
Vec output
Vec costs

Private Types

typedef Object inherited

Private Member Functions

void build_ ()
 This does the actual building.

Detailed Description

Definition at line 53 of file SequentialValidation.h.


Member Typedef Documentation

Reimplemented from PLearn::Object.

Definition at line 55 of file SequentialValidation.h.


Constructor & Destructor Documentation

PLearn::SequentialValidation::SequentialValidation ( )

Member Function Documentation

string PLearn::SequentialValidation::_classname_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 56 of file SequentialValidation.cc.

OptionList & PLearn::SequentialValidation::_getOptionList_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 56 of file SequentialValidation.cc.

RemoteMethodMap & PLearn::SequentialValidation::_getRemoteMethodMap_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 56 of file SequentialValidation.cc.

bool PLearn::SequentialValidation::_isa_ ( const Object o) [static]

Reimplemented from PLearn::Object.

Definition at line 56 of file SequentialValidation.cc.

Object * PLearn::SequentialValidation::_new_instance_for_typemap_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 56 of file SequentialValidation.cc.

StaticInitializer SequentialValidation::_static_initializer_ & PLearn::SequentialValidation::_static_initialize_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 56 of file SequentialValidation.cc.

void PLearn::SequentialValidation::build ( ) [virtual]

Simply calls inherited::build() then build_()

Reimplemented from PLearn::Object.

Definition at line 83 of file SequentialValidation.cc.

References PLearn::Object::build(), and build_().

Here is the call graph for this function:

void PLearn::SequentialValidation::build_ ( ) [private]

This does the actual building.

Reimplemented from PLearn::Object.

Definition at line 77 of file SequentialValidation.cc.

References dataset, and PLearn::VMat::width().

Referenced by build().

{
    if ( dataset && dataset->inputsize() < 0 )
        dataset->defineSizes(dataset->width(), 0, 0);
}

Here is the call graph for this function:

Here is the caller graph for this function:

string PLearn::SequentialValidation::classname ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 56 of file SequentialValidation.cc.

void PLearn::SequentialValidation::createStatCollectors ( ) [virtual]

Create the stat collectors.

Definition at line 533 of file SequentialValidation.cc.

References a, accessory_learners, accessory_train_stats, learner, PLearn::TVec< T >::length(), n, PLearn::TVec< T >::resize(), sequence_stats, stcol, test_stats, timewise_stats, and train_stats.

Referenced by run().

{
    // Always manage the accessory_learners first since they may be used
    // within the main trader.
    accessory_train_stats = new VecStatsCollector(); 
    for (int a=0, n=accessory_learners.length() ; a<n ; ++a)
        accessory_learners[a]->setTrainStatsCollector( accessory_train_stats );
  
    // stats for a train on one split
    stcol.resize(2);
    train_stats = new VecStatsCollector();
    train_stats->setFieldNames(learner->getTrainCostNames());
    learner->setTrainStatsCollector(train_stats);  
    stcol[0] = train_stats;

    // stats for a test on one split
    test_stats = new VecStatsCollector();
    test_stats->setFieldNames(learner->getTestCostNames());
    stcol[1] = test_stats;

    // stats over all sequence
    sequence_stats = new VecStatsCollector();

    // timewise stats (may not be used)
    timewise_stats = new VecStatsCollector();
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::SequentialValidation::createStatSpecs ( ) [virtual]

Create the stat specs.

Definition at line 560 of file SequentialValidation.cc.

References PLearn::TVec< T >::length(), PLearn::TVec< T >::resize(), statnames, statspecs, timewise_statnames, and timewise_statspecs.

Referenced by run().

{
    // Stat specs (overall)
    const int nstats = statnames.length();
    statspecs.resize(nstats);
    for (int k=0; k<nstats; k++)
        statspecs[k].init(statnames[k]);

    // Stat specs (timewise)
    const int timewise_nstats = timewise_statnames.length();
    timewise_statspecs.resize(timewise_nstats);
    for (int k=0; k<timewise_nstats; ++k)
        timewise_statspecs[k].init(timewise_statnames[k]);
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::SequentialValidation::createStatVMats ( ) [virtual]

Create the vmatrix required for saving the statistics.

Definition at line 575 of file SequentialValidation.cc.

References expdir, global_stats_vm, PLearn::join(), learner, save_sequence_stats, PLearn::saveStringInFile(), PLearn::TVec< T >::size(), split_stats_vm, statnames, statspecs, timewise_statnames, timewise_stats_vm, and timewise_statspecs.

Referenced by run().

{
    TVec<string> traincostnames = learner->getTrainCostNames();
    TVec<string> testcostnames  = learner->getTestCostNames();
    const int nstats = statnames.size();
    const int timewise_nstats = timewise_statnames.size();

    saveStringInFile(expdir / "train_cost_names.txt", join(traincostnames,"\n")+"\n");
    saveStringInFile(expdir / "test_cost_names.txt",  join(testcostnames,"\n")+"\n");

    global_stats_vm = new FileVMatrix(expdir / "global_stats.pmat", 0, nstats);
    for(int k=0; k<nstats; k++)
        global_stats_vm->declareField(k,statspecs[k].statName());
    global_stats_vm->saveFieldInfos();

    if (save_sequence_stats) {
        split_stats_vm = new FileVMatrix(expdir+"sequence_stats.pmat", 0,
                                         1+nstats);
        split_stats_vm->declareField(0,"splitnum");
        for(int k=0; k<nstats; k++)
            split_stats_vm->declareField(k+1,statspecs[k].setname + "." + statspecs[k].intstatname);
        split_stats_vm->saveFieldInfos();
    }

    if (timewise_nstats > 0) {
        timewise_stats_vm = new FileVMatrix(expdir+"timewise_stats.pmat", 0,
                                            timewise_nstats);
        for (int k=0; k<timewise_nstats; ++k)
            timewise_stats_vm->declareField(k, timewise_statspecs[k].statName());
        timewise_stats_vm->saveFieldInfos();
    }
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::SequentialValidation::declareOptions ( OptionList ol) [static, protected]

Declare this class' options.

Reimplemented from PLearn::Object.

Definition at line 89 of file SequentialValidation.cc.

References accessory_learners, PLearn::OptionBase::buildoption, dataset, PLearn::declareOption(), PLearn::Object::declareOptions(), expdir, init_train_size, last_test_time, learner, measure_after_train, provide_learner_expdir, report_memory_usage, report_stats, save_data_sets, save_final_model, save_initial_model, save_initial_seqval, save_sequence_stats, save_stat_collectors, save_test_costs, save_test_outputs, statnames, timewise_statnames, train_step, and warmup_size.

{
    declareOption(
        ol, "report_stats", &SequentialValidation::report_stats,
        OptionBase::buildoption,
        "If true, the computed global statistics specified in statnames will be saved in global_stats.pmat \n"
        "and the corresponding per-split statistics will be saved in split_stats.pmat \n"
        "For reference, all cost names (as given by the learner's getTrainCostNames() and getTestCostNames() ) \n"
        "will be reported in files train_cost_names.txt and test_cost_names.txt");

    declareOption(
        ol, "statnames", &SequentialValidation::statnames,
        OptionBase::buildoption,
        "A list of global statistics we are interested in.\n"
        "These are strings of the form S1[S2[dataset.cost_name]] where:\n"
        "  - dataset is train or test1 or test2 ... (train being \n"
        "    the first dataset in a split, test1 the second, ...) \n"
        "  - cost_name is one of the training or test cost names (depending on dataset) understood \n"
        "    by the underlying learner (see its getTrainCostNames and getTestCostNames methods) \n"
        "  - S1 and S2 are a statistic, i.e. one of: E (expectation), V(variance), MIN, MAX, STDDEV, ... \n"
        "    S2 is computed over the samples of a given dataset split. S1 is over the splits. \n");

    declareOption(
        ol, "timewise_statnames", &SequentialValidation::timewise_statnames,
        OptionBase::buildoption,
        "Statistics to be collected into a VecStatsCollector at each timestep.");
  
    declareOption(
        ol, "expdir", &SequentialValidation::expdir,
        OptionBase::buildoption,
        "Path of this experiment's directory in which to save all experiment results (will be created if it does not already exist). \n");

    declareOption(
        ol, "learner", &SequentialValidation::learner,
        OptionBase::buildoption,
        "The SequentialLearner to train/test. \n");

    declareOption(
        ol, "accessory_learners", &SequentialValidation::accessory_learners,
        OptionBase::buildoption,
        "Accessory learners that must be managed in parallel with the main one." );
  
    declareOption(
        ol, "dataset", &SequentialValidation::dataset,
        OptionBase::buildoption,
        "The dataset to use for training/testing. \n");

    declareOption(
        ol, "init_train_size", &SequentialValidation::init_train_size,
        OptionBase::buildoption,
        "Size of the first training set.  Before starting the train/test cycle,\n"
        "the method setTestStartTime() is called on the learner with init_train_size\n"
        "as argument.");

    declareOption(
        ol, "warmup_size", &SequentialValidation::warmup_size,
        OptionBase::buildoption,
        "If specified, this is a number of time-steps that are taken FROM THE\n"
        "END of init_train_size to start \"testing\" (i.e. alternating between\n"
        "train and test), but WITHOUT ACCUMULATING ANY TEST STATISTICS.  In\n"
        "other words, this is a \"warmup\" period just before the true test.\n"
        "Before starting the real test period, the setTestStartTime() method is\n"
        "called on the learner, followed by resetInternalState().  Note that\n"
        "the very first \"init_train_size\" is REDUCED by the warmup_size.\n");
  
    declareOption(
        ol, "train_step", &SequentialValidation::train_step,
        OptionBase::buildoption,
        "At how many timesteps must we retrain? (default: 1)");
  
    declareOption(
        ol, "last_test_time", &SequentialValidation::last_test_time,
        OptionBase::buildoption,
        "The last time-step to use for testing (Default = -1, i.e. use all data)");
  
    declareOption(
        ol, "save_final_model", &SequentialValidation::save_final_model,
        OptionBase::buildoption,
        "If true, the final model will be saved in model.psave \n");

    declareOption(
        ol, "save_initial_model", &SequentialValidation::save_initial_model,
        OptionBase::buildoption,
        "If true, the initial model will be saved in initial_model.psave. \n");

    declareOption(
        ol, "save_initial_seqval", &SequentialValidation::save_initial_seqval,
        OptionBase::buildoption,
        "If true, this SequentialValidation object will be saved in sequential_validation.psave. \n");

    declareOption(
        ol, "save_data_sets", &SequentialValidation::save_data_sets,
        OptionBase::buildoption,
        "If true, the data sets (train/test) for each split will be saved. \n");

    declareOption(
        ol, "save_test_outputs", &SequentialValidation::save_test_outputs,
        OptionBase::buildoption,
        "If true, the outputs of the tests will be saved in test_outputs.pmat \n");

    declareOption(
        ol, "save_test_costs", &SequentialValidation::save_test_costs,
        OptionBase::buildoption,
        "If true, the costs of the tests will be saved in test_costs.pmat \n");

    declareOption(
        ol, "save_stat_collectors", &SequentialValidation::save_stat_collectors,
        OptionBase::buildoption,
        "If true, stat collectors of each data sets (train/test) will be saved for each split. \n");

    declareOption(
        ol, "provide_learner_expdir", &SequentialValidation::provide_learner_expdir,
        OptionBase::buildoption,
        "If true, learning results from the learner will be saved. \n");

    declareOption(
        ol, "save_sequence_stats",
        &SequentialValidation::save_sequence_stats,
        OptionBase::buildoption,
        "Whether the statistics accumulated at each time step should\n"
        "be saved in the file \"sequence_stats.pmat\".  WARNING: this\n"
        "file can get big!  (Default = 1, i.e. true)");

    declareOption(
        ol, "report_memory_usage",
        &SequentialValidation::report_memory_usage,
        OptionBase::buildoption,
        "Whether to report memory usage in a directory expdir/MemoryUsage.\n"
        "Memory usage is reported AT THE BEGINNING OF EACH time-step, using\n"
        "both the /proc/PID/status method, and the 'mem_usage PID' method\n"
        "(if available).  This is only supported on Linux at the moment.\n"
        "(Default = false)");

    declareOption(
        ol, "measure_after_train",
        &SequentialValidation::measure_after_train,
        OptionBase::buildoption,
        "List of options to \"measure\" AFTER training at each timestep, but\n"
        "BEFORE testing.  The options are specified as a list of pairs\n"
        "'option':'filename', where the option is measured with respect to the\n"
        "sequential validation object itself.  Hence, if the learner contains\n"
        "an option 'advisor' that you want to save at each time step, you would\n"
        "write [\"learner.advisor\":\"advisor.psave\"].  The files are saved in the\n"
        "splitdir directory, which is unique for each timestep.");
  
    inherited::declareOptions(ol);
}

Here is the call graph for this function:

static const PPath& PLearn::SequentialValidation::declaringFile ( ) [inline, static]

Reimplemented from PLearn::Object.

Definition at line 261 of file SequentialValidation.h.

SequentialValidation * PLearn::SequentialValidation::deepCopy ( CopiesMap copies) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 56 of file SequentialValidation.cc.

OptionList & PLearn::SequentialValidation::getOptionList ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 56 of file SequentialValidation.cc.

OptionMap & PLearn::SequentialValidation::getOptionMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 56 of file SequentialValidation.cc.

RemoteMethodMap & PLearn::SequentialValidation::getRemoteMethodMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 56 of file SequentialValidation.cc.

void PLearn::SequentialValidation::makeDeepCopyFromShallowCopy ( CopiesMap copies) [virtual]

Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be.

Reimplemented from PLearn::Object.

Definition at line 639 of file SequentialValidation.cc.

References accessory_learners, accessory_train_stats, costs, dataset, PLearn::deepCopyField(), dummy_costs, dummy_output, global_stats_vm, input, learner, PLearn::Object::makeDeepCopyFromShallowCopy(), measure_after_test, measure_after_train, output, sequence_stats, split_stats_vm, statnames, statspecs, stcol, target, test_stats, timewise_statnames, timewise_stats, timewise_stats_vm, timewise_statspecs, and train_stats.

Here is the call graph for this function:

int PLearn::SequentialValidation::maxTimeStep ( ) const [protected, virtual]

Utility method to return the highest possible timestep plus 1.

Definition at line 515 of file SequentialValidation.cc.

References dataset, PLearn::VMat::length(), and PLASSERT.

Referenced by run().

{
    PLASSERT( dataset );
    return dataset.length();
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::SequentialValidation::measureOptions ( const TVec< pair< string, string > > &  options,
PPath  where_to_save 
) [protected, virtual]

Utility function to measure a list of options and save them in the specified directory.

Definition at line 521 of file SequentialValidation.cc.

References PLearn::Object::getOption(), i, n, PLearn::openFile(), and PLearn::PStream::raw_ascii.

Referenced by run().

{
    for (int i=0, n=options.size() ; i<n ; ++i) {
        const string& optionname = options[i].first;
        PPath filename = where_to_save / options[i].second;
        string optvalue = getOption(optionname);
        PStream out = openFile(filename, PStream::raw_ascii, "w");
        out << optvalue;
    }
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::SequentialValidation::reportMemoryUsage ( int  t) [protected, virtual]

Utility method to report the amount of memory used at timestep t.

Definition at line 478 of file SequentialValidation.cc.

References PLearn::append_slash(), expdir, and PLearn::tostring().

Referenced by run().

{
    pid_t pid = getpid();
    char t_str[100];
    sprintf(t_str, "%05d", t);

    string memdir = append_slash(expdir) + "MemoryUsage";
    string method1 = string("cat /proc/")+tostring(pid)+"/status > "
        + memdir + "/status_" + t_str;
    string method2 = string("mem_usage ")+tostring(pid)+" > "
        + memdir + "/mem_usage_" + t_str;

    system(method1.c_str());
    system(method2.c_str());
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::SequentialValidation::reportStats ( const Vec global_result) [protected, virtual]

Utility method to report statistics.

Definition at line 468 of file SequentialValidation.cc.

References expdir, report_stats, and PLearn::saveAscii().

Referenced by run().

{
    if (!report_stats)
        return;
  
    saveAscii(expdir+"global_result.avec", global_result);
//  saveAscii(expdir+"predictions.amat", learner->predictions);
//  saveAscii(expdir+"errors.amat", learner->errors, learner->getTestCostNames());
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::SequentialValidation::run ( ) [virtual]

The main method; runs the experiment.

Reimplemented from PLearn::Object.

Definition at line 237 of file SequentialValidation.cc.

References a, accessory_learners, PLearn::append_slash(), costs, createStatCollectors(), createStatSpecs(), createStatVMats(), PLearn::endl(), expdir, PLearn::force_mkdir(), global_stats_vm, init_train_size, PLearn::StatSpec::intstatname, last_test_time, learner, PLearn::TVec< T >::length(), maxTimeStep(), measure_after_test, measure_after_train, measureOptions(), output, PLearn::pathexists(), PLERROR, report_memory_usage, report_stats, reportMemoryUsage(), reportStats(), PLearn::TVec< T >::resize(), PLearn::Object::save(), save_data_sets, save_final_model, save_initial_model, save_initial_seqval, save_stat_collectors, save_test_costs, save_test_outputs, sequence_stats, setExperimentDirectory(), PLearn::StatSpec::setnum, setTestStartTime(), shouldTrain(), PLearn::TVec< T >::size(), split_stats_vm, statnames, statspecs, stcol, PLearn::TVec< T >::subVec(), test_stats, testLearners(), testVMat(), timewise_statnames, timewise_stats, timewise_stats_vm, timewise_statspecs, PLearn::tostring(), train_stats, trainLearners(), trainVMat(), warmup_size, and warmupModel().

{  
    if (expdir=="")
        PLERROR("No expdir specified for SequentialValidation.");
    else
    {
        if(pathexists(expdir))
            PLERROR("Directory (or file) %s already exists. First move it out of the way.", expdir.c_str());
        if(!force_mkdir(expdir))
            PLERROR("Could not create experiment directory %s", expdir.c_str());
    }

    if (!learner)
        PLERROR("SequentialValidation::run: learner not specified.");

    if (warmup_size >= init_train_size)
        PLERROR("SequentialValidation::run: 'warmup_size' must be strictly smaller than "
                "'init_train_size'");

    if (warmup_size < 0 || init_train_size < 0)
        PLERROR("SequentialValidation::run: negative warmup_size or init_train_size.");
  
    // Get a first dataset to set inputsize() and targetsize()
    VMat train_vmat = trainVMat(init_train_size);
    for ( int a=0; a < accessory_learners.length(); a++ )
        accessory_learners[a]->setTrainingSet( train_vmat, false );
    learner->setTrainingSet( train_vmat, false );
  
    setExperimentDirectory( append_slash(expdir) );

    // If we need to report memory usage, create the appropriate directory
    if (report_memory_usage)
        force_mkdir( expdir / "MemoryUsage" );

    // Save this experiment description in the expdir (buildoptions only)
    if (save_initial_seqval)
        PLearn::save(expdir / "sequential_validation.psave", *this);

    // Create the stat collectors and set them into the learner(s)
    createStatCollectors();
    createStatSpecs();
  
    // Warm up the model before starting the real experiment; this is done
    // after setting the training stats collectors into everybody...
    if (warmup_size > 0)
        warmupModel(warmup_size);

    // Create all VMatrix related to saving statistics
    if (report_stats)
        createStatVMats();

    // Final model initialization before the test
    setTestStartTime(init_train_size, true /* call_build */);
  
    VMat test_outputs;
    VMat test_costs;
    if (save_test_outputs)
        test_outputs = new FileVMatrix(expdir / "test_outputs.pmat",0,
                                       learner->outputsize());
    if (save_test_costs)
        test_costs = new FileVMatrix(expdir / "test_costs.pmat",0,
                                     learner->getTestCostNames());

    // Some further initializations
    int maxt = (last_test_time >= 0? last_test_time : maxTimeStep() - 1);
    int splitnum = 0;
    output.resize(learner->outputsize());
    costs.resize(learner->nTestCosts());
    for (int t=init_train_size; t <= maxt; t++, splitnum++)
    {
#ifdef DEBUG
        cout << "SequentialValidation::run() -- sub_train.length = " << t << " et sub_test.length = " << t+horizon << endl;
#endif
        if (report_memory_usage)
            reportMemoryUsage(t);

        // Create splitdirs
        PPath splitdir = expdir / "test_t="+tostring(t);
        if (save_data_sets                 ||
            save_initial_model             ||
            save_stat_collectors           ||
            save_final_model               ||
            measure_after_train.size() > 0 ||
            measure_after_test.size()  > 0  )
            force_mkdir(splitdir);
    
        // Ensure a first train and, afterwards, train only if we arrive at an allowed
        // training time-step
        if ( t == init_train_size || shouldTrain(t)) {
            // Compute training set.  Don't compute test set right away in case
            // it's a complicated structure that cannot co-exist with an
            // instantiated training set
            VMat sub_train = trainVMat(t);
            if (save_data_sets)
                PLearn::save(splitdir / "training_set.psave", sub_train);
            if (save_initial_model)
                PLearn::save(splitdir / "initial_learner.psave",learner);

            // Perform train
            trainLearners(sub_train);
      
            // Save post-train stuff
            if (save_stat_collectors)
                PLearn::save(splitdir / "train_stats.psave",train_stats);
            if (save_final_model)
                PLearn::save(splitdir / "final_learner.psave",learner);
            measureOptions(measure_after_train, splitdir);
        }

        // TEST: simply use computeOutputAndCosts for 1 observation in this
        // implementation
        VMat sub_test = testVMat(t);
        testLearners(sub_test);
    
        // Save what is required from the test run
        if (save_data_sets)
            PLearn::save(splitdir / "test_set.psave", sub_test);
        if (test_outputs)
            test_outputs->appendRow(output);
        if (test_costs)
            test_costs->appendRow(costs);
        if (save_stat_collectors)
            PLearn::save(splitdir / "test_stats.psave",test_stats);
        measureOptions(measure_after_test, splitdir);

        const int nstats = statnames.size();
        Vec splitres(1+nstats);
        splitres[0] = splitnum;

        // Compute statnames for this split only
        for(int k=0; k<nstats; k++)
        {
            StatSpec& sp = statspecs[k];
            if (sp.setnum>=stcol.length())
                PLERROR("SequentialValidation::run, trying to access a test set (test%d) beyond the last one (test%d)",
                        sp.setnum, stcol.length()-1);
            splitres[k+1] = stcol[sp.setnum]->getStat(sp.intstatname);
        }

        if (split_stats_vm)
            split_stats_vm->appendRow(splitres);

        // Add to overall stats collector
        sequence_stats->update(splitres.subVec(1,nstats));

        // Now compute timewise statnames.  First loop is on the inner
        // statistics; then update the stats collector; then loop on the outer
        // statistics
        if (timewise_stats_vm) {
            const int timewise_nstats = timewise_statnames.size();
            Vec timewise_res(timewise_nstats);
            for (int k=0; k<timewise_nstats; ++k) {
                StatSpec& sp = timewise_statspecs[k];
                if (sp.setnum>=stcol.length())
                    PLERROR("SequentialValidation::run, trying to access a test set "
                            "(test%d) beyond the last one (test%d)",
                            sp.setnum, stcol.length()-1);
                timewise_res[k] = stcol[sp.setnum]->getStat(sp.intstatname);
            }
            timewise_stats->update(timewise_res);
            for (int k=0; k<timewise_nstats; ++k)
                timewise_res[k] =
                    timewise_stats->getStats(k).getStat(timewise_statspecs[k].extstat);
            timewise_stats_vm->appendRow(timewise_res);
        }
    }

    sequence_stats->finalize();

    const int nstats = statnames.size();
    Vec global_result(nstats);
    for (int k=0; k<nstats; k++)
        global_result[k] = sequence_stats->getStats(k).getStat(statspecs[k].extstat);

    if (global_stats_vm)
        global_stats_vm->appendRow(global_result);
  
    reportStats(global_result);
}

Here is the call graph for this function:

void PLearn::SequentialValidation::setExperimentDirectory ( const PPath _expdir) [virtual]

Definition at line 461 of file SequentialValidation.cc.

References expdir, learner, and provide_learner_expdir.

Referenced by run().

{
    expdir = _expdir;
    if(provide_learner_expdir)
        learner->setExperimentDirectory(expdir / "Model");
}

Here is the caller graph for this function:

void PLearn::SequentialValidation::setTestStartTime ( int  test_start_time,
bool  call_build = true 
) [virtual]

Set the test-start time of learner and accessory learners; call resetInternalState() and optionally build()

Definition at line 431 of file SequentialValidation.cc.

References a, accessory_learners, learner, PLearn::TVec< T >::length(), n, and PLASSERT.

Referenced by run(), and warmupModel().

{
    // Ensure correct build of learner and reset internal state.  We call
    // setTestStartTime TWICE, because some learners need it before build,
    // and because other learners, such as SequentialSelector-types, will not
    // have finished to construct the complete structure of sub-learners
    // until AFTER build, and we want the setTestStartTime() message to
    // propagate to everybody.

    PLASSERT( test_start_time > 0 );
  
    // Start with the accessory learners
    for (int a=0, n=accessory_learners.length() ; a<n ; ++a ) {
        if (call_build) {
            accessory_learners[a]->setTestStartTime(test_start_time);
            accessory_learners[a]->build();
        }
        accessory_learners[a]->setTestStartTime(test_start_time);
        accessory_learners[a]->resetInternalState();
    }

    // And now the main learner
    if (call_build) {
        learner->setTestStartTime(test_start_time);
        learner->build();
    }
    learner->setTestStartTime(test_start_time);
    learner->resetInternalState();
}

Here is the call graph for this function:

Here is the caller graph for this function:

bool PLearn::SequentialValidation::shouldTrain ( int  t) [protected, virtual]

Utility method that returns true if train() should be called at timestep t.

By default, this is determined from the 'train_step' option, but can be redefined in derived classes

Definition at line 494 of file SequentialValidation.cc.

References init_train_size, and train_step.

Referenced by run().

{
    if ( train_step <= 0 )
        return false;

    return (t - init_train_size) % train_step == 0;
}

Here is the caller graph for this function:

void PLearn::SequentialValidation::testLearners ( VMat  test_set) [virtual]

Test learner on LAST OBSERVATION of test_set; also call computeOutputAndCosts on accessory learners.

Definition at line 622 of file SequentialValidation.cc.

References a, accessory_learners, costs, dummy_costs, dummy_output, PLearn::VMat::getExample(), input, learner, PLearn::TVec< T >::length(), PLearn::VMat::length(), n, output, target, and test_stats.

Referenced by run(), and warmupModel().

{
    real weight;
    test_set.getExample(test_set.length()-1, input, target, weight);
    for (int a=0, n=accessory_learners.length() ; a<n ; ++a )
    {
        accessory_learners[a]->setTestSet(test_set);         // temporary hack
        accessory_learners[a]->computeOutputAndCosts(input, target,
                                                     dummy_output, dummy_costs);
    }
    test_stats->forget();
    learner->setTestSet(test_set);           // temporary hack
    learner->computeOutputAndCosts(input, target, output, costs);
    test_stats->update(costs);
    test_stats->finalize();
}

Here is the call graph for this function:

Here is the caller graph for this function:

VMat PLearn::SequentialValidation::testVMat ( int  t) [protected, virtual]

Utility method to return the test VMatrix at timestep t (i.e.

all timesteps 0..t, t included)

Definition at line 509 of file SequentialValidation.cc.

References dataset, PLASSERT, and PLearn::VMat::subMatRows().

Referenced by run(), and warmupModel().

{
    PLASSERT( dataset );
    return dataset.subMatRows(0,t+1);
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::SequentialValidation::trainLearners ( VMat  training_set) [virtual]

Train the main learner (and accessory learners)

Definition at line 608 of file SequentialValidation.cc.

References a, accessory_learners, accessory_train_stats, learner, PLearn::TVec< T >::length(), n, and train_stats.

Referenced by run(), and warmupModel().

{
    for (int a=0, n=accessory_learners.length(); a<n ; ++a)
    {
        accessory_train_stats->forget();
        accessory_learners[a]->setTrainingSet(training_set, false);
        accessory_learners[a]->train();        
    }
    train_stats->forget();
    learner->setTrainingSet(training_set, false);
    learner->train();
    train_stats->finalize();  
}

Here is the call graph for this function:

Here is the caller graph for this function:

VMat PLearn::SequentialValidation::trainVMat ( int  t) [protected, virtual]

Utility method to return the training VMatrix at timestep t (i.e.

all timesteps 0..t-1, t excluded)

Definition at line 502 of file SequentialValidation.cc.

References dataset, PLASSERT, and PLearn::VMat::subMatRows().

Referenced by run(), and warmupModel().

{
    // exclude t, last training pair is (t-2,t-1)
    PLASSERT( dataset );
    return dataset.subMatRows(0,t);
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::SequentialValidation::warmupModel ( int  warmup_size) [virtual]

If warmup_size > 0, warmup the learner before running the experiment.

Definition at line 417 of file SequentialValidation.cc.

References init_train_size, PLASSERT, setTestStartTime(), testLearners(), testVMat(), trainLearners(), and trainVMat().

Referenced by run().

{
    PLASSERT( warmup_size < init_train_size );
    setTestStartTime(init_train_size - warmup_size, true /* call_build */);
  
    for (int t = init_train_size-warmup_size ; t<init_train_size ; ++t) {
        VMat sub_train = trainVMat(t);           // train
        trainLearners(sub_train);

        VMat sub_test = testVMat(t);             // test
        testLearners(sub_test);
    }
}

Here is the call graph for this function:

Here is the caller graph for this function:


Member Data Documentation

Reimplemented from PLearn::Object.

Definition at line 261 of file SequentialValidation.h.

Accessory learners that must be managed in parallel with the main one.

Definition at line 136 of file SequentialValidation.h.

Referenced by createStatCollectors(), declareOptions(), makeDeepCopyFromShallowCopy(), run(), setTestStartTime(), testLearners(), and trainLearners().

Training stat collector for accessory learners.

Definition at line 64 of file SequentialValidation.h.

Referenced by createStatCollectors(), makeDeepCopyFromShallowCopy(), and trainLearners().

Definition at line 98 of file SequentialValidation.h.

Referenced by makeDeepCopyFromShallowCopy(), run(), and testLearners().

The training/test set.

Definition at line 128 of file SequentialValidation.h.

Referenced by build_(), declareOptions(), makeDeepCopyFromShallowCopy(), maxTimeStep(), testVMat(), and trainVMat().

for accessory_learners

Definition at line 96 of file SequentialValidation.h.

Referenced by makeDeepCopyFromShallowCopy(), and testLearners().

for accessory_learners

Definition at line 95 of file SequentialValidation.h.

Referenced by makeDeepCopyFromShallowCopy(), and testLearners().

the directory where everything will be saved

Definition at line 145 of file SequentialValidation.h.

Referenced by createStatVMats(), declareOptions(), reportMemoryUsage(), reportStats(), run(), and setExperimentDirectory().

vmat where to save global result stats specified in statnames

Definition at line 85 of file SequentialValidation.h.

Referenced by createStatVMats(), makeDeepCopyFromShallowCopy(), and run().

Size of first training set (default: 1)

Definition at line 107 of file SequentialValidation.h.

Referenced by declareOptions(), run(), shouldTrain(), and warmupModel().

Buffers.

Definition at line 94 of file SequentialValidation.h.

Referenced by makeDeepCopyFromShallowCopy(), and testLearners().

The last time-step to use for testing (Default = -1, i.e. use all data)

Definition at line 125 of file SequentialValidation.h.

Referenced by declareOptions(), and run().

List of options to "measure" AFTER test, in the same format as 'measure_after_test'.

Definition at line 182 of file SequentialValidation.h.

Referenced by makeDeepCopyFromShallowCopy(), and run().

List of options to "measure" AFTER training at each timestep, but BEFORE testing.

The options are specified as a list of pairs 'option':'filename', where the option is measured with respect to the sequential validation object itself. Hence, if the learner contains an option 'advisor' that you want to save at each time step, you would write ["learner.advisor":"advisor.psave"]. The files are saved in the splitdir directory, which is unique for each timestep.

Definition at line 178 of file SequentialValidation.h.

Referenced by declareOptions(), makeDeepCopyFromShallowCopy(), and run().

Definition at line 97 of file SequentialValidation.h.

Referenced by makeDeepCopyFromShallowCopy(), run(), and testLearners().

Definition at line 155 of file SequentialValidation.h.

Referenced by declareOptions(), and setExperimentDirectory().

Whether to report memory usage in a directory expdir/MemoryUsage.

Memory usage is reported AT THE BEGINNING OF EACH time-step, using both the /proc/PID/status method, and the 'mem_usage PID' method (if available). This is only supported on Linux at the moment. (Default = false)

Definition at line 167 of file SequentialValidation.h.

Referenced by declareOptions(), and run().

Definition at line 147 of file SequentialValidation.h.

Referenced by declareOptions(), reportStats(), and run().

Definition at line 151 of file SequentialValidation.h.

Referenced by declareOptions(), and run().

Definition at line 148 of file SequentialValidation.h.

Referenced by declareOptions(), and run().

Definition at line 149 of file SequentialValidation.h.

Referenced by declareOptions(), and run().

Definition at line 150 of file SequentialValidation.h.

Referenced by declareOptions(), and run().

Whether the statistics accumulated at each time step should be saved in the file "sequence_stats.pmat".

WARNING: this file can get big! (Default = 1, i.e. true)

Definition at line 160 of file SequentialValidation.h.

Referenced by createStatVMats(), and declareOptions().

Definition at line 154 of file SequentialValidation.h.

Referenced by declareOptions(), and run().

Definition at line 153 of file SequentialValidation.h.

Referenced by declareOptions(), and run().

Definition at line 152 of file SequentialValidation.h.

Referenced by declareOptions(), and run().

Sequence stat collector.

Definition at line 70 of file SequentialValidation.h.

Referenced by createStatCollectors(), makeDeepCopyFromShallowCopy(), and run().

vmat where to save per split result stats

Definition at line 88 of file SequentialValidation.h.

Referenced by createStatVMats(), makeDeepCopyFromShallowCopy(), and run().

Global statistics or split statistics to be computed.

Definition at line 139 of file SequentialValidation.h.

Referenced by createStatSpecs(), createStatVMats(), declareOptions(), makeDeepCopyFromShallowCopy(), and run().

Statspec corresponding to statnames.

Definition at line 79 of file SequentialValidation.h.

Referenced by createStatSpecs(), createStatVMats(), makeDeepCopyFromShallowCopy(), and run().

Definition at line 76 of file SequentialValidation.h.

Referenced by createStatCollectors(), makeDeepCopyFromShallowCopy(), and run().

Definition at line 94 of file SequentialValidation.h.

Referenced by makeDeepCopyFromShallowCopy(), and testLearners().

Test stat collector.

Definition at line 67 of file SequentialValidation.h.

Referenced by createStatCollectors(), makeDeepCopyFromShallowCopy(), run(), and testLearners().

Timewise statistics to be computed.

Definition at line 142 of file SequentialValidation.h.

Referenced by createStatSpecs(), createStatVMats(), declareOptions(), makeDeepCopyFromShallowCopy(), and run().

Timewise stat collector.

Definition at line 73 of file SequentialValidation.h.

Referenced by createStatCollectors(), makeDeepCopyFromShallowCopy(), and run().

vmat where to save timewise statistics

Definition at line 91 of file SequentialValidation.h.

Referenced by createStatVMats(), makeDeepCopyFromShallowCopy(), and run().

Statspec corresponding to timewise_statnames.

Definition at line 82 of file SequentialValidation.h.

Referenced by createStatSpecs(), createStatVMats(), makeDeepCopyFromShallowCopy(), and run().

Training stat collector for main learner.

Definition at line 61 of file SequentialValidation.h.

Referenced by createStatCollectors(), makeDeepCopyFromShallowCopy(), run(), and trainLearners().

At how many timesteps must we retrain? (default: 1) If this is zero, train() is never called.

Definition at line 122 of file SequentialValidation.h.

Referenced by declareOptions(), and shouldTrain().

If specified, this is a number of time-steps that are taken FROM THE END of init_train_size to start "testing" (i.e.

alternating between train and test), but WITHOUT ACCUMULATING ANY TEST STATISTICS. In other words, this is a "warmup" period just before the true test. Before starting the real test period, the setTestStartTime() method is called on the learner, followed by resetInternalState(). Note that the very first "init_train_size" is REDUCED by the warmup_size.

Definition at line 118 of file SequentialValidation.h.

Referenced by declareOptions(), and run().


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines