PLearn 0.1
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Static Protected Member Functions | Protected Attributes | Private Types | Private Member Functions
PLearn::SequentialModelSelector Class Reference

#include <SequentialModelSelector.h>

Inheritance diagram for PLearn::SequentialModelSelector:
Inheritance graph
[legend]
Collaboration diagram for PLearn::SequentialModelSelector:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 SequentialModelSelector ()
 Constructor.
real sequenceCost (const Vec &sequence_errors)
 compute the cost of the given sequence of errors (based on the cost_type)
real paired_t_test (const int &m1, const int &m2, int cc=0) const
 Computes a paired t-test between common_cost[cc] series of models m1 and m2.
virtual void build ()
 simply calls inherited::build() then build_()
virtual void setExperimentDirectory (const PPath &_expdir)
 Redefines so that it ALSO calls the method on all the learners in the TVec models.
virtual void train ()
virtual void test (VMat testset, PP< VecStatsCollector > test_stats, VMat testoutputs=0, VMat testcosts=0) const
virtual void computeOutput (const Vec &input, Vec &output) const
 *** SUBCLASS WRITING: ***
virtual void computeCostsFromOutputs (const Vec &input, const Vec &output, const Vec &target, Vec &costs) const
 *** SUBCLASS WRITING: ***
virtual void computeOutputAndCosts (const Vec &input, const Vec &target, Vec &output, Vec &costs) const
 Default calls computeOutput and computeCostsFromOutputs.
virtual void computeCostsOnly (const Vec &input, const Vec &target, Vec &costs) const
 Default calls computeOutputAndCosts.
virtual void matlabSave (const string &matlab_subdir)
virtual TVec< string > getTestCostNames () const
 This should return the names of the costs computed by computeCostsFromOutputs.
virtual TVec< string > getTrainCostNames () const
 This should return the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats.
virtual void forget ()
 *** SUBCLASS WRITING: ***
virtual string classname () const
virtual OptionListgetOptionList () const
virtual OptionMapgetOptionMap () const
virtual RemoteMethodMapgetRemoteMethodMap () const
virtual SequentialModelSelectordeepCopy (CopiesMap &copies) const
virtual void makeDeepCopyFromShallowCopy (CopiesMap &copies)
 Transforms a shallow copy into a deep copy.

Static Public Member Functions

static string _classname_ ()
 Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be.
static OptionList_getOptionList_ ()
static RemoteMethodMap_getRemoteMethodMap_ ()
static Object_new_instance_for_typemap_ ()
static bool _isa_ (const Object *o)
static void _static_initialize_ ()
static const PPathdeclaringFile ()

Public Attributes

bool stepwise_save
TVec< PP< SequentialLearner > > models
 List of all the models.
TVec< string > model_names
 If the user desires to provide a name for each model instead of model_i.
TVec< string > common_costs
int comparison_type
int comparison_window

Static Public Attributes

static StaticInitializer _static_initializer_

Static Protected Member Functions

static void declareOptions (OptionList &ol)
 Declare this class' options.

Protected Attributes

TMat< intcommon_cost_indices
 See common_costs options for details. Row index: model. Column index: common_cost.
TVec< intbest_model
Vec sequence_costs
bool report_paired_T_tests

Private Types

typedef SequentialLearner inherited

Private Member Functions

void build_ ()
 This does the actual building.
void checkModelNames () const

Detailed Description

Definition at line 49 of file SequentialModelSelector.h.


Member Typedef Documentation

Reimplemented from PLearn::SequentialLearner.

Definition at line 55 of file SequentialModelSelector.h.


Constructor & Destructor Documentation

PLearn::SequentialModelSelector::SequentialModelSelector ( )

Constructor.

Definition at line 48 of file SequentialModelSelector.cc.


Member Function Documentation

string PLearn::SequentialModelSelector::_classname_ ( ) [static]

Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be.

Reimplemented from PLearn::SequentialLearner.

Definition at line 46 of file SequentialModelSelector.cc.

OptionList & PLearn::SequentialModelSelector::_getOptionList_ ( ) [static]

Reimplemented from PLearn::SequentialLearner.

Definition at line 46 of file SequentialModelSelector.cc.

RemoteMethodMap & PLearn::SequentialModelSelector::_getRemoteMethodMap_ ( ) [static]

Reimplemented from PLearn::SequentialLearner.

Definition at line 46 of file SequentialModelSelector.cc.

bool PLearn::SequentialModelSelector::_isa_ ( const Object o) [static]

Reimplemented from PLearn::SequentialLearner.

Definition at line 46 of file SequentialModelSelector.cc.

Object * PLearn::SequentialModelSelector::_new_instance_for_typemap_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 46 of file SequentialModelSelector.cc.

StaticInitializer SequentialModelSelector::_static_initializer_ & PLearn::SequentialModelSelector::_static_initialize_ ( ) [static]

Reimplemented from PLearn::SequentialLearner.

Definition at line 46 of file SequentialModelSelector.cc.

void PLearn::SequentialModelSelector::build ( ) [virtual]

simply calls inherited::build() then build_()

Reimplemented from PLearn::SequentialLearner.

Definition at line 106 of file SequentialModelSelector.cc.

References PLearn::SequentialLearner::build(), and build_().

Here is the call graph for this function:

void PLearn::SequentialModelSelector::build_ ( ) [private]

This does the actual building.

Reimplemented from PLearn::SequentialLearner.

Definition at line 71 of file SequentialModelSelector.cc.

References best_model, c, common_cost_indices, common_costs, forget(), PLearn::SequentialLearner::horizon, PLearn::SequentialLearner::init_train_size, PLearn::TVec< T >::length(), m, PLearn::SequentialLearner::max_seq_len, models, PLERROR, PLWARNING, PLearn::TVec< T >::resize(), PLearn::TMat< T >::resize(), and sequence_costs.

Referenced by build().

{
    // Precondition check on init_train_size
    if(init_train_size < horizon)
    {
        PLWARNING("The init_train_size provided by the user was %d. However since the horizon is %d\n"
                  "the init_train_size will be modified to be %d, the horizon value. This modification is\n"
                  "necessary because, within a model selection context, a first internal test is needed to\n"
                  "determine the first best model (see the SequentialModelSelector::train method\n"
                  "implementation). If you want to avoid getting this warning, you should set init_train_size\n"
                  "to horizon (%d) directly.", 
                  init_train_size, horizon, horizon, horizon);
        init_train_size = horizon;
    }
  
    int nb_models = models.length();
    if(nb_models < 1)
        PLERROR("The 'models' option is mandatory in a SequentialModelSelector.");
      
    int nb_common_costs = common_costs.length();
    if(nb_common_costs < 1)
        PLERROR("The 'common_costs' requires at least 1 cost from which models will\n"
                "be compared to choose the best model.");  

    common_cost_indices.resize(nb_models, nb_common_costs); 
    for(int m=0; m < nb_models; m++)
        for(int c=0; c < nb_common_costs; c++) 
            common_cost_indices(m, c) = models[m]->getTestCostIndex( common_costs[c] );
  
    best_model.resize(max_seq_len);
    sequence_costs.resize(nb_models);

    forget();
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::SequentialModelSelector::checkModelNames ( ) const [private]

Definition at line 460 of file SequentialModelSelector.cc.

References PLearn::TVec< T >::length(), m, model_names, models, PLERROR, PLearn::TVec< T >::resize(), and PLearn::tostring().

Referenced by getTestCostNames(), and setExperimentDirectory().

{
    int nb_models = models.length();
    int nb_names = model_names.length();
    if(nb_names == 0)
    {
        model_names.resize(nb_models);
        for(int m=0; m < nb_models; m++)
            model_names[m] = "Model_"+tostring(m);
    }
    else if(nb_names != nb_models)
        PLERROR("Names must be provided for all models (%d = nb_names != nb_models = %d).", nb_names, nb_models);
}

Here is the call graph for this function:

Here is the caller graph for this function:

string PLearn::SequentialModelSelector::classname ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 46 of file SequentialModelSelector.cc.

void PLearn::SequentialModelSelector::computeCostsFromOutputs ( const Vec input,
const Vec output,
const Vec target,
Vec costs 
) const [virtual]

*** SUBCLASS WRITING: ***

This should be defined in subclasses to compute the weighted costs from already computed output. The costs should correspond to the cost names returned by getTestCostNames().

NOTE: In exotic cases, the cost may also depend on some info in the input, that's why the method also gets so see it.

Reimplemented from PLearn::SequentialLearner.

Definition at line 442 of file SequentialModelSelector.cc.

References best_model, PLearn::SequentialLearner::last_train_t, and models.

{
    models[best_model[last_train_t]]->computeCostsFromOutputs(input, output, target, costs);
}
void PLearn::SequentialModelSelector::computeCostsOnly ( const Vec input,
const Vec target,
Vec costs 
) const [virtual]

Default calls computeOutputAndCosts.

This may be overridden if there is a more efficient way to compute the costs directly, without computing the whole output vector.

Reimplemented from PLearn::SequentialLearner.

Definition at line 454 of file SequentialModelSelector.cc.

References best_model, PLearn::SequentialLearner::last_train_t, and models.

{
    models[best_model[last_train_t]]->computeCostsOnly(input, target, costs);
}
void PLearn::SequentialModelSelector::computeOutput ( const Vec input,
Vec output 
) const [virtual]

*** SUBCLASS WRITING: ***

This should be defined in subclasses to compute the output from the input.

Reimplemented from PLearn::SequentialLearner.

Definition at line 437 of file SequentialModelSelector.cc.

References best_model, PLearn::SequentialLearner::last_train_t, and models.

{
    models[best_model[last_train_t]]->computeOutput(input, output);
}
void PLearn::SequentialModelSelector::computeOutputAndCosts ( const Vec input,
const Vec target,
Vec output,
Vec costs 
) const [virtual]

Default calls computeOutput and computeCostsFromOutputs.

You may override this if you have a more efficient way to compute both output and weighted costs at the same time.

Reimplemented from PLearn::SequentialLearner.

Definition at line 448 of file SequentialModelSelector.cc.

References best_model, PLearn::SequentialLearner::last_train_t, and models.

{
    models[best_model[last_train_t]]->computeOutputAndCosts(input, target, output, costs);
}
void PLearn::SequentialModelSelector::declareOptions ( OptionList ol) [static, protected]

Declare this class' options.

Reimplemented from PLearn::SequentialLearner.

Definition at line 112 of file SequentialModelSelector.cc.

References PLearn::OptionBase::buildoption, common_costs, comparison_type, comparison_window, PLearn::declareOption(), PLearn::SequentialLearner::declareOptions(), model_names, models, report_paired_T_tests, and stepwise_save.

{    
    declareOption(ol, "stepwise_save", &SequentialModelSelector::stepwise_save, OptionBase::buildoption,
                  "Does the model selector hass to save errors at each step.\n"
                  "Default: true.");

    declareOption(ol, "models", &SequentialModelSelector::models,
                  OptionBase::buildoption, "List of all the models.\n");

    declareOption(ol, "model_names", &SequentialModelSelector::model_names, OptionBase::buildoption,
                  "If the user desires to provide a name for each model instead of model_i.");

    declareOption(ol, "common_costs", &SequentialModelSelector::common_costs, OptionBase::buildoption,
                  "The names of costs that are common to all models and that the user wishes the model\n"
                  "selector to keep track of. The first one is considered to be the main cost, the one\n"
                  "from which models will be compared to choose the best model.");

    declareOption(ol, "report_paired_T_tests", &SequentialModelSelector::report_paired_T_tests, OptionBase::buildoption,
                  "If true, the model selector will report as costs the paired T tests on common_cost_indices[0] for\n"
                  "models[0] against each other model. The model selector will only report T tests once,\n"
                  "at t=(max_seq_len-1)\n"
                  "\n"
                  "Default: false.");

    declareOption(ol, "comparison_type", &SequentialModelSelector::comparison_type,
                  OptionBase::buildoption, 
                  "From the common_costs list, the first cost is the one from which models will be compared\n"
                  "to choose the best model. But should the best model be chosen according to the\n"
                  "\n"
                  "max/min\n"
                  "  +/-   1: Mean\n"
                  "  +/-   2: Mean / Variance\n"
                  "  +/-   3: more to come.\n"
                  "\n"
                  "of the cost realizations. \n"
                  "Default: 1.\n");

    declareOption(ol, "comparison_window", &SequentialModelSelector::comparison_window, OptionBase::buildoption,
                  "If positive, the comparison performed on the basis of common_cost[0] will be applyed only\n"
                  "the comparison_window last elements of the cost sequence.\n"
                  "Default: -1. (No window)");

    inherited::declareOptions(ol);
}

Here is the call graph for this function:

static const PPath& PLearn::SequentialModelSelector::declaringFile ( ) [inline, static]

Reimplemented from PLearn::SequentialLearner.

Definition at line 182 of file SequentialModelSelector.h.

SequentialModelSelector * PLearn::SequentialModelSelector::deepCopy ( CopiesMap copies) const [virtual]

Reimplemented from PLearn::SequentialLearner.

Definition at line 46 of file SequentialModelSelector.cc.

void PLearn::SequentialModelSelector::forget ( ) [virtual]

*** SUBCLASS WRITING: ***

(Re-)initializes the PLearner in its fresh state (that state may depend on the 'seed' option) and sets 'stage' back to 0 (this is the stage of a fresh learner!)

A typical forget() method should do the following:

  • initialize the learner's parameters, using this random generator
  • stage = 0;

This method is typically called by the build_() method, after it has finished setting up the parameters, and if it deemed useful to set or reset the learner in its fresh state. (remember build may be called after modifying options that do not necessarily require the learner to restart from a fresh state...) forget is also called by the setTrainingSet method, after calling build(), so it will generally be called TWICE during setTrainingSet!

Reimplemented from PLearn::SequentialLearner.

Definition at line 200 of file SequentialModelSelector.cc.

References best_model, PLearn::TVec< T >::fill(), PLearn::SequentialLearner::forget(), i, PLearn::SequentialLearner::max_seq_len, models, PLearn::TVec< T >::resize(), and PLearn::TVec< T >::size().

Referenced by build_().

{
    best_model.resize(max_seq_len);
    best_model.fill(-1);  // by default
    for (int i=0; i<models.size(); i++)
        models[i]->forget();

    inherited::forget();
}

Here is the call graph for this function:

Here is the caller graph for this function:

OptionList & PLearn::SequentialModelSelector::getOptionList ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 46 of file SequentialModelSelector.cc.

OptionMap & PLearn::SequentialModelSelector::getOptionMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 46 of file SequentialModelSelector.cc.

RemoteMethodMap & PLearn::SequentialModelSelector::getRemoteMethodMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 46 of file SequentialModelSelector.cc.

TVec< string > PLearn::SequentialModelSelector::getTestCostNames ( ) const [virtual]

This should return the names of the costs computed by computeCostsFromOutputs.

Implements PLearn::PLearner.

Definition at line 474 of file SequentialModelSelector.cc.

References PLearn::TVec< T >::append(), c, checkModelNames(), common_costs, PLearn::TVec< T >::length(), m, model_names, models, and report_paired_T_tests.

{ 
    TVec<string> tcnames = common_costs;
  
    int nb_models = models.length();
    checkModelNames();

    for(int m=0; m < nb_models; m++)
    {
        TVec<string> tcm = models[m]->getTestCostNames();
        for(int c=0; c < tcm.length(); c++)
            tcnames.append(model_names[m] + "::" + tcm[c]);
    }
  
    if( report_paired_T_tests )
        for(int m=1; m < nb_models; m++)
            tcnames.append("T-test_" + model_names[0] + "_vs_" + model_names[m]);
  
    return tcnames;
}

Here is the call graph for this function:

TVec< string > PLearn::SequentialModelSelector::getTrainCostNames ( ) const [virtual]

This should return the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats.

Implements PLearn::PLearner.

Definition at line 495 of file SequentialModelSelector.cc.

References common_costs.

{ return common_costs; }
void PLearn::SequentialModelSelector::makeDeepCopyFromShallowCopy ( CopiesMap copies) [virtual]

Transforms a shallow copy into a deep copy.

Reimplemented from PLearn::SequentialLearner.

Definition at line 498 of file SequentialModelSelector.cc.

References PLearn::deepCopyField(), PLearn::SequentialLearner::makeDeepCopyFromShallowCopy(), and models.

{
    inherited::makeDeepCopyFromShallowCopy(copies);
    deepCopyField(models, copies);
    //deepCopyField(mean_costs, copies);
} 

Here is the call graph for this function:

void PLearn::SequentialModelSelector::matlabSave ( const string &  matlab_subdir) [virtual]

Any SequentialLearner will have the possiblity to save some data in a matlab 'readable' format. The data will be saved in expdir/matlab_subdir through the global matlabSave function (MatIO.h).

Reimplemented from PLearn::SequentialLearner.

Definition at line 505 of file SequentialModelSelector.cc.

References PLearn::add(), PLearn::TVec< T >::append(), PLearn::append_slash(), common_cost_indices, common_costs, PLearn::concat(), g, PLearn::SequentialLearner::getCostSequence(), PLearn::PLearner::getExperimentDirectory(), PLearn::hconcat(), PLearn::TVec< T >::length(), m, model_names, models, and PLearn::TVec< T >::subVec().

{
    string save_dir = append_slash(getExperimentDirectory()) + matlab_subdir;
    Vec dummy, add(1); add[0] = 0;
  
    TVec<string> legend(1, "ModelSelector");
    legend.append(model_names);
  
    int nb_models = models.length();
    int nb_common_costs = common_costs.length();
    for(int g=0; g < nb_common_costs; g++)
    {
        Vec cs;

        Array<Mat> columns(nb_models+1);
        cs = getCostSequence(g);

        // Since the model selector first runs tests in its train method, the models' sequence
        //  length will probably be to long...
        int msel_cs_len = cs.length(); 

        columns[0] = Mat(cs.length(), 1, cs);
        for(int m=0; m < nb_models; m++)
        {
            cs = models[m]->getCostSequence( common_cost_indices(m, g) );
            cs = cs.subVec(cs.length() - msel_cs_len, msel_cs_len); //... length correction (see the comment above)
            columns[m+1] = Mat(cs.length(), 1, cs);
        }
    
        Mat concat = hconcat(columns);
        PLearn::matlabSave( save_dir, common_costs[g], 
                            concat,
                            add, dummy, legend);
    }
  
    for(int m=0; m < nb_models; m++)
        models[m]->matlabSave(matlab_subdir);
}

Here is the call graph for this function:

real PLearn::SequentialModelSelector::paired_t_test ( const int m1,
const int m2,
int  cc = 0 
) const

Computes a paired t-test between common_cost[cc] series of models m1 and m2.

Definition at line 182 of file SequentialModelSelector.cc.

References PLearn::TMat< T >::column(), common_cost_indices, PLearn::SequentialLearner::errors, PLearn::TVec< T >::length(), PLearn::mean(), models, n, PLERROR, PLearn::remove_missing(), PLearn::sqrt(), PLearn::sumsquare(), PLearn::TMat< T >::toVecCopy(), and u.

Referenced by test().

{
    Vec u = remove_missing(models[m1]->errors.column( common_cost_indices(m1, cc) ).toVecCopy());
    Vec v = remove_missing(models[m2]->errors.column( common_cost_indices(m2, cc) ).toVecCopy());

    int n = u.length();
    if( v.length() != n )
        PLERROR("SequentialModelSelector::paired_t_test -- Can't make a paired t-test on to unequally lengthed vectors (%d != %d).",
                n, v.length());
  
    real ubar = mean(u);
    real vbar = mean(v);
    Vec u2 = u - ubar;
    Vec v2 = v - vbar;

    return (ubar - vbar) * sqrt( n*(n-1) / sumsquare(u2-v2));
}

Here is the call graph for this function:

Here is the caller graph for this function:

real PLearn::SequentialModelSelector::sequenceCost ( const Vec sequence_errors)

compute the cost of the given sequence of errors (based on the cost_type)

Definition at line 157 of file SequentialModelSelector.cc.

References PLearn::abs(), comparison_type, comparison_window, PLearn::TVec< T >::length(), PLearn::mean(), MISSING_VALUE, PLERROR, PLearn::TVec< T >::subVec(), and PLearn::variance().

Referenced by train().

{
    int seq_len = sequence_errors.length();
    if(seq_len == 0)
        return MISSING_VALUE;
  
    int window_start = seq_len - comparison_window;
    Vec windowed;
    if(comparison_window < 1 || window_start < 0)     
        windowed = sequence_errors; // The option is inactive or the sequence is not yet comparison_window long 
    else
        windowed = sequence_errors.subVec(window_start, comparison_window);
  
    int type = abs(comparison_type);
    real mean_ = mean(windowed);
    if (type == 1)
        return mean_;
  
    if (type == 2)
        return mean_ / variance(windowed, mean_);
  
    PLERROR("Invalid comparison type %d!", comparison_type);
    return MISSING_VALUE;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::SequentialModelSelector::setExperimentDirectory ( const PPath _expdir) [virtual]

Redefines so that it ALSO calls the method on all the learners in the TVec models.

Reimplemented from PLearn::PLearner.

Definition at line 55 of file SequentialModelSelector.cc.

References PLearn::append_slash(), checkModelNames(), PLearn::TVec< T >::length(), m, model_names, and models.

{
    int nb_models = models.length();
    checkModelNames();

    PLearner::setExperimentDirectory(_expdir);
    string model_m;
    for(int m=0; m < nb_models; m++)
    {
        model_m = models[m]->getExperimentDirectory();
        if(model_m == "")
            model_m = model_names[m];
        models[m]->setExperimentDirectory(append_slash(_expdir)+ model_m);
    }
}

Here is the call graph for this function:

void PLearn::SequentialModelSelector::test ( VMat  testset,
PP< VecStatsCollector test_stats,
VMat  testoutputs = 0,
VMat  testcosts = 0 
) const [virtual]

*** SUBCLASS WRITING: *** The method should:

  • call computeOutputAndCosts on the test set
  • save the outputs and the costs in the predictions & errors matrices, beginning at position last_call_train_t

Implements PLearn::SequentialLearner.

Definition at line 346 of file SequentialModelSelector.cc.

References PLearn::TVec< T >::append(), PLearn::append_slash(), best_model, common_cost_indices, PLearn::endl(), PLearn::SequentialLearner::errors, PLearn::PLearner::expdir, i, PLearn::SequentialLearner::last_test_t, PLearn::SequentialLearner::last_train_t, PLearn::TVec< T >::length(), PLearn::VMat::length(), m, MAX, PLearn::SequentialLearner::max_seq_len, MISSING_VALUE, models, paired_t_test(), PLWARNING, PLearn::SequentialLearner::predictions, report_paired_T_tests, PLearn::PLearner::report_progress, PLearn::saveAsciiWithoutSize(), PLearn::TVec< T >::size(), stepwise_save, PLearn::tostring(), and PLearn::update().

{
    // Since model selection train method may not train before (init_train_size+horizon)
    //  we have to verify if a first train was actually done prior to entering the 
    //  test core
    if(last_train_t == -1)
    {
        PLWARNING("SequentialModelSelector::test -- Skipped because there were no prior train.");
        return;
    }

    int start_t = MAX(last_train_t+1,last_test_t+1);
    if( test_set.length()-1 < start_t )
        return;

    PP<ProgressBar> pb;
    if (report_progress)
        pb = new ProgressBar("Testing SequentialModelSelector learner",test_set.length());

    TVec< PP<VecStatsCollector> > dummy_stats(models.length());  
    for (int t=start_t; t < test_set.length(); t++)
    { 
        Vec best_model_costs;      
        Vec models_update;
    
        for (int i=0; i<models.size(); i++)
        {    
            if(t == start_t)
                dummy_stats[i] = new VecStatsCollector();

            models[i]->test(test_set, dummy_stats[i]);
      
            if(i == best_model[last_train_t])
            {
                best_model_costs = models[i]->errors(t);
                models_update.append( best_model_costs );
            }
            else
                models_update.append( models[i]->errors(t) );
        }
    
        // The update is composed of the common_costs and the modelwise costs
        Vec update = best_model_costs( common_cost_indices(best_model[last_train_t]) );
        update.append(models_update);
    
        // Report T test processing: only reports T tests once, at t=(max_seq_len-1)
        if( report_paired_T_tests )
            for(int m=1; m < models.length(); m++)
            {
                if( test_set.length() == max_seq_len )
                    update.append( paired_t_test(0, m) );      
                else
                    update.append( MISSING_VALUE );
            }
    
        test_stats->update( update );
  
        predictions(t) << models[best_model[last_train_t]]->predictions(t); 
        errors(t) << update;
        if (testoutputs) testoutputs->appendRow( predictions(t) );
        if (testcosts) testcosts->appendRow(update);

#ifdef DEBUG
        cout << "predictions(" << t << "): " << predictions(t) << endl
             << "errors(" << t << "): " << errors(t) << endl;
#endif

        if(pb)
            pb->update(t);
    }
    if(pb)
        pb->close();

  
    last_test_t = test_set.length()-1;
#ifdef DEBUG
    cout << "SequentialModelSelector.last_test_t = " << last_test_t << endl;
#endif
  

    if(stepwise_save)
    {
        string s1 = append_slash(expdir) + "predictions_test_t=" + tostring(last_test_t);
        saveAsciiWithoutSize(s1, predictions);
    
        string s2 = append_slash(expdir) + "errors_test_t=" + tostring(last_test_t);
        saveAsciiWithoutSize(s2, errors);
    }
}

Here is the call graph for this function:

void PLearn::SequentialModelSelector::train ( ) [virtual]

*** SUBCLASS WRITING: *** Does the actual training. Subclasses must implement this method. The method should upon entry, call setTrainingSet(training_set); The method should:

  • do nothing if we already called it with this value of train.length or a value between [last_train_t-train_step+1,last_train_t]
  • if not, train and update the value of last_train_t
  • in either case, update the value of last_call_train_t

First call to train: if the models[i]->init_train_size = init_train_size-horizon; statement was done during the build and if init_train_size was modified afterwards (e.g. by SequentialValidation) the models would not have been updated properly.

Note also that the sublearners are provided init_train_size-horizon since the model selector needs to go horizon time steps back in time.

Implements PLearn::SequentialLearner.

Definition at line 210 of file SequentialModelSelector.cc.

References PLearn::append_slash(), PLearn::argmax(), PLearn::argmin(), best_model, PLearn::TMat< T >::column(), common_cost_indices, comparison_type, PLearn::endl(), PLearn::SequentialLearner::errors, PLearn::PLearner::expdir, PLearn::SequentialLearner::horizon, i, PLearn::SequentialLearner::init_train_size, PLearn::SequentialLearner::last_call_train_t, PLearn::SequentialLearner::last_test_t, PLearn::SequentialLearner::last_train_t, PLearn::TVec< T >::length(), PLearn::VMat::length(), MAX, models, PLERROR, PLWARNING, PLearn::SequentialLearner::predictions, PLearn::remove_missing(), PLearn::PLearner::report_progress, PLearn::saveAsciiWithoutSize(), sequence_costs, sequenceCost(), PLearn::TVec< T >::size(), stepwise_save, PLearn::PLearner::sub_test(), PLearn::TMat< T >::subMat(), PLearn::VMat::subMatRows(), PLearn::tostring(), PLearn::TMat< T >::toVecCopy(), PLearn::PLearner::train_set, PLearn::PLearner::train_stats, PLearn::SequentialLearner::train_step, and PLearn::update().

{  
    if(last_call_train_t == -1) 
        for (int i=0; i<models.size(); i++)
        {
            models[i]->horizon = horizon;
            models[i]->init_train_size = init_train_size-horizon;
        }

    last_call_train_t = train_set.length()-1;

    // The init_train_size option is the train_set size at which we should start training.
    //  The corresponding time index is (init_train_size-1). See also the build_ precondition
    //  check on init_train_size.
    int init_train_t = init_train_size-1;
    int start_t = MAX(init_train_t, last_train_t+train_step);
    if( start_t < last_test_t )
        PLERROR("SequentialModelSelector::train -- start_t = %d < %d = last_test_t", start_t, last_test_t);

    if( last_call_train_t < start_t )
        return;

    PP<ProgressBar> pb;
    if (report_progress)
        pb = new ProgressBar("Training SequentialModelSelector learner",train_set.length());

    TVec< PP<VecStatsCollector> > dummy_stats(models.length());
    for (int t=start_t; t < train_set.length(); t++)
    {
        // The time t index is equal to a set last row index. Its length will therefore be: 
        int set_length = t+1;

#ifdef DEBUG
        cout << "SequentialModelSelector::train() -- sub_train.length = " << set_length-horizon << " et sub_test.length = " << set_length << endl;
#endif
        
        VMat sub_train = train_set.subMatRows(0,set_length-horizon); // last training pair is (t-2*horizon,t-horizon)
        sub_train->defineSizes(train_set->inputsize(), train_set->targetsize(), train_set->weightsize());

        VMat sub_test  = train_set.subMatRows(0,set_length); // last test pair is (t-horizon,t)
        sub_test->defineSizes(train_set->inputsize(), train_set->targetsize(), train_set->weightsize());

        for (int i=0; i < models.size(); i++)
        {
            if(t == start_t)
                dummy_stats[i] = new VecStatsCollector();

            models[i]->setTrainingSet(sub_train, false);
            models[i]->train();
            models[i]->test(sub_test, dummy_stats[i]); // last cost computed goes at t-1, last prediction at t-1-horizon
      
            Vec sequence_errors = remove_missing(models[i]->errors.column( common_cost_indices(i, 0) ).toVecCopy());
      
#ifdef DEBUG
            cout << "models["<<i<<"]->getTestCostNames()[common_cost_indices(i, 0)]: " 
                 << models[i]->getTestCostNames()[common_cost_indices(i, 0)] << endl;
            PLWARNING("models[%d]->errors.subMat(0,%d,%d,1)", i, common_cost_indices(i, 0), t);
            cout << remove_missing( models[i]->errors.subMat(0,common_cost_indices(i,0),set_length,1).toVecCopy() ) << endl;
            cout << "---\nOR\n---\n" << sequence_errors << endl; 
#endif
      
            sequence_costs[i] = sequenceCost(sequence_errors);
        }
    
        // we set the best model for this time step
        if(comparison_type > 0)
            best_model[t] = argmax(sequence_costs, true);
        else
            best_model[t] = argmin(sequence_costs, true);

        if(best_model[t] == -1)
            best_model[t] = 0;   // All models provided only MISSING_VALUE as common_costs[0] !!!

#ifdef DEBUG
        cout << "sequence_costs: " << sequence_costs << endl;
        cout << "SequentialModelSelector::train() -- t = " << t << " et best_model = " << best_model[t] << endl;
#endif

        if(predictions(t-horizon).hasMissing())
            predictions(t-horizon) << models[best_model[t]]->predictions(t-horizon);
    
#ifdef DEBUG
        cout << "SequentialModelSelector::train() -- train_set.length = " << set_length << endl;
#endif
        if (pb)
            pb->update(t);
    }
  
    // Now train with everything that is available -- last training pair is (t-horizon,t)
    Vec best_model_costs; 
    for (int i=0; i<models.size(); i++)
    {
        models[i]->setTrainingSet(train_set, false);
        models[i]->train();
    
        if(i == best_model[last_call_train_t])
            best_model_costs = models[i]->errors(last_call_train_t);
    }

    if(train_stats)
    {
        Vec update = best_model_costs( common_cost_indices(best_model[last_call_train_t]) );
        train_stats->update( update );
    }

    predictions(last_call_train_t) << models[ best_model[last_call_train_t] ]->predictions(last_call_train_t);
  
    if(pb)
        pb->close();


    last_train_t = last_call_train_t;
#ifdef DEBUG
    cout << "SequentialModelSelector.last_train_t = " << last_train_t << endl;
#endif
  

    if(stepwise_save)
    {
        string s1 = append_slash(expdir) + "predictions_train_t=" + tostring(last_train_t); //"seq_model/predictions_train_t=" + tostring(last_train_t);
        saveAsciiWithoutSize(s1, predictions);

        string s2 = append_slash(expdir) + "errors_train_t=" + tostring(last_train_t);
        saveAsciiWithoutSize(s2, errors);
    }
}

Here is the call graph for this function:


Member Data Documentation

Reimplemented from PLearn::SequentialLearner.

Definition at line 182 of file SequentialModelSelector.h.

See common_costs options for details. Row index: model. Column index: common_cost.

Definition at line 69 of file SequentialModelSelector.h.

Referenced by build_(), matlabSave(), paired_t_test(), test(), and train().

The names of costs that are common to all models and that the user wishes the model selector to keep track of. The first one is considered to be the main cost, the one from which models will be compared to choose the best model.

Definition at line 112 of file SequentialModelSelector.h.

Referenced by build_(), declareOptions(), getTestCostNames(), getTrainCostNames(), and matlabSave().

From the common_costs list, the first cost is the one from which models will be compared to choose the best model. But should the best model be chosen according to the

max/min +/- 1: Mean +/- 2: Mean / Variance +/- 3: more to come.

of the cost realizations. Default: 1.

Definition at line 126 of file SequentialModelSelector.h.

Referenced by declareOptions(), sequenceCost(), and train().

If positive, the comparison performed on the basis of common_cost[0] will be applyed only the comparison_window last elements of the cost sequence. Default: -1. (No window)

Definition at line 133 of file SequentialModelSelector.h.

Referenced by declareOptions(), and sequenceCost().

If the user desires to provide a name for each model instead of model_i.

Definition at line 105 of file SequentialModelSelector.h.

Referenced by checkModelNames(), declareOptions(), getTestCostNames(), matlabSave(), and setExperimentDirectory().

If true, the model selector will report as costs the paired T tests on common_cost_indices[0] for models[0] against each other model. The model selector will only report T tests once, at t=(max_seq_len-1)

Default: false.

Definition at line 80 of file SequentialModelSelector.h.

Referenced by declareOptions(), getTestCostNames(), and test().

Definition at line 71 of file SequentialModelSelector.h.

Referenced by build_(), and train().

Does the model selector hass to save errors at each step. Default: true.

Definition at line 99 of file SequentialModelSelector.h.

Referenced by declareOptions(), test(), and train().


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines