PLearn 0.1
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Static Protected Member Functions | Protected Attributes | Private Types | Private Member Functions
PLearn::VPLProcessor Class Reference

#include <VPLProcessor.h>

Inheritance diagram for PLearn::VPLProcessor:
Inheritance graph
[legend]
Collaboration diagram for PLearn::VPLProcessor:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 VPLProcessor ()
 Default constructor.
virtual void build ()
 Simply calls inherited::build() then build_().
virtual void makeDeepCopyFromShallowCopy (CopiesMap &copies)
 Transforms a shallow copy into a deep copy.
virtual string classname () const
virtual OptionListgetOptionList () const
virtual OptionMapgetOptionMap () const
virtual RemoteMethodMapgetRemoteMethodMap () const
virtual VPLProcessordeepCopy (CopiesMap &copies) const
virtual int outputsize () const
 SUBCLASS WRITING: override this so that it returns the size of this learner's output, as a function of its inputsize(), targetsize() and set options.
virtual void forget ()
 Forwarded to inner learner.
virtual void setTrainingSet (VMat training_set, bool call_forget=true)
virtual void computeOutput (const Vec &input, Vec &output) const
 *** SUBCLASS WRITING: ***
virtual void computeOutputAndCosts (const Vec &input, const Vec &target, Vec &output, Vec &costs) const
 Default calls computeOutput and computeCostsFromOutputs.
virtual void computeCostsFromOutputs (const Vec &input, const Vec &output, const Vec &target, Vec &costs) const
 *** SUBCLASS WRITING: ***
virtual TVec< string > getOutputNames () const
 If there's an output_prg, it returns output_prg_fieldnames If there's no output_prg, the call is forwarded to the inner learner.
virtual VMat processDataSet (VMat dataset) const
 Process a full dataset (possibly containing input,target,weight,extra parts).
virtual void train ()
 *** SUBCLASS WRITING: ***
virtual TVec< std::string > getTestCostNames () const
 *** SUBCLASS WRITING: ***
virtual TVec< string > getTrainCostNames () const
 *** SUBCLASS WRITING: ***

Static Public Member Functions

static string _classname_ ()
static OptionList_getOptionList_ ()
static RemoteMethodMap_getRemoteMethodMap_ ()
static Object_new_instance_for_typemap_ ()
static bool _isa_ (const Object *o)
static void _static_initialize_ ()
static const PPathdeclaringFile ()

Public Attributes

string filtering_prg
string input_prg
string target_prg
string weight_prg
string extra_prg

Static Public Attributes

static StaticInitializer _static_initializer_

Static Protected Member Functions

static void declareOptions (OptionList &ol)
 Declares this class' options.

Protected Attributes

VMatLanguage input_prg_
TVec< string > input_prg_fieldnames
Vec processed_input
TVec< string > orig_fieldnames
int orig_inputsize
int orig_targetsize
bool use_filtering_prg_for_repeat
string repeat_id_field_name
string repeat_count_field_name

Private Types

typedef PLearner inherited

Private Member Functions

void build_ ()
 This does the actual building.
void initializeInputPrograms ()

Detailed Description

Definition at line 52 of file VPLProcessor.h.


Member Typedef Documentation

Reimplemented from PLearn::PLearner.

Definition at line 57 of file VPLProcessor.h.


Constructor & Destructor Documentation

PLearn::VPLProcessor::VPLProcessor ( )

Default constructor.

Definition at line 53 of file VPLProcessor.cc.


Member Function Documentation

string PLearn::VPLProcessor::_classname_ ( ) [static]

Reimplemented from PLearn::PLearner.

Definition at line 66 of file VPLProcessor.cc.

OptionList & PLearn::VPLProcessor::_getOptionList_ ( ) [static]

Reimplemented from PLearn::PLearner.

Definition at line 66 of file VPLProcessor.cc.

RemoteMethodMap & PLearn::VPLProcessor::_getRemoteMethodMap_ ( ) [static]

Reimplemented from PLearn::PLearner.

Definition at line 66 of file VPLProcessor.cc.

bool PLearn::VPLProcessor::_isa_ ( const Object o) [static]

Reimplemented from PLearn::PLearner.

Definition at line 66 of file VPLProcessor.cc.

Object * PLearn::VPLProcessor::_new_instance_for_typemap_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 66 of file VPLProcessor.cc.

StaticInitializer VPLProcessor::_static_initializer_ & PLearn::VPLProcessor::_static_initialize_ ( ) [static]

Reimplemented from PLearn::PLearner.

Definition at line 66 of file VPLProcessor.cc.

void PLearn::VPLProcessor::build ( ) [virtual]

Simply calls inherited::build() then build_().

Reimplemented from PLearn::PLearner.

Definition at line 132 of file VPLProcessor.cc.

References PLearn::PLearner::build(), and build_().

Here is the call graph for this function:

void PLearn::VPLProcessor::build_ ( ) [private]

This does the actual building.

Reimplemented from PLearn::PLearner.

Definition at line 125 of file VPLProcessor.cc.

References initializeInputPrograms(), PLearn::PP< T >::isNull(), orig_inputsize, orig_targetsize, and PLearn::PLearner::train_set.

Referenced by build().

{
    // We're probably reloading a saved VPLProcessor
    if(train_set.isNull() && (orig_inputsize>0 || orig_targetsize>0) )
        initializeInputPrograms();
}

Here is the call graph for this function:

Here is the caller graph for this function:

string PLearn::VPLProcessor::classname ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 66 of file VPLProcessor.cc.

void PLearn::VPLProcessor::computeCostsFromOutputs ( const Vec input,
const Vec output,
const Vec target,
Vec costs 
) const [virtual]

*** SUBCLASS WRITING: ***

This should be defined in subclasses to compute the weighted costs from already computed output. The costs should correspond to the cost names returned by getTestCostNames().

NOTE: In exotic cases, the cost may also depend on some info in the input, that's why the method also gets so see it.

Implements PLearn::PLearner.

Definition at line 276 of file VPLProcessor.cc.

References computeOutputAndCosts().

{ 
    Vec nonconst_output = output; // to make the constipated compiler happy
    computeOutputAndCosts(input, target, nonconst_output, costs); 
}

Here is the call graph for this function:

void PLearn::VPLProcessor::computeOutput ( const Vec input,
Vec output 
) const [virtual]

*** SUBCLASS WRITING: ***

This should be defined in subclasses to compute the output from the input.

Reimplemented from PLearn::PLearner.

Definition at line 252 of file VPLProcessor.cc.

References input_prg, input_prg_, input_prg_fieldnames, PLearn::TVec< T >::length(), outputsize(), processed_input, PLearn::TVec< T >::resize(), and PLearn::VMatLanguage::run().

Referenced by computeOutputAndCosts().

{
    output.resize(outputsize());
    Vec newinput = input;
    if (!input_prg.empty())
    {
        processed_input.resize(input_prg_fieldnames.length());
        input_prg_.run(input, processed_input);
        newinput= processed_input;
    }

    output << newinput;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::VPLProcessor::computeOutputAndCosts ( const Vec input,
const Vec target,
Vec output,
Vec costs 
) const [virtual]

Default calls computeOutput and computeCostsFromOutputs.

You may override this if you have a more efficient way to compute both output and weighted costs at the same time.

Reimplemented from PLearn::PLearner.

Definition at line 266 of file VPLProcessor.cc.

References computeOutput(), PLearn::TVec< T >::fill(), PLearn::PLearner::nTestCosts(), outputsize(), and PLearn::TVec< T >::resize().

Referenced by computeCostsFromOutputs().

{ 
    output.resize(outputsize());
    costs.resize(nTestCosts());

    costs.fill(-1);
    return computeOutput(input, output);
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::VPLProcessor::declareOptions ( OptionList ol) [static, protected]

Declares this class' options.

Reimplemented from PLearn::PLearner.

Definition at line 68 of file VPLProcessor.cc.

References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::PLearner::declareOptions(), extra_prg, filtering_prg, input_prg, PLearn::OptionBase::learntoption, orig_fieldnames, orig_inputsize, orig_targetsize, repeat_count_field_name, repeat_id_field_name, target_prg, use_filtering_prg_for_repeat, and weight_prg.

{
    declareOption(ol, "filtering_prg", &VPLProcessor::filtering_prg, OptionBase::buildoption,
                  "Optional program string in VPL language to apply as filtering on the training VMat.\n"
                  "This program is to produce a single value interpreted as a boolean: only the rows for which\n"
                  "it evaluates to non-zero will be kept.\n"
                  "An empty string means NO FILTERING.");

    declareOption(ol, "input_prg", &VPLProcessor::input_prg, OptionBase::buildoption,
                  "Program string in VPL language to be applied to each raw input \n"
                  "to generate the new preprocessed input.\n"
                  "Note that names must be given to the generated values with :fieldname VPL syntax.\n"
                  "An empty string means NO PREPROCESSING. (initial raw input is used as is)");

    declareOption(ol, "target_prg", &VPLProcessor::target_prg, OptionBase::buildoption,
                  "Program string in VPL language to be applied to a dataset row\n"
                  "to generate a proper target.\n"
                  "Note that names must be given to the generated values with :fieldname VPL syntax.\n"
                  "If it's an empty string, then we'll use the original target from the data set");
  
    declareOption(ol, "weight_prg", &VPLProcessor::weight_prg, OptionBase::buildoption,
                  "Program string in VPL language to be applied to a dataset row\n"
                  "to generate a proper weight.\n"
                  "Note that names must be given to the generated values with :fieldname VPL syntax.\n"
                  "If it's an empty string, then we'll use the original weight from the data set");

    declareOption(ol, "extra_prg", &VPLProcessor::extra_prg, OptionBase::buildoption,
                  "Program string in VPL language to be applied to a dataset row\n"
                  "to generate proper extra fields.\n"
                  "Note that names must be given to the generated values with :fieldname VPL syntax.\n"
                  "If it's an empty string, then we'll use the original extra fields from the data set");


    declareOption(ol, "use_filtering_prg_for_repeat", &VPLProcessor::use_filtering_prg_for_repeat, OptionBase::buildoption,
                  "When true, the result of the filtering program indicates the number of times a row should be repeated (0..n).\n"
                  "(sets FilteredVMatrix::allow_repeat_rows.)");

    declareOption(ol, "repeat_id_field_name", &VPLProcessor::repeat_id_field_name, OptionBase::buildoption,
                  "Field name for the repetition id (0, 1, ..., n-1).  No field is added if empty.");

    declareOption(ol, "repeat_count_field_name", &VPLProcessor::repeat_count_field_name, OptionBase::buildoption,
                  "Field name for the number of repetitions (n).  No field is added if empty.");

    // learnt

    declareOption(ol, "orig_fieldnames", &VPLProcessor::orig_fieldnames, OptionBase::learntoption,
                  "original fieldnames of the training set");
    declareOption(ol, "orig_inputsize", &VPLProcessor::orig_inputsize, OptionBase::learntoption,
                  "original inputsize of the training set");
    declareOption(ol, "orig_targetsize", &VPLProcessor::orig_targetsize, OptionBase::learntoption,
                  "original targetsize of the training set");


    // Now call the parent class' declareOptions
    inherited::declareOptions(ol);
}

Here is the call graph for this function:

static const PPath& PLearn::VPLProcessor::declaringFile ( ) [inline, static]

Reimplemented from PLearn::PLearner.

Definition at line 136 of file VPLProcessor.h.

VPLProcessor * PLearn::VPLProcessor::deepCopy ( CopiesMap copies) const [virtual]

Reimplemented from PLearn::PLearner.

Definition at line 66 of file VPLProcessor.cc.

void PLearn::VPLProcessor::forget ( ) [virtual]

Forwarded to inner learner.

Reimplemented from PLearn::PLearner.

Definition at line 159 of file VPLProcessor.cc.

References PLearn::PLearner::forget(), and PLearn::PLearner::stage.

Here is the call graph for this function:

OptionList & PLearn::VPLProcessor::getOptionList ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 66 of file VPLProcessor.cc.

OptionMap & PLearn::VPLProcessor::getOptionMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 66 of file VPLProcessor.cc.

TVec< string > PLearn::VPLProcessor::getOutputNames ( ) const [virtual]

If there's an output_prg, it returns output_prg_fieldnames If there's no output_prg, the call is forwarded to the inner learner.

Todo:
What is that outpuy_prg_ doing commented out? Is the fieldnames for the input_prg really the right thing to return for *output* names?

Reimplemented from PLearn::PLearner.

Definition at line 285 of file VPLProcessor.cc.

References PLearn::PLearner::getTrainingSet(), input_prg, input_prg_fieldnames, and PLERROR.

{
    if (!input_prg.empty())//output_prg_)
        return input_prg_fieldnames;

    VMat trainset= getTrainingSet();
    if (trainset==0)
        PLERROR("in VPLProcessor::getOutputNames: no train set specified yet.");

    return trainset->inputFieldNames();
}

Here is the call graph for this function:

RemoteMethodMap & PLearn::VPLProcessor::getRemoteMethodMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 66 of file VPLProcessor.cc.

TVec< std::string > PLearn::VPLProcessor::getTestCostNames ( ) const [virtual]

*** SUBCLASS WRITING: ***

This should return the names of the costs computed by computeCostsFromOutputs.

Implements PLearn::PLearner.

Definition at line 301 of file VPLProcessor.cc.

{return TVec<string>(0);}
TVec< string > PLearn::VPLProcessor::getTrainCostNames ( ) const [virtual]

*** SUBCLASS WRITING: ***

This should return the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats.

Implements PLearn::PLearner.

Definition at line 304 of file VPLProcessor.cc.

{return TVec<string>(0);}
void PLearn::VPLProcessor::initializeInputPrograms ( ) [private]
Todo:
Why is this called initializeInputPrograms() and not initializePrograms()? Why is there no initialization done for fitlering_prg, target_prg, weight_prg and extra_prg?

Definition at line 168 of file VPLProcessor.cc.

References PLearn::VMatLanguage::clear(), PLearn::VMatLanguage::compileString(), input_prg, input_prg_, input_prg_fieldnames, orig_fieldnames, orig_inputsize, PLearn::TVec< T >::resize(), PLearn::VMatLanguage::setSourceFieldNames(), and PLearn::TVec< T >::subVec().

Referenced by build_(), and setTrainingSet().

{
    if (!input_prg.empty())
    {
        input_prg_.setSourceFieldNames(orig_fieldnames.subVec(0,orig_inputsize));
        input_prg_.compileString(input_prg, input_prg_fieldnames);
    }
    else
    {
        input_prg_.clear();
        input_prg_fieldnames.resize(0);
    }

}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::VPLProcessor::makeDeepCopyFromShallowCopy ( CopiesMap copies) [virtual]

Transforms a shallow copy into a deep copy.

Todo:
Why is only input_prg deep-copied?

Reimplemented from PLearn::PLearner.

Definition at line 140 of file VPLProcessor.cc.

References PLearn::deepCopyField(), input_prg_, input_prg_fieldnames, PLearn::VMatLanguage::makeDeepCopyFromShallowCopy(), PLearn::PLearner::makeDeepCopyFromShallowCopy(), orig_fieldnames, and processed_input.

Here is the call graph for this function:

int PLearn::VPLProcessor::outputsize ( ) const [virtual]

SUBCLASS WRITING: override this so that it returns the size of this learner's output, as a function of its inputsize(), targetsize() and set options.

Implements PLearn::PLearner.

Definition at line 151 of file VPLProcessor.cc.

References input_prg, input_prg_fieldnames, PLearn::PLearner::inputsize(), and PLearn::TVec< T >::length().

Referenced by computeOutput(), and computeOutputAndCosts().

{
    if (!input_prg.empty())
        return input_prg_fieldnames.length();

    return inputsize();
}

Here is the call graph for this function:

Here is the caller graph for this function:

VMat PLearn::VPLProcessor::processDataSet ( VMat  dataset) const [virtual]

Process a full dataset (possibly containing input,target,weight,extra parts).

Returns processed view of that dataset. The default version uses computeOutput to process the input part, and simply passes on the other parts unchanged.

Reimplemented from PLearn::PLearner.

Definition at line 204 of file VPLProcessor.cc.

References extra_prg, filtering_prg, PLearn::PLearner::getExperimentDirectory(), input_prg, repeat_count_field_name, repeat_id_field_name, target_prg, PLearn::tostring(), use_filtering_prg_for_repeat, PLearn::PLearner::verbosity, and weight_prg.

{
    VMat filtered_dataset = dataset;
    PPath filtered_dataset_metadatadir = getExperimentDirectory() / "filtered_dataset.metadata";
    if (!filtering_prg.empty())
        filtered_dataset = new FilteredVMatrix(dataset, filtering_prg, filtered_dataset_metadatadir, verbosity>1,
                                               use_filtering_prg_for_repeat, repeat_id_field_name, repeat_count_field_name);

    // Since ProcessingVMatrix produces 0 length vectors when given an empty
    // program (which is not the behavior that VPLProcessors is documented as
    // implementing), we need to replace each program that is an empty string
    // by a small VPL snippet that copies all the fields for the input or
    // target, etc.

    // First compute the start of each section (input, target, etc.) in the
    // columns of the dataset.
    const int start_of_targets = dataset->inputsize();
    const int start_of_weights = start_of_targets + dataset->targetsize();
    const int start_of_extras = start_of_weights + dataset->weightsize();

    // Now compute each processing_*_prg program.
    string processing_input_prg = input_prg;
    if (processing_input_prg.empty() && dataset->inputsize() > 0) {
        processing_input_prg = "[%0:%" + tostring(start_of_targets-1) + "]";
    }
    
    string processing_target_prg = target_prg;
    if (processing_target_prg.empty() && dataset->targetsize() > 0) {
        processing_target_prg = "[%" + tostring(start_of_targets) + ":%" +
            tostring(start_of_weights-1) + "]";
    }

    string processing_weight_prg = weight_prg;
    if (processing_weight_prg.empty() && dataset->weightsize() > 0) {
        processing_weight_prg = "[%" + tostring(start_of_weights) + ":%" +
            tostring(start_of_extras-1) + "]";
    }

    string processing_extras_prg = extra_prg;
    if (processing_extras_prg.empty() && dataset->extrasize() > 0) {
        processing_extras_prg = "[%" + tostring(start_of_extras) + ":END]";
    }
    
    return new ProcessingVMatrix(filtered_dataset, processing_input_prg,
                                 processing_target_prg, processing_weight_prg,
                                 processing_extras_prg);
}

Here is the call graph for this function:

void PLearn::VPLProcessor::setTrainingSet ( VMat  training_set,
bool  call_forget = true 
) [virtual]
Todo:
Check if this method works according to the documentation.

For example, the next to last line (VMatr processed_trainset = new ...) has no impact.

Reimplemented from PLearn::PLearner.

Definition at line 186 of file VPLProcessor.cc.

References extra_prg, filtering_prg, PLearn::PLearner::getExperimentDirectory(), initializeInputPrograms(), input_prg, orig_fieldnames, orig_inputsize, orig_targetsize, repeat_count_field_name, repeat_id_field_name, PLearn::PLearner::setTrainingSet(), target_prg, use_filtering_prg_for_repeat, PLearn::PLearner::verbosity, and weight_prg.

{
    orig_fieldnames = training_set->fieldNames();
    orig_inputsize  = training_set->inputsize();
    orig_targetsize  = training_set->targetsize();
    initializeInputPrograms();

    VMat filtered_trainset = training_set;
    PPath filtered_trainset_metadatadir = getExperimentDirectory() / "filtered_train_set.metadata";
    if (!filtering_prg.empty())
        filtered_trainset = new FilteredVMatrix(training_set, filtering_prg, filtered_trainset_metadatadir, verbosity>1,
                                                use_filtering_prg_for_repeat, repeat_id_field_name, repeat_count_field_name);

    // XXX The next line does nothing!
    VMat processed_trainset = new ProcessingVMatrix(filtered_trainset, input_prg, target_prg, weight_prg, extra_prg);
    inherited::setTrainingSet(training_set, call_forget); // will call forget if needed
}

Here is the call graph for this function:

void PLearn::VPLProcessor::train ( ) [virtual]

*** SUBCLASS WRITING: ***

The role of the train method is to bring the learner up to stage==nstages, updating the stats with training costs measured on-line in the process.

TYPICAL CODE:

  static Vec input;  // static so we don't reallocate/deallocate memory each time...
  static Vec target; // (but be careful that static means shared!)
  input.resize(inputsize());    // the train_set's inputsize()
  target.resize(targetsize());  // the train_set's targetsize()
  real weight;
  
  if(!train_stats)   // make a default stats collector, in case there's none
      train_stats = new VecStatsCollector();
  
  if(nstages<stage)  // asking to revert to a previous stage!
      forget();      // reset the learner to stage=0
  
  while(stage<nstages)
  {
      // clear statistics of previous epoch
      train_stats->forget(); 
            
      //... train for 1 stage, and update train_stats,
      // using train_set->getSample(input, target, weight);
      // and train_stats->update(train_costs)
          
      ++stage;
      train_stats->finalize(); // finalize statistics for this epoch
  }

Implements PLearn::PLearner.

Definition at line 298 of file VPLProcessor.cc.

{}

Member Data Documentation

Reimplemented from PLearn::PLearner.

Definition at line 136 of file VPLProcessor.h.

Definition at line 91 of file VPLProcessor.h.

Referenced by declareOptions(), processDataSet(), and setTrainingSet().

Definition at line 86 of file VPLProcessor.h.

Referenced by declareOptions(), processDataSet(), and setTrainingSet().

Definition at line 72 of file VPLProcessor.h.

Referenced by build_(), declareOptions(), initializeInputPrograms(), and setTrainingSet().

Definition at line 73 of file VPLProcessor.h.

Referenced by build_(), declareOptions(), and setTrainingSet().

Definition at line 65 of file VPLProcessor.h.

Referenced by computeOutput(), and makeDeepCopyFromShallowCopy().

Definition at line 77 of file VPLProcessor.h.

Referenced by declareOptions(), processDataSet(), and setTrainingSet().

Definition at line 76 of file VPLProcessor.h.

Referenced by declareOptions(), processDataSet(), and setTrainingSet().

Definition at line 89 of file VPLProcessor.h.

Referenced by declareOptions(), processDataSet(), and setTrainingSet().

Definition at line 75 of file VPLProcessor.h.

Referenced by declareOptions(), processDataSet(), and setTrainingSet().

Definition at line 90 of file VPLProcessor.h.

Referenced by declareOptions(), processDataSet(), and setTrainingSet().


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines