PLearn 0.1
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Private Types | Private Member Functions | Static Private Member Functions | Private Attributes
PLearn::MeanMedianModeImputationVMatrix Class Reference

#include <MeanMedianModeImputationVMatrix.h>

Inheritance diagram for PLearn::MeanMedianModeImputationVMatrix:
Inheritance graph
[legend]
Collaboration diagram for PLearn::MeanMedianModeImputationVMatrix:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 MeanMedianModeImputationVMatrix ()
virtual ~MeanMedianModeImputationVMatrix ()
virtual void build ()
 Simply calls inherited::build() then build_().
virtual void makeDeepCopyFromShallowCopy (CopiesMap &copies)
 Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be.
virtual real get (int i, int j) const
 This method must be implemented in all subclasses.
virtual void getSubRow (int i, int j, Vec v) const
 It is suggested that this method be implemented in subclasses to speed up accesses (default version repeatedly calls get(i,j) which may have a significant overhead).
virtual void getRow (int i, Vec v) const
 These methods do not usually need to be overridden in subclasses (default versions call getSubRow, which should do just fine)
virtual void getColumn (int i, Vec v) const
 Copies column i into v (which must have appropriate length equal to the VMat's length).
VMat getMeanMedianModeFile ()
virtual string classname () const
virtual OptionListgetOptionList () const
virtual OptionMapgetOptionMap () const
virtual RemoteMethodMapgetRemoteMethodMap () const
virtual
MeanMedianModeImputationVMatrix
deepCopy (CopiesMap &copies) const

Static Public Member Functions

static void declareOptions (OptionList &ol)
 Declares this class' options.
static string _classname_ ()
 MeanMedianModeImputationVMatrix.
static OptionList_getOptionList_ ()
static RemoteMethodMap_getRemoteMethodMap_ ()
static Object_new_instance_for_typemap_ ()
static bool _isa_ (const Object *o)
static void _static_initialize_ ()
static const PPathdeclaringFile ()

Public Attributes

VMat train_set
 A referenced train set.
real number_of_train_samples_to_use
 The number of samples from the train set that will be examined to compute the required statistic for each variable.
TVec< pair< string, string > > imputation_spec
 Pairs of instruction of the form field_name : mean | median | mode | none.
bool missing_field_error
 if true will generate an error if the imputation_spec reference a field not in the source.
string default_instruction

Static Public Attributes

static StaticInitializer _static_initializer_

Private Types

typedef ImputationVMatrix inherited

Private Member Functions

void build_ ()
 This does the actual building.
virtual void setMetaDataDir (const PPath &the_metadatadir)
 This should be called by the build method of every VMatrix that has a metadatadir.
void createMeanMedianModeFile (PPath file_name)
void loadMeanMedianModeFile (PPath file_name)
void computeMeanMedianModeVectors ()

Static Private Member Functions

static void sortColumn (Vec input_vec, int start, int end)
static void sortSmallSubArray (Vec input_vec, int start_index, int end_index)
static void swapValues (Vec input_vec, int index_i, int index_j)
static real compare (real field1, real field2)

Private Attributes

TVec< string > train_field_names
VMat mean_median_mode_file
Vec variable_mean
 The vector of variable means observed from the train set.
Vec variable_median
 The vector of variable medians observed from the train set.
Vec variable_mode
 The vector of variable modes observed from the train set.
TVec< intvariable_imputation_instruction
 The vector of coded instruction for each variables.

Detailed Description

Definition at line 55 of file MeanMedianModeImputationVMatrix.h.


Member Typedef Documentation

Reimplemented from PLearn::ImputationVMatrix.

Definition at line 57 of file MeanMedianModeImputationVMatrix.h.


Constructor & Destructor Documentation

PLearn::MeanMedianModeImputationVMatrix::MeanMedianModeImputationVMatrix ( )
PLearn::MeanMedianModeImputationVMatrix::~MeanMedianModeImputationVMatrix ( ) [virtual]

Definition at line 64 of file MeanMedianModeImputationVMatrix.cc.

{
}

Member Function Documentation

string PLearn::MeanMedianModeImputationVMatrix::_classname_ ( ) [static]
OptionList & PLearn::MeanMedianModeImputationVMatrix::_getOptionList_ ( ) [static]

Reimplemented from PLearn::ImputationVMatrix.

Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.

RemoteMethodMap & PLearn::MeanMedianModeImputationVMatrix::_getRemoteMethodMap_ ( ) [static]

Reimplemented from PLearn::ImputationVMatrix.

Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.

bool PLearn::MeanMedianModeImputationVMatrix::_isa_ ( const Object o) [static]

Reimplemented from PLearn::ImputationVMatrix.

Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.

Object * PLearn::MeanMedianModeImputationVMatrix::_new_instance_for_typemap_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.

StaticInitializer MeanMedianModeImputationVMatrix::_static_initializer_ & PLearn::MeanMedianModeImputationVMatrix::_static_initialize_ ( ) [static]

Reimplemented from PLearn::ImputationVMatrix.

Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.

void PLearn::MeanMedianModeImputationVMatrix::build ( ) [virtual]

Simply calls inherited::build() then build_().

Reimplemented from PLearn::ImputationVMatrix.

Definition at line 142 of file MeanMedianModeImputationVMatrix.cc.

References PLearn::ImputationVMatrix::build(), and build_().

Referenced by PLearn::Preprocessing::manageTrainTestUnknownSets().

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::MeanMedianModeImputationVMatrix::build_ ( ) [private]

This does the actual building.

Reimplemented from PLearn::ImputationVMatrix.

Definition at line 249 of file MeanMedianModeImputationVMatrix.cc.

References PLearn::TVec< T >::append(), PLearn::TVec< T >::clear(), PLearn::TVec< T >::copy(), PLearn::VMatrix::declareFieldNames(), default_instruction, PLearn::TVec< T >::fill(), PLearn::TVec< T >::first(), i, imputation_spec, PLearn::TVec< T >::length(), PLearn::VMat::length(), missing_field_error, n, number_of_train_samples_to_use, PLERROR, PLWARN_ERR, PLearn::TVec< T >::resize(), PLearn::VMatrix::setMetaInfoFrom(), PLearn::TVec< T >::size(), PLearn::ImputationVMatrix::source, PLearn::string_begins_with(), PLearn::tostring(), train_field_names, train_set, PLearn::VMatrix::updateMtime(), variable_imputation_instruction, variable_mean, variable_median, variable_mode, and PLearn::VMat::width().

Referenced by build().

{
    if (!source) PLERROR("In MeanMedianModeImputationVMatrix:: source vmat must be supplied");
    if (!train_set)
      train_set = source;

    updateMtime(train_set);
    updateMtime(source);

    int train_length = train_set->length();
    int train_width = train_set->width();

    if (number_of_train_samples_to_use > 0.0){
        if (number_of_train_samples_to_use < 1.0) train_length = (int) (number_of_train_samples_to_use * (real) train_length);
        else train_length = (int) number_of_train_samples_to_use;
    }
    if (train_length > train_set->length()) train_length = train_set->length();
    if(train_length < 1) 
      PLERROR("In MeanMedianModeImputationVMatrix::length of the number of"
              " train samples to use must be at least 1, got: %i", train_length);

    if(train_set->inputsize() < 1) 
      PLERROR("In MeanMedianModeImputationVMatrix::inputsize of the train vmat"
              " must be supplied, got : %i", train_set->inputsize());
    source->compatibleSizeError(train_set);
    setMetaInfoFrom(source);

    train_field_names.resize(train_width);
    train_field_names = train_set->fieldNames();

    declareFieldNames(train_field_names);
    variable_mean.resize(train_width);
    variable_median.resize(train_width);
    variable_mode.resize(train_width);
    variable_imputation_instruction.resize(train_width);

    if(default_instruction.empty()) variable_imputation_instruction.clear();
    else if (default_instruction == "mean") variable_imputation_instruction.fill(1);
    else if (default_instruction == "median") variable_imputation_instruction.fill(2);
    else if (default_instruction == "mode") variable_imputation_instruction.fill(3);
    else if (default_instruction == "none") variable_imputation_instruction.fill(4);
    else if (default_instruction == "err") variable_imputation_instruction.fill(5);
    else
      PLERROR("In MeanMedianModeImputationVMatrix: unsupported default_imputation instruction: %s ",
              default_instruction.c_str());


    TVec<string> nofields;
    
    //We sho
    TVec<pair<string,string> > save_imputation_spec = imputation_spec;
    imputation_spec = save_imputation_spec.copy();
    TVec<string> not_expanded;
    for (int spec_col = 0; spec_col < imputation_spec.size(); spec_col++)
    {
        int train_col;
        string fname = imputation_spec[spec_col].first;
        for (train_col = 0; train_col < train_width; train_col++)
        {
          if (fname == train_field_names[train_col]) break;
        }
        char last_char = fname[fname.size()-1];
        if (train_col >= train_width && last_char!='*'){
          nofields.append(fname.c_str());
          continue;
        }
        else if(train_col >= train_width && last_char=='*')
        {
          bool expended = false;
          fname.resize(fname.size()-1);//remove the last caracter (*)
          for(train_col = 0; train_col < train_width; train_col++)
          {
            if(string_begins_with(train_field_names[train_col],fname))
            {
              pair<string,string> n=make_pair(train_field_names[train_col],
                                              imputation_spec[spec_col].second);
//                    perr<<"expanding "<<train_field_names[train_col] << " to " << n <<endl;
              
              imputation_spec.append(n);
              expended = true;
            }
          }
          if(!expended){
            not_expanded.append(imputation_spec[spec_col].first);
          }
          continue;
        }
        if (imputation_spec[spec_col].second == "mean") variable_imputation_instruction[train_col] = 1;
        else if (imputation_spec[spec_col].second == "median") variable_imputation_instruction[train_col] = 2;
        else if (imputation_spec[spec_col].second == "mode") variable_imputation_instruction[train_col] = 3;
        else if (imputation_spec[spec_col].second == "none") variable_imputation_instruction[train_col] = 4;
        else if (imputation_spec[spec_col].second == "err") variable_imputation_instruction[train_col] = 5;
        else
          PLERROR("In MeanMedianModeImputationVMatrix: unsupported imputation instruction: %s : %s",
                     (imputation_spec[spec_col].first).c_str(), (imputation_spec[spec_col].second).c_str());
    }
    if(not_expanded.length()>0){
      PLWARN_ERR(!missing_field_error,
                 "In MeanMedianModeImputationVMatrix::build_() - "
                 "For %d spec, we didn't found partial match '%s'",
                 not_expanded.length(), tostring(not_expanded).c_str());
    }
        

    imputation_spec = save_imputation_spec;

    if(nofields.length()>0)
      PLWARN_ERR(!missing_field_error,
                 "In MeanMedianModeImputationVMatrix::build_() Their is %d"
                 " fields in the imputation_spec that are not in train set:"
                 " '%s'",nofields.length(),
                 tostring(nofields).c_str());

    TVec<string> no_instruction;
    for(int i = 0;i<variable_imputation_instruction.size();i++)
      if(variable_imputation_instruction[i]==0)
        no_instruction.append(train_field_names[i]);
    if(no_instruction.size()>0)
      PLERROR("In MeanMedianModeImputationVMatrix::build_() In the source"
              " VMatrix their is %d field(s) that do not have instruction: '%s'.",
              no_instruction.size(),tostring(no_instruction).c_str());

}

Here is the call graph for this function:

Here is the caller graph for this function:

string PLearn::MeanMedianModeImputationVMatrix::classname ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.

real PLearn::MeanMedianModeImputationVMatrix::compare ( real  field1,
real  field2 
) [static, private]

Definition at line 590 of file MeanMedianModeImputationVMatrix.cc.

References PLearn::is_missing().

Referenced by sortColumn(), and sortSmallSubArray().

{
  if (is_missing(field1) && is_missing(field2)) return 0.0;
  if (is_missing(field1)) return +1.0;
  if (is_missing(field2)) return -1.0;
  return field1 - field2;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::MeanMedianModeImputationVMatrix::computeMeanMedianModeVectors ( ) [private]

Definition at line 437 of file MeanMedianModeImputationVMatrix.cc.

References PLearn::TVec< T >::clear(), PLearn::VMat::getColumn(), PLearn::is_missing(), PLearn::VMat::length(), sortColumn(), train_set, PLearn::ProgressBar::update(), variable_mean, variable_median, variable_mode, and PLearn::VMatrix::width_.

Referenced by setMetaDataDir().

{
    TVec<int> variable_present_count(width_);
    TVec<int> variable_mode_count(width_);
    variable_mean.clear();
    variable_median.clear();
    variable_mode.clear();
    variable_present_count.clear();
    variable_mode_count.clear();
    Vec variable_vec(train_set->length());
    cout << fixed << showpoint;
    ProgressBar* pb = 0;
    pb = new ProgressBar("Computing the mean, median and mode vectors", width_);
    VMat train_set_mem = new MemoryVMatrix(train_set);
    for (int train_col = 0; train_col < width_; train_col++)
    {
        real current_value = 0.0;
        int current_value_count = 0;
        train_set_mem->getColumn(train_col, variable_vec);
        sortColumn(variable_vec, 0, train_set_mem->length());
        for (int train_row = 0; train_row < train_set_mem->length(); train_row++)
        {
            if (is_missing(variable_vec[train_row]))
              continue;
            variable_mean[train_col] += variable_vec[train_row];
            variable_present_count[train_col] += 1;
            if (variable_vec[train_row] != current_value)
            {
                if (current_value_count > variable_mode_count[train_col])
                {
                    variable_mode[train_col] = current_value;
                    variable_mode_count[train_col] = current_value_count;
                }
                current_value_count = 0;
                current_value = variable_vec[train_row];
            }
            current_value_count += 1;
        }
        if (variable_present_count[train_col] > 0)
        {
            variable_mean[train_col] = variable_mean[train_col] / variable_present_count[train_col];
            variable_median[train_col] = variable_vec[(variable_present_count[train_col] / 2)];
        }
        if (current_value_count > variable_mode_count[train_col])
        {
            variable_mode[train_col] = current_value;
            variable_mode_count[train_col] = current_value_count;
        }
        pb->update( train_col );
        /*
        cout << "col: "         << setw(3)  <<                     train_col
             << " present: "    << setw(5)  <<                     variable_present_count[train_col]
             << " missing: "    << setw(5)  <<                     variable_missing_count[train_col]
             << " mean: "       << setw(11) << setprecision(2) <<  variable_mean[train_col]
             << " median: "     << setw(11) << setprecision(2) <<  variable_median[train_col]
             << " mode count: " << setw(5)  <<                     variable_mode_count[train_col]
             << " mode: "       << setw(11) << setprecision(2) <<  variable_mode[train_col]
             << " name: "       <<                                 train_field_names[train_col]
             << endl;
        */
    }
    delete pb;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::MeanMedianModeImputationVMatrix::createMeanMedianModeFile ( PPath  file_name) [private]

Definition at line 407 of file MeanMedianModeImputationVMatrix.cc.

References mean_median_mode_file, train_field_names, variable_mean, variable_median, and variable_mode.

Referenced by setMetaDataDir().

{
    mean_median_mode_file = new FileVMatrix(file_name, 3, train_field_names);
    mean_median_mode_file->setMetaInfoFrom(this);
    mean_median_mode_file->putRow(0, variable_mean);
    mean_median_mode_file->putRow(1, variable_median);
    mean_median_mode_file->putRow(2, variable_mode);
}

Here is the caller graph for this function:

void PLearn::MeanMedianModeImputationVMatrix::declareOptions ( OptionList ol) [static]

Declares this class' options.

Reimplemented from PLearn::ImputationVMatrix.

Definition at line 68 of file MeanMedianModeImputationVMatrix.cc.

References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::ImputationVMatrix::declareOptions(), default_instruction, PLearn::VMatrix::extrasize_, imputation_spec, PLearn::VMatrix::inputsize_, PLearn::OptionBase::learntoption, PLearn::VMatrix::length_, missing_field_error, PLearn::OptionBase::nosave, number_of_train_samples_to_use, PLearn::redeclareOption(), PLearn::VMatrix::targetsize_, train_set, variable_imputation_instruction, variable_mean, variable_median, variable_mode, PLearn::VMatrix::weightsize_, and PLearn::VMatrix::width_.

{
  declareOption(ol, "train_set", &MeanMedianModeImputationVMatrix::train_set, OptionBase::buildoption, 
                "A referenced train set.\n"
                "The mean, median or mode is computed with the observed values in this data set.\n"
                "It is used in combination with the option number_of_train_samples_to_use\n");

  declareOption(ol, "number_of_train_samples_to_use", &MeanMedianModeImputationVMatrix::number_of_train_samples_to_use, OptionBase::buildoption, 
                "The number of samples from the train set that will be examined to compute the required statistic for each variable.\n" 
                "If equal to zero, all the samples from the train set are used to calculated the statistics.\n"
                "If it is a fraction between 0 and 1, this proportion of the samples are used.\n"
                "If greater or equal to 1, the integer portion is interpreted as the number of samples to use.");
      
  declareOption(ol, "imputation_spec", &MeanMedianModeImputationVMatrix::imputation_spec, OptionBase::buildoption, 
                "Pairs of instruction of the form field_name : mean | median | mode | none | err.\n"
                " -mean  : put the mean of the field if one value is missing\n"
                " -median: put the median of the field if one value is missing\n"
                " -mode  : put the mode(most frequent value) of the field if one value is missing\n"
                " -none  : let the missing value in this field\n"
                " -err   : make it an error to have a missing value in this field"
                );

  declareOption(ol, "missing_field_error", &MeanMedianModeImputationVMatrix::missing_field_error,
                OptionBase::buildoption, 
                "If True, will generate an error if some field in the"
                " imputation_spec are present but not in the source. Otherwise"
                " will generate a warning. This also applies for regex spec.");

  declareOption(ol, "default_instruction", &MeanMedianModeImputationVMatrix::default_instruction,
                OptionBase::buildoption, 
                "The default instruction to use. If empty(default), will generate"
                " an error is some source variable don't have an one in imputation_spec.");

  declareOption(ol, "variable_mean", &MeanMedianModeImputationVMatrix::variable_mean, OptionBase::learntoption, 
                "The vector of variable means observed from the train set.");

  declareOption(ol, "variable_median", &MeanMedianModeImputationVMatrix::variable_median, OptionBase::learntoption, 
                "The vector of variable medians observed from the train set.");

  declareOption(ol, "variable_mode", &MeanMedianModeImputationVMatrix::variable_mode, OptionBase::learntoption, 
                "The vector of variable modes observed from the train set.");

  declareOption(ol, "variable_imputation_instruction", &MeanMedianModeImputationVMatrix::variable_imputation_instruction, OptionBase::learntoption, 
                "The vector of coded instruction for each variables.");

  inherited::declareOptions(ol);

  declareOption(ol, "length", &MeanMedianModeImputationVMatrix::length_,
                OptionBase::nosave,
                "The number of example. Computed each time from source.");
  
  redeclareOption(ol, "inputsize", &MeanMedianModeImputationVMatrix::inputsize_,
                  OptionBase::nosave,
                  "Taken from source in  MeanMedianModeImputationVMatrix.");
  
  redeclareOption(ol, "targetsize",
                  &MeanMedianModeImputationVMatrix::targetsize_,
                  OptionBase::nosave,
                  "Taken from source in MeanMedianModeImputationVMatrix.");
  
  redeclareOption(ol, "weightsize",
                  &MeanMedianModeImputationVMatrix::weightsize_,
                  OptionBase::nosave,
                  "Taken from source in MeanMedianModeImputationVMatrix.");
  
  redeclareOption(ol, "extrasize", &MeanMedianModeImputationVMatrix::extrasize_,
                  OptionBase::nosave,
                  "Taken from source in MeanMedianModeImputationVMatrix.");
  
  redeclareOption(ol, "width", &MeanMedianModeImputationVMatrix::width_,
                  OptionBase::nosave,
                  "Taken from source in MeanMedianModeImputationVMatrix.");
}

Here is the call graph for this function:

static const PPath& PLearn::MeanMedianModeImputationVMatrix::declaringFile ( ) [inline, static]

Reimplemented from PLearn::ImputationVMatrix.

Definition at line 123 of file MeanMedianModeImputationVMatrix.h.

MeanMedianModeImputationVMatrix * PLearn::MeanMedianModeImputationVMatrix::deepCopy ( CopiesMap copies) const [virtual]

Reimplemented from PLearn::ImputationVMatrix.

Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.

real PLearn::MeanMedianModeImputationVMatrix::get ( int  i,
int  j 
) const [virtual]

This method must be implemented in all subclasses.

Returns element (i,j).

Implements PLearn::VMatrix.

Definition at line 160 of file MeanMedianModeImputationVMatrix.cc.

References PLearn::is_missing(), j, MISSING_VALUE, PLERROR, PLearn::ImputationVMatrix::source, variable_imputation_instruction, variable_mean, variable_median, and variable_mode.

{ 
  real variable_value = source->get(i, j);
  if (!is_missing(variable_value)) return variable_value;
  else if (variable_imputation_instruction[j] == 1) return variable_mean[j];
  else if (variable_imputation_instruction[j] == 2) return variable_median[j];
  else if (variable_imputation_instruction[j] == 3) return variable_mode[j];
  else if (variable_imputation_instruction[j] == 4) return variable_value;
  else if (variable_imputation_instruction[j] == 5)
    return MISSING_VALUE;//PLERROR("");//PLERROR("In MeanMedianModeImputationVMatrix::get(%d,%d) - the value is"
      //           " missing and have a instruction err!",i,j);
  else
    PLERROR("In MeanMedianModeImputationVMatrix::get(%d,%d) - "
            "unknow variable_imputation_instruction value of %d",i,j,
            variable_imputation_instruction[j] );
  //Should not be executed, to remove a warning
  return MISSING_VALUE;
}

Here is the call graph for this function:

void PLearn::MeanMedianModeImputationVMatrix::getColumn ( int  i,
Vec  v 
) const [virtual]

Copies column i into v (which must have appropriate length equal to the VMat's length).

Reimplemented from PLearn::VMatrix.

Definition at line 227 of file MeanMedianModeImputationVMatrix.cc.

References i, PLearn::is_missing(), PLearn::TVec< T >::length(), PLERROR, PLearn::ImputationVMatrix::source, variable_imputation_instruction, variable_mean, variable_median, and variable_mode.

{  
  source-> getColumn(i, v);
  for (int source_row = 0; source_row < v->length(); source_row++)
    if (is_missing(v[source_row])){
      if (variable_imputation_instruction[i] == 1) v[source_row] = variable_mean[i];
      else if (variable_imputation_instruction[i] == 2) v[source_row] = variable_median[i];
      else if (variable_imputation_instruction[i] == 3) v[source_row] = variable_mode[i];
      else if (variable_imputation_instruction[i] == 4)
        ;//do nothing
      else if (variable_imputation_instruction[i] == 5)
        return PLERROR("In MeanMedianModeImputationVMatrix::getColumn(%d) - the value is"
                       " missing and have a instruction err!",i);
      else
        PLERROR("In MeanMedianModeImputationVMatrix::getRow(%d, Vec) - "
                "unknow variable_imputation_instruction value of %d",i,
                variable_imputation_instruction[i] );
    }
}

Here is the call graph for this function:

VMat PLearn::MeanMedianModeImputationVMatrix::getMeanMedianModeFile ( )

Definition at line 432 of file MeanMedianModeImputationVMatrix.cc.

References mean_median_mode_file.

Referenced by PLearn::Preprocessing::manageTrainTestUnknownSets().

Here is the caller graph for this function:

OptionList & PLearn::MeanMedianModeImputationVMatrix::getOptionList ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.

OptionMap & PLearn::MeanMedianModeImputationVMatrix::getOptionMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.

RemoteMethodMap & PLearn::MeanMedianModeImputationVMatrix::getRemoteMethodMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.

void PLearn::MeanMedianModeImputationVMatrix::getRow ( int  i,
Vec  v 
) const [virtual]

These methods do not usually need to be overridden in subclasses (default versions call getSubRow, which should do just fine)

Copies row i into v (which must have appropriate length equal to the VMat's width).

Reimplemented from PLearn::VMatrix.

Definition at line 203 of file MeanMedianModeImputationVMatrix.cc.

References PLearn::VMatrix::fieldName(), PLearn::is_missing(), PLearn::TVec< T >::length(), PLERROR, PLearn::ImputationVMatrix::source, variable_imputation_instruction, variable_mean, variable_median, and variable_mode.

{  
  source-> getRow(i, v);
  for (int source_col = 0; source_col < v->length(); source_col++)
    if (is_missing(v[source_col])){
      if (variable_imputation_instruction[source_col] == 1)
        v[source_col] = variable_mean[source_col];
      else if (variable_imputation_instruction[source_col] == 2)
        v[source_col] = variable_median[source_col];
      else if (variable_imputation_instruction[source_col] == 3)
        v[source_col] = variable_mode[source_col]; 
      else if (variable_imputation_instruction[source_col] == 4)
        ;//do nothing
      else if (variable_imputation_instruction[source_col] == 5)
        return PLERROR("In MeanMedianModeImputationVMatrix::getRow(%d) -"
                       " the value is missing for column %s"
                       " and have a instruction err!",i, fieldName(source_col).c_str());
      else
        PLERROR("In MeanMedianModeImputationVMatrix::getRow(%d, Vec) - "
                "unknow variable_imputation_instruction value of %d",i,
                variable_imputation_instruction[source_col] );
    }
}

Here is the call graph for this function:

void PLearn::MeanMedianModeImputationVMatrix::getSubRow ( int  i,
int  j,
Vec  v 
) const [virtual]

It is suggested that this method be implemented in subclasses to speed up accesses (default version repeatedly calls get(i,j) which may have a significant overhead).

Fills v with the subrow i lying between columns j (inclusive) and j+v.length() (exclusive).

Reimplemented from PLearn::VMatrix.

Definition at line 179 of file MeanMedianModeImputationVMatrix.cc.

References PLearn::VMat::getSubRow(), PLearn::is_missing(), j, PLearn::TVec< T >::length(), PLERROR, PLearn::ImputationVMatrix::source, variable_imputation_instruction, variable_mean, variable_median, and variable_mode.

{  
  source->getSubRow(i, j, v);
  for (int source_col = 0; source_col < v->length(); source_col++) 
    if (is_missing(v[source_col])){
      if (variable_imputation_instruction[source_col + j] == 1)
        v[source_col] = variable_mean[source_col + j];
      else if (variable_imputation_instruction[source_col + j] == 2)
        v[source_col] = variable_median[source_col + j];
      else if (variable_imputation_instruction[source_col + j] == 3)
        v[source_col] = variable_mode[source_col + j];
      else if (variable_imputation_instruction[source_col + j] == 4)
        ;//do nothing
      else if (variable_imputation_instruction[source_col + j] == 5)
        return PLERROR("In MeanMedianModeImputationVMatrix::getSubRow(%d,%d) - the value is"
                       " missing and have a instruction err!",i,j);
      else
        PLERROR("In MeanMedianModeImputationVMatrix::getSubRow(%d,%d, Vec) - "
                "unknow variable_imputation_instruction value of %d",i,j,
                variable_imputation_instruction[source_col + j] );
    }

}

Here is the call graph for this function:

void PLearn::MeanMedianModeImputationVMatrix::loadMeanMedianModeFile ( PPath  file_name) [private]

Definition at line 416 of file MeanMedianModeImputationVMatrix.cc.

References PLearn::VMatrix::compatibleSizeError(), PLearn::VMatrix::fieldNames(), mean_median_mode_file, PLERROR, train_set, variable_mean, variable_median, and variable_mode.

Referenced by setMetaDataDir().

{
    train_set->isUpToDate(file_name,true,true);

    mean_median_mode_file = new FileVMatrix(file_name);
    compatibleSizeError(mean_median_mode_file, "Bad file "+file_name);
    if(mean_median_mode_file->fieldNames()!=fieldNames())
      PLERROR("In MeanMedianModeImputationVMatrix::loadMeanMedianModeFile(%s) -"
              " The file don't have the same field name as the source. "
              "Delete it to have it recreated it automatically.",
              file_name.c_str());
    mean_median_mode_file->getRow(0, variable_mean);
    mean_median_mode_file->getRow(1, variable_median);
    mean_median_mode_file->getRow(2, variable_mode);
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::MeanMedianModeImputationVMatrix::makeDeepCopyFromShallowCopy ( CopiesMap copies) [virtual]

Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be.

This needs to be overridden by every class that adds "complex" data members to the class, such as Vec, Mat, PP<Something>, etc. Typical implementation:

  void CLASS_OF_THIS::makeDeepCopyFromShallowCopy(CopiesMap& copies)
  {
      inherited::makeDeepCopyFromShallowCopy(copies);
      deepCopyField(complex_data_member1, copies);
      deepCopyField(complex_data_member2, copies);
      ...
  }
Parameters:
copiesA map used by the deep-copy mechanism to keep track of already-copied objects.

Reimplemented from PLearn::ImputationVMatrix.

Definition at line 148 of file MeanMedianModeImputationVMatrix.cc.

References PLearn::deepCopyField(), imputation_spec, PLearn::ImputationVMatrix::makeDeepCopyFromShallowCopy(), number_of_train_samples_to_use, train_set, variable_imputation_instruction, variable_mean, variable_median, and variable_mode.

Here is the call graph for this function:

void PLearn::MeanMedianModeImputationVMatrix::setMetaDataDir ( const PPath the_metadatadir) [private, virtual]

This should be called by the build method of every VMatrix that has a metadatadir.

It will create said directory if it doesn's already exist. Throws a PLERROR if called with an empty string.

Reimplemented from PLearn::VMatrix.

Definition at line 372 of file MeanMedianModeImputationVMatrix.cc.

References computeMeanMedianModeVectors(), createMeanMedianModeFile(), PLearn::VMatrix::getMetaDataDir(), PLearn::VMatrix::hasMetaDataDir(), PLearn::isfile(), loadMeanMedianModeFile(), PLERROR, PLearn::rm(), PLearn::VMatrix::setMetaDataDir(), PLearn::ImputationVMatrix::source, and train_set.

{
  inherited::setMetaDataDir(the_metadatadir);
  if(!train_set->hasMetaDataDir() && !hasMetaDataDir())
    PLERROR("In MeanMedianModeImputationVMatrix::setMetaDataDir() - the "
            " train_set should have a metadata dir or we should have one!");
  else if(!train_set->hasMetaDataDir())
    train_set->setMetaDataDir(getMetaDataDir()/"train_set");
  
  PPath train_metadata = train_set->getMetaDataDir();
  PPath mean_median_mode_file_name = train_metadata + "mean_median_mode_file.pmat";

  bool uptodate = train_set->isUpToDate(mean_median_mode_file_name,false)
    && source->isUpToDate(mean_median_mode_file_name,false);

  train_set->lockMetaDataDir();
  try{
    if (!uptodate)
      {
        computeMeanMedianModeVectors();
        createMeanMedianModeFile(mean_median_mode_file_name);
      }
    else loadMeanMedianModeFile(mean_median_mode_file_name);
  }catch(const PLearnError& e){
    train_set->unlockMetaDataDir();

    //we erase the file if we are creating it
    // as it can be partilly saved.
    if(!uptodate && isfile(mean_median_mode_file_name))
      rm(mean_median_mode_file_name);
    throw e;
  }
  train_set->unlockMetaDataDir();
}

Here is the call graph for this function:

void PLearn::MeanMedianModeImputationVMatrix::sortColumn ( Vec  input_vec,
int  start,
int  end 
) [static, private]

Definition at line 501 of file MeanMedianModeImputationVMatrix.cc.

References compare(), PLERROR, sortSmallSubArray(), and swapValues().

Referenced by computeMeanMedianModeVectors().

{
  int start_index = start;
  int end_index = end - 1;
  int forward_index;
  int backward_index;
  int stack_index = -1;
  real pivot_value;
  TVec<int> stack(50);
  for (;;)
  {
    if ((end_index - start_index) < 7)
    {
      if (end_index > start_index)
      {
        sortSmallSubArray(input_vec, start_index, end_index);
      }
      if (stack_index < 0)
      {
        break;
      }
      end_index = stack[stack_index--];
      start_index = stack[stack_index--];
    }
    else
    {
      swapValues(input_vec, start_index + 1, (start_index + end_index) / 2);
      if (compare(input_vec[start_index], input_vec[end_index]) > 0.0) swapValues(input_vec, start_index, end_index);
      if (compare(input_vec[start_index + 1], input_vec[end_index]) > 0.0) swapValues(input_vec, start_index + 1, end_index);
      if (compare(input_vec[start_index], input_vec[start_index + 1]) > 0.0) swapValues(input_vec, start_index, start_index + 1);
      forward_index = start_index + 1;
      backward_index = end_index;
      pivot_value = input_vec[start_index + 1];
      for (;;)
      {
        do forward_index++; while (compare(input_vec[forward_index], pivot_value) < 0.0);
        do backward_index--; while (compare(input_vec[backward_index], pivot_value) > 0.0);
        if (backward_index < forward_index)
        {
          break;
        }
        swapValues(input_vec, forward_index, backward_index);
      }
      swapValues(input_vec, start_index + 1, backward_index);
      stack_index += 2;
      if (stack_index > 50)
        PLERROR("RegressionTreeRegistersVMatrix: the stack for sorting the rows is too small");
      if ((end_index - forward_index + 1) >= (backward_index - start_index))
      {
        stack[stack_index] = end_index;
        stack[stack_index - 1] = forward_index;
        end_index = backward_index - 1;
      }
      else
      {
        stack[stack_index] = backward_index - 1;
        stack[stack_index - 1] = start_index;
        start_index = forward_index;
      }
    }
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::MeanMedianModeImputationVMatrix::sortSmallSubArray ( Vec  input_vec,
int  start_index,
int  end_index 
) [static, private]

Definition at line 564 of file MeanMedianModeImputationVMatrix.cc.

References compare().

Referenced by sortColumn().

{
  int index_i;
  int index_j;
  for (index_i = start_index + 1; index_i <= end_index; index_i++)
  {
    real saved_value = input_vec[index_i];
    for (index_j = index_i - 1; index_j >= start_index; index_j--)
    {
      if (compare(input_vec[index_j], saved_value) <= 0.0)
      {
        break;
      }
      input_vec[index_j + 1] = input_vec[index_j];
    }
    input_vec[index_j + 1] = saved_value;
  }  
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::MeanMedianModeImputationVMatrix::swapValues ( Vec  input_vec,
int  index_i,
int  index_j 
) [static, private]

Definition at line 583 of file MeanMedianModeImputationVMatrix.cc.

Referenced by sortColumn().

{
  real saved_value = input_vec[index_i];
  input_vec[index_i] = input_vec[index_j];
  input_vec[index_j] = saved_value;
}

Here is the caller graph for this function:


Member Data Documentation

Reimplemented from PLearn::ImputationVMatrix.

Definition at line 123 of file MeanMedianModeImputationVMatrix.h.

Definition at line 79 of file MeanMedianModeImputationVMatrix.h.

Referenced by build_(), and declareOptions().

Pairs of instruction of the form field_name : mean | median | mode | none.

Definition at line 74 of file MeanMedianModeImputationVMatrix.h.

Referenced by build_(), declareOptions(), makeDeepCopyFromShallowCopy(), and PLearn::Preprocessing::manageTrainTestUnknownSets().

if true will generate an error if the imputation_spec reference a field not in the source.

Otherwise generate a warning.

Definition at line 78 of file MeanMedianModeImputationVMatrix.h.

Referenced by build_(), and declareOptions().

The number of samples from the train set that will be examined to compute the required statistic for each variable.

If equal to zero, all the samples from the train set are used to calculated the statistics. If it is a fraction between 0 and 1, this proportion of the samples are used. If greater or equal to 1, the integer portion is interpreted as the number of samples to use.

Definition at line 71 of file MeanMedianModeImputationVMatrix.h.

Referenced by build_(), declareOptions(), makeDeepCopyFromShallowCopy(), and PLearn::Preprocessing::manageTrainTestUnknownSets().

Definition at line 97 of file MeanMedianModeImputationVMatrix.h.

Referenced by build_(), and createMeanMedianModeFile().

A referenced train set.

The mean, median or mode is computed with the observed values in this data set. It is used in combination with the option number_of_train_samples_to_use.

Definition at line 65 of file MeanMedianModeImputationVMatrix.h.

Referenced by build_(), computeMeanMedianModeVectors(), declareOptions(), loadMeanMedianModeFile(), makeDeepCopyFromShallowCopy(), PLearn::Preprocessing::manageTrainTestUnknownSets(), and setMetaDataDir().

The vector of coded instruction for each variables.

Definition at line 111 of file MeanMedianModeImputationVMatrix.h.

Referenced by build_(), declareOptions(), get(), getColumn(), getRow(), getSubRow(), and makeDeepCopyFromShallowCopy().


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines