PLearn 0.1
|
#include <MeanMedianModeImputationVMatrix.h>
Public Member Functions | |
MeanMedianModeImputationVMatrix () | |
virtual | ~MeanMedianModeImputationVMatrix () |
virtual void | build () |
Simply calls inherited::build() then build_(). | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be. | |
virtual real | get (int i, int j) const |
This method must be implemented in all subclasses. | |
virtual void | getSubRow (int i, int j, Vec v) const |
It is suggested that this method be implemented in subclasses to speed up accesses (default version repeatedly calls get(i,j) which may have a significant overhead). | |
virtual void | getRow (int i, Vec v) const |
These methods do not usually need to be overridden in subclasses (default versions call getSubRow, which should do just fine) | |
virtual void | getColumn (int i, Vec v) const |
Copies column i into v (which must have appropriate length equal to the VMat's length). | |
VMat | getMeanMedianModeFile () |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual MeanMedianModeImputationVMatrix * | deepCopy (CopiesMap &copies) const |
Static Public Member Functions | |
static void | declareOptions (OptionList &ol) |
Declares this class' options. | |
static string | _classname_ () |
MeanMedianModeImputationVMatrix. | |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
VMat | train_set |
A referenced train set. | |
real | number_of_train_samples_to_use |
The number of samples from the train set that will be examined to compute the required statistic for each variable. | |
TVec< pair< string, string > > | imputation_spec |
Pairs of instruction of the form field_name : mean | median | mode | none. | |
bool | missing_field_error |
if true will generate an error if the imputation_spec reference a field not in the source. | |
string | default_instruction |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Private Types | |
typedef ImputationVMatrix | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. | |
virtual void | setMetaDataDir (const PPath &the_metadatadir) |
This should be called by the build method of every VMatrix that has a metadatadir. | |
void | createMeanMedianModeFile (PPath file_name) |
void | loadMeanMedianModeFile (PPath file_name) |
void | computeMeanMedianModeVectors () |
Static Private Member Functions | |
static void | sortColumn (Vec input_vec, int start, int end) |
static void | sortSmallSubArray (Vec input_vec, int start_index, int end_index) |
static void | swapValues (Vec input_vec, int index_i, int index_j) |
static real | compare (real field1, real field2) |
Private Attributes | |
TVec< string > | train_field_names |
VMat | mean_median_mode_file |
Vec | variable_mean |
The vector of variable means observed from the train set. | |
Vec | variable_median |
The vector of variable medians observed from the train set. | |
Vec | variable_mode |
The vector of variable modes observed from the train set. | |
TVec< int > | variable_imputation_instruction |
The vector of coded instruction for each variables. |
Definition at line 55 of file MeanMedianModeImputationVMatrix.h.
typedef ImputationVMatrix PLearn::MeanMedianModeImputationVMatrix::inherited [private] |
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 57 of file MeanMedianModeImputationVMatrix.h.
PLearn::MeanMedianModeImputationVMatrix::MeanMedianModeImputationVMatrix | ( | ) |
Definition at line 58 of file MeanMedianModeImputationVMatrix.cc.
: number_of_train_samples_to_use(0.0), missing_field_error(true) { }
PLearn::MeanMedianModeImputationVMatrix::~MeanMedianModeImputationVMatrix | ( | ) | [virtual] |
Definition at line 64 of file MeanMedianModeImputationVMatrix.cc.
{ }
string PLearn::MeanMedianModeImputationVMatrix::_classname_ | ( | ) | [static] |
MeanMedianModeImputationVMatrix.
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.
OptionList & PLearn::MeanMedianModeImputationVMatrix::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.
RemoteMethodMap & PLearn::MeanMedianModeImputationVMatrix::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.
Object * PLearn::MeanMedianModeImputationVMatrix::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.
StaticInitializer MeanMedianModeImputationVMatrix::_static_initializer_ & PLearn::MeanMedianModeImputationVMatrix::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.
void PLearn::MeanMedianModeImputationVMatrix::build | ( | ) | [virtual] |
Simply calls inherited::build() then build_().
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 142 of file MeanMedianModeImputationVMatrix.cc.
References PLearn::ImputationVMatrix::build(), and build_().
Referenced by PLearn::Preprocessing::manageTrainTestUnknownSets().
{ inherited::build(); build_(); }
void PLearn::MeanMedianModeImputationVMatrix::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 249 of file MeanMedianModeImputationVMatrix.cc.
References PLearn::TVec< T >::append(), PLearn::TVec< T >::clear(), PLearn::TVec< T >::copy(), PLearn::VMatrix::declareFieldNames(), default_instruction, PLearn::TVec< T >::fill(), PLearn::TVec< T >::first(), i, imputation_spec, PLearn::TVec< T >::length(), PLearn::VMat::length(), missing_field_error, n, number_of_train_samples_to_use, PLERROR, PLWARN_ERR, PLearn::TVec< T >::resize(), PLearn::VMatrix::setMetaInfoFrom(), PLearn::TVec< T >::size(), PLearn::ImputationVMatrix::source, PLearn::string_begins_with(), PLearn::tostring(), train_field_names, train_set, PLearn::VMatrix::updateMtime(), variable_imputation_instruction, variable_mean, variable_median, variable_mode, and PLearn::VMat::width().
Referenced by build().
{ if (!source) PLERROR("In MeanMedianModeImputationVMatrix:: source vmat must be supplied"); if (!train_set) train_set = source; updateMtime(train_set); updateMtime(source); int train_length = train_set->length(); int train_width = train_set->width(); if (number_of_train_samples_to_use > 0.0){ if (number_of_train_samples_to_use < 1.0) train_length = (int) (number_of_train_samples_to_use * (real) train_length); else train_length = (int) number_of_train_samples_to_use; } if (train_length > train_set->length()) train_length = train_set->length(); if(train_length < 1) PLERROR("In MeanMedianModeImputationVMatrix::length of the number of" " train samples to use must be at least 1, got: %i", train_length); if(train_set->inputsize() < 1) PLERROR("In MeanMedianModeImputationVMatrix::inputsize of the train vmat" " must be supplied, got : %i", train_set->inputsize()); source->compatibleSizeError(train_set); setMetaInfoFrom(source); train_field_names.resize(train_width); train_field_names = train_set->fieldNames(); declareFieldNames(train_field_names); variable_mean.resize(train_width); variable_median.resize(train_width); variable_mode.resize(train_width); variable_imputation_instruction.resize(train_width); if(default_instruction.empty()) variable_imputation_instruction.clear(); else if (default_instruction == "mean") variable_imputation_instruction.fill(1); else if (default_instruction == "median") variable_imputation_instruction.fill(2); else if (default_instruction == "mode") variable_imputation_instruction.fill(3); else if (default_instruction == "none") variable_imputation_instruction.fill(4); else if (default_instruction == "err") variable_imputation_instruction.fill(5); else PLERROR("In MeanMedianModeImputationVMatrix: unsupported default_imputation instruction: %s ", default_instruction.c_str()); TVec<string> nofields; //We sho TVec<pair<string,string> > save_imputation_spec = imputation_spec; imputation_spec = save_imputation_spec.copy(); TVec<string> not_expanded; for (int spec_col = 0; spec_col < imputation_spec.size(); spec_col++) { int train_col; string fname = imputation_spec[spec_col].first; for (train_col = 0; train_col < train_width; train_col++) { if (fname == train_field_names[train_col]) break; } char last_char = fname[fname.size()-1]; if (train_col >= train_width && last_char!='*'){ nofields.append(fname.c_str()); continue; } else if(train_col >= train_width && last_char=='*') { bool expended = false; fname.resize(fname.size()-1);//remove the last caracter (*) for(train_col = 0; train_col < train_width; train_col++) { if(string_begins_with(train_field_names[train_col],fname)) { pair<string,string> n=make_pair(train_field_names[train_col], imputation_spec[spec_col].second); // perr<<"expanding "<<train_field_names[train_col] << " to " << n <<endl; imputation_spec.append(n); expended = true; } } if(!expended){ not_expanded.append(imputation_spec[spec_col].first); } continue; } if (imputation_spec[spec_col].second == "mean") variable_imputation_instruction[train_col] = 1; else if (imputation_spec[spec_col].second == "median") variable_imputation_instruction[train_col] = 2; else if (imputation_spec[spec_col].second == "mode") variable_imputation_instruction[train_col] = 3; else if (imputation_spec[spec_col].second == "none") variable_imputation_instruction[train_col] = 4; else if (imputation_spec[spec_col].second == "err") variable_imputation_instruction[train_col] = 5; else PLERROR("In MeanMedianModeImputationVMatrix: unsupported imputation instruction: %s : %s", (imputation_spec[spec_col].first).c_str(), (imputation_spec[spec_col].second).c_str()); } if(not_expanded.length()>0){ PLWARN_ERR(!missing_field_error, "In MeanMedianModeImputationVMatrix::build_() - " "For %d spec, we didn't found partial match '%s'", not_expanded.length(), tostring(not_expanded).c_str()); } imputation_spec = save_imputation_spec; if(nofields.length()>0) PLWARN_ERR(!missing_field_error, "In MeanMedianModeImputationVMatrix::build_() Their is %d" " fields in the imputation_spec that are not in train set:" " '%s'",nofields.length(), tostring(nofields).c_str()); TVec<string> no_instruction; for(int i = 0;i<variable_imputation_instruction.size();i++) if(variable_imputation_instruction[i]==0) no_instruction.append(train_field_names[i]); if(no_instruction.size()>0) PLERROR("In MeanMedianModeImputationVMatrix::build_() In the source" " VMatrix their is %d field(s) that do not have instruction: '%s'.", no_instruction.size(),tostring(no_instruction).c_str()); }
string PLearn::MeanMedianModeImputationVMatrix::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.
real PLearn::MeanMedianModeImputationVMatrix::compare | ( | real | field1, |
real | field2 | ||
) | [static, private] |
Definition at line 590 of file MeanMedianModeImputationVMatrix.cc.
References PLearn::is_missing().
Referenced by sortColumn(), and sortSmallSubArray().
{ if (is_missing(field1) && is_missing(field2)) return 0.0; if (is_missing(field1)) return +1.0; if (is_missing(field2)) return -1.0; return field1 - field2; }
void PLearn::MeanMedianModeImputationVMatrix::computeMeanMedianModeVectors | ( | ) | [private] |
Definition at line 437 of file MeanMedianModeImputationVMatrix.cc.
References PLearn::TVec< T >::clear(), PLearn::VMat::getColumn(), PLearn::is_missing(), PLearn::VMat::length(), sortColumn(), train_set, PLearn::ProgressBar::update(), variable_mean, variable_median, variable_mode, and PLearn::VMatrix::width_.
Referenced by setMetaDataDir().
{ TVec<int> variable_present_count(width_); TVec<int> variable_mode_count(width_); variable_mean.clear(); variable_median.clear(); variable_mode.clear(); variable_present_count.clear(); variable_mode_count.clear(); Vec variable_vec(train_set->length()); cout << fixed << showpoint; ProgressBar* pb = 0; pb = new ProgressBar("Computing the mean, median and mode vectors", width_); VMat train_set_mem = new MemoryVMatrix(train_set); for (int train_col = 0; train_col < width_; train_col++) { real current_value = 0.0; int current_value_count = 0; train_set_mem->getColumn(train_col, variable_vec); sortColumn(variable_vec, 0, train_set_mem->length()); for (int train_row = 0; train_row < train_set_mem->length(); train_row++) { if (is_missing(variable_vec[train_row])) continue; variable_mean[train_col] += variable_vec[train_row]; variable_present_count[train_col] += 1; if (variable_vec[train_row] != current_value) { if (current_value_count > variable_mode_count[train_col]) { variable_mode[train_col] = current_value; variable_mode_count[train_col] = current_value_count; } current_value_count = 0; current_value = variable_vec[train_row]; } current_value_count += 1; } if (variable_present_count[train_col] > 0) { variable_mean[train_col] = variable_mean[train_col] / variable_present_count[train_col]; variable_median[train_col] = variable_vec[(variable_present_count[train_col] / 2)]; } if (current_value_count > variable_mode_count[train_col]) { variable_mode[train_col] = current_value; variable_mode_count[train_col] = current_value_count; } pb->update( train_col ); /* cout << "col: " << setw(3) << train_col << " present: " << setw(5) << variable_present_count[train_col] << " missing: " << setw(5) << variable_missing_count[train_col] << " mean: " << setw(11) << setprecision(2) << variable_mean[train_col] << " median: " << setw(11) << setprecision(2) << variable_median[train_col] << " mode count: " << setw(5) << variable_mode_count[train_col] << " mode: " << setw(11) << setprecision(2) << variable_mode[train_col] << " name: " << train_field_names[train_col] << endl; */ } delete pb; }
void PLearn::MeanMedianModeImputationVMatrix::createMeanMedianModeFile | ( | PPath | file_name | ) | [private] |
Definition at line 407 of file MeanMedianModeImputationVMatrix.cc.
References mean_median_mode_file, train_field_names, variable_mean, variable_median, and variable_mode.
Referenced by setMetaDataDir().
{ mean_median_mode_file = new FileVMatrix(file_name, 3, train_field_names); mean_median_mode_file->setMetaInfoFrom(this); mean_median_mode_file->putRow(0, variable_mean); mean_median_mode_file->putRow(1, variable_median); mean_median_mode_file->putRow(2, variable_mode); }
void PLearn::MeanMedianModeImputationVMatrix::declareOptions | ( | OptionList & | ol | ) | [static] |
Declares this class' options.
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 68 of file MeanMedianModeImputationVMatrix.cc.
References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::ImputationVMatrix::declareOptions(), default_instruction, PLearn::VMatrix::extrasize_, imputation_spec, PLearn::VMatrix::inputsize_, PLearn::OptionBase::learntoption, PLearn::VMatrix::length_, missing_field_error, PLearn::OptionBase::nosave, number_of_train_samples_to_use, PLearn::redeclareOption(), PLearn::VMatrix::targetsize_, train_set, variable_imputation_instruction, variable_mean, variable_median, variable_mode, PLearn::VMatrix::weightsize_, and PLearn::VMatrix::width_.
{ declareOption(ol, "train_set", &MeanMedianModeImputationVMatrix::train_set, OptionBase::buildoption, "A referenced train set.\n" "The mean, median or mode is computed with the observed values in this data set.\n" "It is used in combination with the option number_of_train_samples_to_use\n"); declareOption(ol, "number_of_train_samples_to_use", &MeanMedianModeImputationVMatrix::number_of_train_samples_to_use, OptionBase::buildoption, "The number of samples from the train set that will be examined to compute the required statistic for each variable.\n" "If equal to zero, all the samples from the train set are used to calculated the statistics.\n" "If it is a fraction between 0 and 1, this proportion of the samples are used.\n" "If greater or equal to 1, the integer portion is interpreted as the number of samples to use."); declareOption(ol, "imputation_spec", &MeanMedianModeImputationVMatrix::imputation_spec, OptionBase::buildoption, "Pairs of instruction of the form field_name : mean | median | mode | none | err.\n" " -mean : put the mean of the field if one value is missing\n" " -median: put the median of the field if one value is missing\n" " -mode : put the mode(most frequent value) of the field if one value is missing\n" " -none : let the missing value in this field\n" " -err : make it an error to have a missing value in this field" ); declareOption(ol, "missing_field_error", &MeanMedianModeImputationVMatrix::missing_field_error, OptionBase::buildoption, "If True, will generate an error if some field in the" " imputation_spec are present but not in the source. Otherwise" " will generate a warning. This also applies for regex spec."); declareOption(ol, "default_instruction", &MeanMedianModeImputationVMatrix::default_instruction, OptionBase::buildoption, "The default instruction to use. If empty(default), will generate" " an error is some source variable don't have an one in imputation_spec."); declareOption(ol, "variable_mean", &MeanMedianModeImputationVMatrix::variable_mean, OptionBase::learntoption, "The vector of variable means observed from the train set."); declareOption(ol, "variable_median", &MeanMedianModeImputationVMatrix::variable_median, OptionBase::learntoption, "The vector of variable medians observed from the train set."); declareOption(ol, "variable_mode", &MeanMedianModeImputationVMatrix::variable_mode, OptionBase::learntoption, "The vector of variable modes observed from the train set."); declareOption(ol, "variable_imputation_instruction", &MeanMedianModeImputationVMatrix::variable_imputation_instruction, OptionBase::learntoption, "The vector of coded instruction for each variables."); inherited::declareOptions(ol); declareOption(ol, "length", &MeanMedianModeImputationVMatrix::length_, OptionBase::nosave, "The number of example. Computed each time from source."); redeclareOption(ol, "inputsize", &MeanMedianModeImputationVMatrix::inputsize_, OptionBase::nosave, "Taken from source in MeanMedianModeImputationVMatrix."); redeclareOption(ol, "targetsize", &MeanMedianModeImputationVMatrix::targetsize_, OptionBase::nosave, "Taken from source in MeanMedianModeImputationVMatrix."); redeclareOption(ol, "weightsize", &MeanMedianModeImputationVMatrix::weightsize_, OptionBase::nosave, "Taken from source in MeanMedianModeImputationVMatrix."); redeclareOption(ol, "extrasize", &MeanMedianModeImputationVMatrix::extrasize_, OptionBase::nosave, "Taken from source in MeanMedianModeImputationVMatrix."); redeclareOption(ol, "width", &MeanMedianModeImputationVMatrix::width_, OptionBase::nosave, "Taken from source in MeanMedianModeImputationVMatrix."); }
static const PPath& PLearn::MeanMedianModeImputationVMatrix::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 123 of file MeanMedianModeImputationVMatrix.h.
MeanMedianModeImputationVMatrix * PLearn::MeanMedianModeImputationVMatrix::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.
This method must be implemented in all subclasses.
Returns element (i,j).
Implements PLearn::VMatrix.
Definition at line 160 of file MeanMedianModeImputationVMatrix.cc.
References PLearn::is_missing(), j, MISSING_VALUE, PLERROR, PLearn::ImputationVMatrix::source, variable_imputation_instruction, variable_mean, variable_median, and variable_mode.
{ real variable_value = source->get(i, j); if (!is_missing(variable_value)) return variable_value; else if (variable_imputation_instruction[j] == 1) return variable_mean[j]; else if (variable_imputation_instruction[j] == 2) return variable_median[j]; else if (variable_imputation_instruction[j] == 3) return variable_mode[j]; else if (variable_imputation_instruction[j] == 4) return variable_value; else if (variable_imputation_instruction[j] == 5) return MISSING_VALUE;//PLERROR("");//PLERROR("In MeanMedianModeImputationVMatrix::get(%d,%d) - the value is" // " missing and have a instruction err!",i,j); else PLERROR("In MeanMedianModeImputationVMatrix::get(%d,%d) - " "unknow variable_imputation_instruction value of %d",i,j, variable_imputation_instruction[j] ); //Should not be executed, to remove a warning return MISSING_VALUE; }
Copies column i into v (which must have appropriate length equal to the VMat's length).
Reimplemented from PLearn::VMatrix.
Definition at line 227 of file MeanMedianModeImputationVMatrix.cc.
References i, PLearn::is_missing(), PLearn::TVec< T >::length(), PLERROR, PLearn::ImputationVMatrix::source, variable_imputation_instruction, variable_mean, variable_median, and variable_mode.
{ source-> getColumn(i, v); for (int source_row = 0; source_row < v->length(); source_row++) if (is_missing(v[source_row])){ if (variable_imputation_instruction[i] == 1) v[source_row] = variable_mean[i]; else if (variable_imputation_instruction[i] == 2) v[source_row] = variable_median[i]; else if (variable_imputation_instruction[i] == 3) v[source_row] = variable_mode[i]; else if (variable_imputation_instruction[i] == 4) ;//do nothing else if (variable_imputation_instruction[i] == 5) return PLERROR("In MeanMedianModeImputationVMatrix::getColumn(%d) - the value is" " missing and have a instruction err!",i); else PLERROR("In MeanMedianModeImputationVMatrix::getRow(%d, Vec) - " "unknow variable_imputation_instruction value of %d",i, variable_imputation_instruction[i] ); } }
VMat PLearn::MeanMedianModeImputationVMatrix::getMeanMedianModeFile | ( | ) |
Definition at line 432 of file MeanMedianModeImputationVMatrix.cc.
References mean_median_mode_file.
Referenced by PLearn::Preprocessing::manageTrainTestUnknownSets().
{ return mean_median_mode_file; }
OptionList & PLearn::MeanMedianModeImputationVMatrix::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.
OptionMap & PLearn::MeanMedianModeImputationVMatrix::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.
RemoteMethodMap & PLearn::MeanMedianModeImputationVMatrix::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 56 of file MeanMedianModeImputationVMatrix.cc.
These methods do not usually need to be overridden in subclasses (default versions call getSubRow, which should do just fine)
Copies row i into v (which must have appropriate length equal to the VMat's width).
Reimplemented from PLearn::VMatrix.
Definition at line 203 of file MeanMedianModeImputationVMatrix.cc.
References PLearn::VMatrix::fieldName(), PLearn::is_missing(), PLearn::TVec< T >::length(), PLERROR, PLearn::ImputationVMatrix::source, variable_imputation_instruction, variable_mean, variable_median, and variable_mode.
{ source-> getRow(i, v); for (int source_col = 0; source_col < v->length(); source_col++) if (is_missing(v[source_col])){ if (variable_imputation_instruction[source_col] == 1) v[source_col] = variable_mean[source_col]; else if (variable_imputation_instruction[source_col] == 2) v[source_col] = variable_median[source_col]; else if (variable_imputation_instruction[source_col] == 3) v[source_col] = variable_mode[source_col]; else if (variable_imputation_instruction[source_col] == 4) ;//do nothing else if (variable_imputation_instruction[source_col] == 5) return PLERROR("In MeanMedianModeImputationVMatrix::getRow(%d) -" " the value is missing for column %s" " and have a instruction err!",i, fieldName(source_col).c_str()); else PLERROR("In MeanMedianModeImputationVMatrix::getRow(%d, Vec) - " "unknow variable_imputation_instruction value of %d",i, variable_imputation_instruction[source_col] ); } }
It is suggested that this method be implemented in subclasses to speed up accesses (default version repeatedly calls get(i,j) which may have a significant overhead).
Fills v with the subrow i lying between columns j (inclusive) and j+v.length() (exclusive).
Reimplemented from PLearn::VMatrix.
Definition at line 179 of file MeanMedianModeImputationVMatrix.cc.
References PLearn::VMat::getSubRow(), PLearn::is_missing(), j, PLearn::TVec< T >::length(), PLERROR, PLearn::ImputationVMatrix::source, variable_imputation_instruction, variable_mean, variable_median, and variable_mode.
{ source->getSubRow(i, j, v); for (int source_col = 0; source_col < v->length(); source_col++) if (is_missing(v[source_col])){ if (variable_imputation_instruction[source_col + j] == 1) v[source_col] = variable_mean[source_col + j]; else if (variable_imputation_instruction[source_col + j] == 2) v[source_col] = variable_median[source_col + j]; else if (variable_imputation_instruction[source_col + j] == 3) v[source_col] = variable_mode[source_col + j]; else if (variable_imputation_instruction[source_col + j] == 4) ;//do nothing else if (variable_imputation_instruction[source_col + j] == 5) return PLERROR("In MeanMedianModeImputationVMatrix::getSubRow(%d,%d) - the value is" " missing and have a instruction err!",i,j); else PLERROR("In MeanMedianModeImputationVMatrix::getSubRow(%d,%d, Vec) - " "unknow variable_imputation_instruction value of %d",i,j, variable_imputation_instruction[source_col + j] ); } }
void PLearn::MeanMedianModeImputationVMatrix::loadMeanMedianModeFile | ( | PPath | file_name | ) | [private] |
Definition at line 416 of file MeanMedianModeImputationVMatrix.cc.
References PLearn::VMatrix::compatibleSizeError(), PLearn::VMatrix::fieldNames(), mean_median_mode_file, PLERROR, train_set, variable_mean, variable_median, and variable_mode.
Referenced by setMetaDataDir().
{ train_set->isUpToDate(file_name,true,true); mean_median_mode_file = new FileVMatrix(file_name); compatibleSizeError(mean_median_mode_file, "Bad file "+file_name); if(mean_median_mode_file->fieldNames()!=fieldNames()) PLERROR("In MeanMedianModeImputationVMatrix::loadMeanMedianModeFile(%s) -" " The file don't have the same field name as the source. " "Delete it to have it recreated it automatically.", file_name.c_str()); mean_median_mode_file->getRow(0, variable_mean); mean_median_mode_file->getRow(1, variable_median); mean_median_mode_file->getRow(2, variable_mode); }
void PLearn::MeanMedianModeImputationVMatrix::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be.
This needs to be overridden by every class that adds "complex" data members to the class, such as Vec
, Mat
, PP<Something>
, etc. Typical implementation:
void CLASS_OF_THIS::makeDeepCopyFromShallowCopy(CopiesMap& copies) { inherited::makeDeepCopyFromShallowCopy(copies); deepCopyField(complex_data_member1, copies); deepCopyField(complex_data_member2, copies); ... }
copies | A map used by the deep-copy mechanism to keep track of already-copied objects. |
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 148 of file MeanMedianModeImputationVMatrix.cc.
References PLearn::deepCopyField(), imputation_spec, PLearn::ImputationVMatrix::makeDeepCopyFromShallowCopy(), number_of_train_samples_to_use, train_set, variable_imputation_instruction, variable_mean, variable_median, and variable_mode.
{ deepCopyField(train_set, copies); deepCopyField(number_of_train_samples_to_use, copies); deepCopyField(imputation_spec, copies); deepCopyField(variable_mean, copies); deepCopyField(variable_median, copies); deepCopyField(variable_mode, copies); deepCopyField(variable_imputation_instruction, copies); inherited::makeDeepCopyFromShallowCopy(copies); }
void PLearn::MeanMedianModeImputationVMatrix::setMetaDataDir | ( | const PPath & | the_metadatadir | ) | [private, virtual] |
This should be called by the build method of every VMatrix that has a metadatadir.
It will create said directory if it doesn's already exist. Throws a PLERROR if called with an empty string.
Reimplemented from PLearn::VMatrix.
Definition at line 372 of file MeanMedianModeImputationVMatrix.cc.
References computeMeanMedianModeVectors(), createMeanMedianModeFile(), PLearn::VMatrix::getMetaDataDir(), PLearn::VMatrix::hasMetaDataDir(), PLearn::isfile(), loadMeanMedianModeFile(), PLERROR, PLearn::rm(), PLearn::VMatrix::setMetaDataDir(), PLearn::ImputationVMatrix::source, and train_set.
{ inherited::setMetaDataDir(the_metadatadir); if(!train_set->hasMetaDataDir() && !hasMetaDataDir()) PLERROR("In MeanMedianModeImputationVMatrix::setMetaDataDir() - the " " train_set should have a metadata dir or we should have one!"); else if(!train_set->hasMetaDataDir()) train_set->setMetaDataDir(getMetaDataDir()/"train_set"); PPath train_metadata = train_set->getMetaDataDir(); PPath mean_median_mode_file_name = train_metadata + "mean_median_mode_file.pmat"; bool uptodate = train_set->isUpToDate(mean_median_mode_file_name,false) && source->isUpToDate(mean_median_mode_file_name,false); train_set->lockMetaDataDir(); try{ if (!uptodate) { computeMeanMedianModeVectors(); createMeanMedianModeFile(mean_median_mode_file_name); } else loadMeanMedianModeFile(mean_median_mode_file_name); }catch(const PLearnError& e){ train_set->unlockMetaDataDir(); //we erase the file if we are creating it // as it can be partilly saved. if(!uptodate && isfile(mean_median_mode_file_name)) rm(mean_median_mode_file_name); throw e; } train_set->unlockMetaDataDir(); }
void PLearn::MeanMedianModeImputationVMatrix::sortColumn | ( | Vec | input_vec, |
int | start, | ||
int | end | ||
) | [static, private] |
Definition at line 501 of file MeanMedianModeImputationVMatrix.cc.
References compare(), PLERROR, sortSmallSubArray(), and swapValues().
Referenced by computeMeanMedianModeVectors().
{ int start_index = start; int end_index = end - 1; int forward_index; int backward_index; int stack_index = -1; real pivot_value; TVec<int> stack(50); for (;;) { if ((end_index - start_index) < 7) { if (end_index > start_index) { sortSmallSubArray(input_vec, start_index, end_index); } if (stack_index < 0) { break; } end_index = stack[stack_index--]; start_index = stack[stack_index--]; } else { swapValues(input_vec, start_index + 1, (start_index + end_index) / 2); if (compare(input_vec[start_index], input_vec[end_index]) > 0.0) swapValues(input_vec, start_index, end_index); if (compare(input_vec[start_index + 1], input_vec[end_index]) > 0.0) swapValues(input_vec, start_index + 1, end_index); if (compare(input_vec[start_index], input_vec[start_index + 1]) > 0.0) swapValues(input_vec, start_index, start_index + 1); forward_index = start_index + 1; backward_index = end_index; pivot_value = input_vec[start_index + 1]; for (;;) { do forward_index++; while (compare(input_vec[forward_index], pivot_value) < 0.0); do backward_index--; while (compare(input_vec[backward_index], pivot_value) > 0.0); if (backward_index < forward_index) { break; } swapValues(input_vec, forward_index, backward_index); } swapValues(input_vec, start_index + 1, backward_index); stack_index += 2; if (stack_index > 50) PLERROR("RegressionTreeRegistersVMatrix: the stack for sorting the rows is too small"); if ((end_index - forward_index + 1) >= (backward_index - start_index)) { stack[stack_index] = end_index; stack[stack_index - 1] = forward_index; end_index = backward_index - 1; } else { stack[stack_index] = backward_index - 1; stack[stack_index - 1] = start_index; start_index = forward_index; } } } }
void PLearn::MeanMedianModeImputationVMatrix::sortSmallSubArray | ( | Vec | input_vec, |
int | start_index, | ||
int | end_index | ||
) | [static, private] |
Definition at line 564 of file MeanMedianModeImputationVMatrix.cc.
References compare().
Referenced by sortColumn().
{ int index_i; int index_j; for (index_i = start_index + 1; index_i <= end_index; index_i++) { real saved_value = input_vec[index_i]; for (index_j = index_i - 1; index_j >= start_index; index_j--) { if (compare(input_vec[index_j], saved_value) <= 0.0) { break; } input_vec[index_j + 1] = input_vec[index_j]; } input_vec[index_j + 1] = saved_value; } }
void PLearn::MeanMedianModeImputationVMatrix::swapValues | ( | Vec | input_vec, |
int | index_i, | ||
int | index_j | ||
) | [static, private] |
Definition at line 583 of file MeanMedianModeImputationVMatrix.cc.
Referenced by sortColumn().
{ real saved_value = input_vec[index_i]; input_vec[index_i] = input_vec[index_j]; input_vec[index_j] = saved_value; }
Reimplemented from PLearn::ImputationVMatrix.
Definition at line 123 of file MeanMedianModeImputationVMatrix.h.
Definition at line 79 of file MeanMedianModeImputationVMatrix.h.
Referenced by build_(), and declareOptions().
TVec< pair<string, string> > PLearn::MeanMedianModeImputationVMatrix::imputation_spec |
Pairs of instruction of the form field_name : mean | median | mode | none.
Definition at line 74 of file MeanMedianModeImputationVMatrix.h.
Referenced by build_(), declareOptions(), makeDeepCopyFromShallowCopy(), and PLearn::Preprocessing::manageTrainTestUnknownSets().
Definition at line 98 of file MeanMedianModeImputationVMatrix.h.
Referenced by createMeanMedianModeFile(), getMeanMedianModeFile(), and loadMeanMedianModeFile().
if true will generate an error if the imputation_spec reference a field not in the source.
Otherwise generate a warning.
Definition at line 78 of file MeanMedianModeImputationVMatrix.h.
Referenced by build_(), and declareOptions().
The number of samples from the train set that will be examined to compute the required statistic for each variable.
If equal to zero, all the samples from the train set are used to calculated the statistics. If it is a fraction between 0 and 1, this proportion of the samples are used. If greater or equal to 1, the integer portion is interpreted as the number of samples to use.
Definition at line 71 of file MeanMedianModeImputationVMatrix.h.
Referenced by build_(), declareOptions(), makeDeepCopyFromShallowCopy(), and PLearn::Preprocessing::manageTrainTestUnknownSets().
TVec<string> PLearn::MeanMedianModeImputationVMatrix::train_field_names [private] |
Definition at line 97 of file MeanMedianModeImputationVMatrix.h.
Referenced by build_(), and createMeanMedianModeFile().
A referenced train set.
The mean, median or mode is computed with the observed values in this data set. It is used in combination with the option number_of_train_samples_to_use.
Definition at line 65 of file MeanMedianModeImputationVMatrix.h.
Referenced by build_(), computeMeanMedianModeVectors(), declareOptions(), loadMeanMedianModeFile(), makeDeepCopyFromShallowCopy(), PLearn::Preprocessing::manageTrainTestUnknownSets(), and setMetaDataDir().
The vector of coded instruction for each variables.
Definition at line 111 of file MeanMedianModeImputationVMatrix.h.
Referenced by build_(), declareOptions(), get(), getColumn(), getRow(), getSubRow(), and makeDeepCopyFromShallowCopy().
The vector of variable means observed from the train set.
Definition at line 102 of file MeanMedianModeImputationVMatrix.h.
Referenced by build_(), computeMeanMedianModeVectors(), createMeanMedianModeFile(), declareOptions(), get(), getColumn(), getRow(), getSubRow(), loadMeanMedianModeFile(), and makeDeepCopyFromShallowCopy().
The vector of variable medians observed from the train set.
Definition at line 105 of file MeanMedianModeImputationVMatrix.h.
Referenced by build_(), computeMeanMedianModeVectors(), createMeanMedianModeFile(), declareOptions(), get(), getColumn(), getRow(), getSubRow(), loadMeanMedianModeFile(), and makeDeepCopyFromShallowCopy().
The vector of variable modes observed from the train set.
Definition at line 108 of file MeanMedianModeImputationVMatrix.h.
Referenced by build_(), computeMeanMedianModeVectors(), createMeanMedianModeFile(), declareOptions(), get(), getColumn(), getRow(), getSubRow(), loadMeanMedianModeFile(), and makeDeepCopyFromShallowCopy().