PLearn 0.1
|
Generate samples from a mixture of two gaussians. More...
#include <TestImputations.h>
Public Member Functions | |
TestImputations () | |
Default constructor. | |
int | outputsize () const |
SUBCLASS WRITING: override this so that it returns the size of this learner's output, as a function of its inputsize(), targetsize() and set options. | |
void | train () |
*** SUBCLASS WRITING: *** | |
void | computeOutput (const Vec &, Vec &) const |
*** SUBCLASS WRITING: *** | |
void | computeCostsFromOutputs (const Vec &, const Vec &, const Vec &, Vec &) const |
*** SUBCLASS WRITING: *** | |
TVec< string > | getTestCostNames () const |
*** SUBCLASS WRITING: *** | |
TVec< string > | getTrainCostNames () const |
*** SUBCLASS WRITING: *** | |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual TestImputations * | deepCopy (CopiesMap &copies) const |
virtual void | build () |
Finish building the object; just call inherited::build followed by build_() | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transforms a shallow copy into a deep copy. | |
Static Public Member Functions | |
static string | _classname_ () |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
int | min_number_of_samples |
### declare public option fields (such as build options) here Start your comments with Doxygen-compatible comments such as //! | |
int | max_number_of_samples |
The maximum number of samples used to test imputations for a variable. | |
PPath | mean_median_mode_file_name |
The Path of the file with those statistics for all the variables. | |
PPath | tree_conditional_mean_directory |
The Path of the dircetory containing the tree conditional means computed for each variable. | |
PPath | covariance_preservation_file_name |
The Path of the file with the train_set empirically observed covariances and means. | |
VMat | reference_set_with_covpres |
The reference set corresponding to the index computed with the ball_tree, with the initial imputations. | |
VMat | reference_set_with_missing |
The reference set corresponding to the index computed with the ball_tree, with missing values. | |
TVec< string > | missing_indicators |
The vector of missing indicator field names to be excluded in the distance computation. | |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declares the class options. | |
Private Types | |
typedef PLearner | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. | |
void | build_ball_tree (int nb_neighbors) |
void | initialize () |
void | computeMeanMedianModeStats () |
void | computeTreeCondMeanStats () |
void | computeCovPresStats () |
real | covariancePreservationValue (int col) |
void | computeNeighborhoodStats (int nb_neighbors, int max_miss_neigbors) |
void | createHeaderFile () |
void | getHeaderRecord () |
void | updateHeaderRecord (int var_col) |
void | createOutputFile () |
void | getOutputRecord (int var_col) |
void | updateOutputRecord (int var_col) |
void | endtestimputation (const char *msg,...) |
Private Attributes | |
ExhaustiveNearestNeighbors * | ball_tree |
Mat | ref_cov |
Mat | ref_mis |
int | train_length |
int | train_width |
Vec | train_input |
TVec< string > | train_names |
PPath | train_metadata |
StatsCollector | train_stats |
The stats_collector responsible for collecting train cost statistics during training. | |
int | train_total |
int | train_missing |
int | train_present |
PPath | header_file_name |
VMat | header_file |
Vec | header_record |
int | to_deal_with_total |
int | to_deal_with_next |
string | to_deal_with_name |
real | to_deal_with_value |
VMat | test_samples_set |
TVec< int > | indices |
int | test_length |
int | test_width |
real | mmmf_mean_err |
real | mmmf_median_err |
real | mmmf_mode_err |
PPath | tcmf_file_name |
VMat | tcmf_file |
real | tcmf_mean_err |
Mat | cvpf_cov |
Vec | cvpf_mu |
real | cvpf_mean_err |
Vec | knnf_input |
Vec | knnf_neighbors |
Vec | knnf_mean_cov_err |
Vec | knnf_mean_miss_err |
Vec | knnf_nmiss_value_count |
Vec | weights |
WeightedDistance * | weighted_distance_kernel |
PPath | output_file_name |
VMat | output_file |
Vec | output_record |
TVec< string > | output_names |
Generate samples from a mixture of two gaussians.
Definition at line 65 of file TestImputations.h.
typedef PLearner PLearn::TestImputations::inherited [private] |
Reimplemented from PLearn::PLearner.
Definition at line 67 of file TestImputations.h.
PLearn::TestImputations::TestImputations | ( | ) |
string PLearn::TestImputations::_classname_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 51 of file TestImputations.cc.
OptionList & PLearn::TestImputations::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 51 of file TestImputations.cc.
RemoteMethodMap & PLearn::TestImputations::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 51 of file TestImputations.cc.
Reimplemented from PLearn::PLearner.
Definition at line 51 of file TestImputations.cc.
Object * PLearn::TestImputations::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 51 of file TestImputations.cc.
StaticInitializer TestImputations::_static_initializer_ & PLearn::TestImputations::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 51 of file TestImputations.cc.
void PLearn::TestImputations::build | ( | ) | [virtual] |
Finish building the object; just call inherited::build followed by build_()
Reimplemented from PLearn::PLearner.
Definition at line 114 of file TestImputations.cc.
{ // ### Nothing to add here, simply calls build_(). inherited::build(); build_(); }
void PLearn::TestImputations::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::PLearner.
Definition at line 124 of file TestImputations.cc.
References PLearn::endl().
{ /* for each variable with missing values in the train set(which is in this case the test set) - radomly choose up to n samples with a value in the variable - build a set with these samples replacing the value with missing - perform the various type of imputations and compute the errors valider meanmedianmode, treeconditionalmean covariancepreservation, neighborhood create a Mat: width is #of variables with missing values row 0: nb_present row 1: mean/mode imputation from preprocessing/final_train_input_preprocessed.pmat.metadata/mean_median_mode_file.pmat row 2: median/mode imputation from preprocessing/final_train_input_preprocessed.pmat.metadata/mean_median_mode_file.pmat row 3: mode row 4: treeconditionalmean imputation from prep/data/targeted_ind_no_imp.vmat.metadata/TreeCondMean/dir/'field_names'/Split0/test1_outputs.pmat row 5: covariance preservation imputation from preprocessing/final_train_input_preprocessed.pmat.metadata/covariance_file.pmat row 6 to 24: (row - 4) * i neighbors imputation from neighborhood/test_train_imputed_with_covariance_preservation.pmat.metadata/neighborhood_file.pmat lire le train_set */ int nb_neighbors=100; MODULE_LOG << "build_() called" << endl; if (train_set) { build_ball_tree(nb_neighbors*3); output_file_name = train_metadata + "/TestImputation2/output.pmat"; for (int iteration = 1; iteration <= train_set->width(); iteration++) { cout << "In TestImputations, Iteration # " << iteration << endl; initialize(); computeMeanMedianModeStats(); computeTreeCondMeanStats(); computeCovPresStats(); computeNeighborhoodStats(nb_neighbors,nb_neighbors*3); train(); } endtestimputation("In TestImputations::build_(): we are done here"); } }
void PLearn::TestImputations::build_ball_tree | ( | int | nb_neighbors | ) | [private] |
Definition at line 162 of file TestImputations.cc.
References PLearn::endl(), PLERROR, and PLearn::tostring().
{ // initialize primary dataset cout << "initialize the train set" << endl; train_length = train_set->length(); train_width = train_set->width(); train_input.resize(train_width); train_names.resize(train_width); train_names << train_set->fieldNames(); train_metadata = train_set->getMetaDataDir(); weights.resize(train_width); weights.fill(1.0); for (int mi_col = 0; mi_col < missing_indicators.length(); mi_col++) { int train_col; for (train_col = 0; train_col < train_width; train_col++) { if (missing_indicators[mi_col] != train_names[train_col]) continue; weights[train_col] = 0.0; break; } if (train_col >= train_width) PLERROR("In TestImputations::build_ball_tree():: no field with this name in input dataset: %s", (missing_indicators[mi_col]).c_str()); } weighted_distance_kernel = new WeightedDistance(weights); /* if (!reference_set_with_covpres) PLERROR("In TestImputations::build_ball_tree() no reference_set_with_covpres provided."); if (!reference_set_with_missing) PLERROR("In TestImputations::build_ball_tree() no reference_set_with_missing provided."); ball_tree = new BallTreeNearestNeighbors(); ball_tree->setOption("rmin", "1"); ball_tree->setOption("train_method", "anchor"); ball_tree->setOption("num_neighbors", "100"); ball_tree->setOption("copy_input", "0"); ball_tree->setOption("copy_target", "0"); ball_tree->setOption("copy_weight", "0"); ball_tree->setOption("copy_index", "1"); ball_tree->setOption("nstages", "-1"); ball_tree->setOption("report_progress", "1"); ball_tree->setTrainingSet(reference_set_with_covpres, true); ball_tree->train(); ref_cov = reference_set_with_covpres->toMat(); ref_mis = reference_set_with_missing->toMat(); */ if (!reference_set_with_covpres) PLERROR("In TestImputations::build_ball_tree() no reference_set_with_covpres provided."); if (!reference_set_with_missing) PLERROR("In TestImputations::build_ball_tree() no reference_set_with_missing provided."); ball_tree = new ExhaustiveNearestNeighbors(); ball_tree->setOption("num_neighbors", tostring(nb_neighbors)); ball_tree->setOption("copy_input", "0"); ball_tree->setOption("copy_target", "0"); ball_tree->setOption("copy_weight", "0"); ball_tree->setOption("copy_index", "1"); ball_tree->setOption("nstages", "-1"); ball_tree->setOption("report_progress", "1"); ball_tree->distance_kernel = weighted_distance_kernel; ball_tree->setTrainingSet(reference_set_with_covpres, true); ball_tree->train(); ref_cov = reference_set_with_covpres->toMat(); ref_mis = reference_set_with_missing->toMat(); /* ExhaustiveNearestNeighbors( # bool: Whether the kernel defined by the 'distance_kernel' option should be # interpreted as a (pseudo-)distance measure (true) or a similarity # measure (false). Default = true. Note that this interpretation is # strictly specific to the class ExhaustiveNearestNeighbors. kernel_is_pseudo_distance = 1 ; # Ker: Alternate name for 'distance_kernel'. (Deprecated; use only so that # existing scripts can run.) kernel = *1 ->DistanceKernel( n = 2 ; pow_distance = 0 ; optimized = 0 ; is_symmetric = 1 ; report_progress = 0 ; specify_dataset = *0 ; cache_gram_matrix = 0 ; data_inputsize = -1 ; n_examples = -1 ) ; # int: Number of nearest-neighbors to compute. This is usually called "K". # The output vector is simply the concatenation of all found neighbors. # (Default = 1) num_neighbors = 1 ; # bool: If true, the output contains a copy of the found input vector(s). # (Default = false) copy_input = 0 ; # bool: If true, the output contains a copy of the found target vector(s). # (Default = true) copy_target = 1 ; # bool: If true, the output contains a copy of the found weight. If no # weight is present in the training set, a weight of 1.0 is put. # (Default = true) copy_weight = 0 ; # bool: If true, the output contains the index of the found neighbor # (as the row number, zero-based, in the training set.) # (Default = false) copy_index = 0 ; # Ker: An optional alternative to the Euclidean distance (DistanceKernel with # n=2 and pow_distance=1). It should be a 'distance-like' kernel rather # than a 'dot-product-like' kernel, i.e. small when the arguments are # similar, and it should always be non-negative, and 0 only if arguments # are equal. distance_kernel = *1 ->DistanceKernel( n = 2 ; pow_distance = 0 ; optimized = 0 ; is_symmetric = 1 ; report_progress = 0 ; specify_dataset = *0 ; cache_gram_matrix = 0 ; data_inputsize = -1 ; n_examples = -1 ) ; */ }
string PLearn::TestImputations::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 51 of file TestImputations.cc.
void PLearn::TestImputations::computeCostsFromOutputs | ( | const Vec & | input, |
const Vec & | output, | ||
const Vec & | target, | ||
Vec & | costs | ||
) | const [virtual] |
*** SUBCLASS WRITING: ***
This should be defined in subclasses to compute the weighted costs from already computed output. The costs should correspond to the cost names returned by getTestCostNames().
NOTE: In exotic cases, the cost may also depend on some info in the input, that's why the method also gets so see it.
Implements PLearn::PLearner.
Definition at line 645 of file TestImputations.cc.
{}
void PLearn::TestImputations::computeCovPresStats | ( | ) | [private] |
Definition at line 434 of file TestImputations.cc.
References PLearn::isfile(), PLearn::VMat::length(), PLERROR, PLearn::pow(), PLearn::ProgressBar::update(), and PLearn::VMat::width().
{ if (!isfile(covariance_preservation_file_name)) PLERROR("In TestImputations::computeCovPresStats() a valid covariance_preservation_file path must be provided."); VMat cvpf_file = new FileVMatrix(covariance_preservation_file_name); int cvpf_length = cvpf_file->length(); int cvpf_width = cvpf_file->width(); if (cvpf_length != train_width + 1) PLERROR("In TestImputations::computeCovPresStats() there should be %i records in the cvp file, got %i.", train_width + 1, cvpf_length); if (cvpf_width != train_width) PLERROR("In TestImputations::computeCovPresStats() train set and cvp width should be the same, got %i.", cvpf_width); //cvpf_file = new FileVMatrix(covariance_preservation_file_name); cvpf_cov.resize(train_width, train_width); cvpf_mu.resize(train_width); for (int cvpf_row = 0; cvpf_row < train_width; cvpf_row++) { for (int cvpf_col = 0; cvpf_col < train_width; cvpf_col++) { cvpf_cov(cvpf_row, cvpf_col) = cvpf_file->get(cvpf_row, cvpf_col); } } for (int cvpf_col = 0; cvpf_col < train_width; cvpf_col++) { cvpf_mu[cvpf_col] = cvpf_file->get(train_width, cvpf_col); } cvpf_mean_err = 0.0; ProgressBar* pb = new ProgressBar( "computing the covariance preservation imputation errors for " + to_deal_with_name, test_length); for (int test_row = 0; test_row < test_length; test_row++) { test_samples_set->getRow(test_row, train_input); cvpf_mean_err += pow(to_deal_with_value - covariancePreservationValue(to_deal_with_next), 2); pb->update( test_row ); } delete pb; cvpf_mean_err = cvpf_mean_err / (real) test_length; //TODO check the formul //cvpf_mean_stddev = sqrt(cvpf_mean_err); }
void PLearn::TestImputations::computeMeanMedianModeStats | ( | ) | [private] |
Definition at line 377 of file TestImputations.cc.
References PLearn::isfile(), PLearn::VMat::length(), PLERROR, PLearn::pow(), PLearn::ProgressBar::update(), and PLearn::VMat::width().
{ if (!isfile(mean_median_mode_file_name)) PLERROR("In TestImputations::computeMeanMedianModeStats() a valid mean_median_mode_file path must be provided."); VMat mmmf_file = new FileVMatrix(mean_median_mode_file_name); int mmmf_length = mmmf_file->length(); int mmmf_width = mmmf_file->width(); if (mmmf_length != 3) PLERROR("In TestImputations::computeMeanMedianModeStats() there should be exactly 3 records in the mmm file, got %i.", mmmf_length); if (mmmf_width != train_width) PLERROR("In TestImputations::computeMeanMedianModeStats() train set and mmm width should be the same, got %i.", mmmf_width); real mmmf_mean = mmmf_file->get(0, to_deal_with_next); real mmmf_median = mmmf_file->get(1, to_deal_with_next); real mmmf_mode = mmmf_file->get(2, to_deal_with_next); mmmf_mean_err = 0.0; mmmf_median_err = 0.0; mmmf_mode_err = 0.0; ProgressBar* pb = new ProgressBar( "computing the mean, median and mode imputation errors for " + to_deal_with_name, test_length); for (int test_row = 0; test_row < test_length; test_row++) { to_deal_with_value = test_samples_set->get(test_row, to_deal_with_next); mmmf_mean_err += pow(to_deal_with_value - mmmf_mean, 2); mmmf_median_err += pow(to_deal_with_value - mmmf_median, 2); mmmf_mode_err += pow(to_deal_with_value - mmmf_mode, 2); pb->update( test_row ); } delete pb; mmmf_mean_err = mmmf_mean_err / (real) test_length; mmmf_median_err = mmmf_median_err / (real) test_length; mmmf_mode_err = mmmf_mode_err / (real) test_length; //TODO check the formul //mmmf_mean_stddev = sqrt(mmmf_mean_err); //mmmf_median_stddev = sqrt(mmmf_median_err); //mmmf_mode_stddev = sqrt(mmmf_mode_err); }
void PLearn::TestImputations::computeNeighborhoodStats | ( | int | nb_neighbors, |
int | max_miss_neigbors | ||
) | [private] |
Definition at line 492 of file TestImputations.cc.
References PLearn::is_missing(), PLERROR, PLearn::pow(), and PLearn::ProgressBar::update().
{ knnf_input.resize(train_width); knnf_neighbors.resize(nb_neighbors+max_miss_neigbors); knnf_mean_cov_err.resize(nb_neighbors); knnf_mean_miss_err.resize(nb_neighbors); knnf_nmiss_value_count.resize(nb_neighbors); knnf_mean_cov_err.clear(); knnf_mean_miss_err.clear(); knnf_nmiss_value_count.clear(); ProgressBar* pb = new ProgressBar( "computing the neighborhood imputation errors for " + to_deal_with_name, test_length); for (int test_row = 0; test_row < test_length; test_row++) { test_samples_set->getRow(test_row, train_input); for (int test_col = 0; test_col < train_width; test_col++) { if (test_col == to_deal_with_next) knnf_input[test_col] = covariancePreservationValue(test_col); else if (is_missing(train_input[test_col])) knnf_input[test_col] = covariancePreservationValue(test_col); else knnf_input[test_col] = train_input[test_col]; } ball_tree->computeOutput(knnf_input, knnf_neighbors); real knnf_sum_cov_value = 0.0; real knnf_sum_miss_value = 0.0; int knnv_value_count = 0; for (int knnf_row = 0; knnf_row < knnf_neighbors.size() && knnv_value_count<nb_neighbors; knnf_row++) { real knnf_value = ref_mis((int) knnf_neighbors[knnf_row], to_deal_with_next); if(!is_missing(knnf_value)) { knnf_sum_miss_value += knnf_value; knnf_nmiss_value_count[knnv_value_count]+=1; knnf_mean_miss_err[knnv_value_count] += pow(to_deal_with_value - (knnf_sum_miss_value / (knnv_value_count+1)), 2); knnv_value_count += 1; } if (!is_missing(knnf_value) && knnf_row<nb_neighbors) { knnf_sum_cov_value += knnf_value; knnf_mean_cov_err[knnf_row] += pow(to_deal_with_value - (knnf_sum_cov_value / (knnf_row+1)), 2); }else if(knnf_row<nb_neighbors){ knnf_value = ref_cov((int) knnf_neighbors[knnf_row], to_deal_with_next); if (is_missing(knnf_value)) PLERROR("In TestImputations::computeNeighborhoodStats(): missing value found in the reference with covariance preserved at: %i , %i", (int) knnf_neighbors[knnf_row], to_deal_with_next); knnf_sum_cov_value += knnf_value; knnf_mean_cov_err[knnf_row] += pow(to_deal_with_value - (knnf_sum_cov_value / (knnf_row+1)), 2); } } pb->update( test_row ); } delete pb; knnf_mean_cov_err/=test_length; knnf_mean_miss_err/=knnf_nmiss_value_count; }
*** SUBCLASS WRITING: ***
This should be defined in subclasses to compute the output from the input.
Reimplemented from PLearn::PLearner.
Definition at line 644 of file TestImputations.cc.
{}
void PLearn::TestImputations::computeTreeCondMeanStats | ( | ) | [private] |
Definition at line 411 of file TestImputations.cc.
References PLearn::isfile(), PLERROR, PLearn::pow(), and PLearn::ProgressBar::update().
{ tcmf_file_name = tree_conditional_mean_directory + "/" + to_deal_with_name + "/Split0/test1_outputs.pmat"; if (!isfile(tcmf_file_name)) PLERROR("In TestImputations::computeTreeCondMeanStats(): The '%s' file was not found in the tcf directory.",tcmf_file_name.c_str()); tcmf_file = new FileVMatrix(tcmf_file_name); int tcmf_length = tcmf_file->length(); if (tcmf_length < train_length) PLERROR("In TestImputations::computeTreeCondMeanStats(): there are only %d records in the tree conditional output file. We need %d.",tcmf_length,train_length); tcmf_mean_err = 0.0; ProgressBar* pb = new ProgressBar( "computing the tree conditional mean imputation errors for " + to_deal_with_name, test_length); for (int test_row = 0; test_row < test_length; test_row++) { to_deal_with_value = test_samples_set->get(test_row, to_deal_with_next); tcmf_mean_err += pow(to_deal_with_value - tcmf_file->get(indices[test_row], 0), 2); pb->update( test_row ); } delete pb; tcmf_mean_err = tcmf_mean_err / (real) test_length; //TODO check the formul //tcmf_mean_stddev = sqrt(tcmf_mean_err); }
Definition at line 473 of file TestImputations.cc.
References PLearn::is_missing().
{ real cvpf_sum_cov_xl = 0; real cvpf_sum_xl_square = 0; for (int cvpf_col = 0; cvpf_col < train_width; cvpf_col++) { if (cvpf_col == col) continue; if (is_missing(train_input[cvpf_col])) continue; cvpf_sum_cov_xl += cvpf_cov(cvpf_col, col) * (train_input[cvpf_col] - cvpf_mu[cvpf_col]); cvpf_sum_xl_square += (train_input[cvpf_col] - cvpf_mu[cvpf_col]) * (train_input[cvpf_col] - cvpf_mu[cvpf_col]); } real cvpf_value; if (cvpf_sum_xl_square == 0.0) cvpf_value = cvpf_mu[col]; else cvpf_value = cvpf_mu[col] + cvpf_sum_cov_xl / cvpf_sum_xl_square; return cvpf_value; }
void PLearn::TestImputations::createHeaderFile | ( | ) | [private] |
Definition at line 546 of file TestImputations.cc.
References PLearn::endl(), and PLERROR.
{ cout << "in createHeaderFile()" << endl; for (int train_col = 0; train_col < train_width; train_col++) { train_stats = train_set->getStats(train_col); train_total = (int)train_stats.n(); train_missing = (int)train_stats.nmissing(); train_present = train_total - train_missing; if (train_missing <= 0.0) header_record[train_col] = 0.0; // no missing, noting to do. else if (train_present < min_number_of_samples){ header_record[train_col] = -1.0; // should not happen PLERROR("In TestImputations::createHeaderFile: train_present(%d) < min_number_of_samples (%d) for variable %d(%s)", train_present,min_number_of_samples,train_col,train_set.fieldName(train_col).c_str()); } else header_record[train_col] = 1.0; // test imputations } header_file = new FileVMatrix(header_file_name, 1, train_names); header_file->putRow(0, header_record); }
void PLearn::TestImputations::createOutputFile | ( | ) | [private] |
Definition at line 601 of file TestImputations.cc.
References PLearn::tostring().
{ output_names.resize(6,knnf_mean_cov_err.size()+knnf_mean_miss_err.size() + knnf_nmiss_value_count.size()); output_names[0] = "test_length"; output_names[1] = "mean"; output_names[2] = "median"; output_names[3] = "mode"; output_names[4] = "tree_cond"; output_names[5] = "cov_pres"; for (int knnf_row = 0; knnf_row < knnf_mean_cov_err.size(); knnf_row++) { output_names.append("KNN_COV_" + tostring(knnf_row+1)); } for (int knnf_row = 0; knnf_row < knnf_mean_cov_err.size(); knnf_row++) { output_names.append("KNN_MISS_" + tostring(knnf_row+1)); } for (int knnf_row = 0; knnf_row < knnf_nmiss_value_count.size(); knnf_row++) { output_names.append("KNN_NB_MISS_" + tostring(knnf_row+1)); } output_record.clear(); output_file = new FileVMatrix(output_file_name, train_width, output_names); for (int train_col = 0; train_col < train_width; train_col++) output_file->putRow(train_col, output_record); }
void PLearn::TestImputations::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declares the class options.
Reimplemented from PLearn::PLearner.
Definition at line 63 of file TestImputations.cc.
References PLearn::OptionBase::buildoption, covariance_preservation_file_name, PLearn::declareOption(), max_number_of_samples, mean_median_mode_file_name, min_number_of_samples, missing_indicators, reference_set_with_covpres, reference_set_with_missing, and tree_conditional_mean_directory.
{ declareOption(ol, "min_number_of_samples", &TestImputations::min_number_of_samples, OptionBase::buildoption, "The minimum number of samples required to test imputations for a variable."); declareOption(ol, "max_number_of_samples", &TestImputations::max_number_of_samples, OptionBase::buildoption, "The maximum number of samples used to test imputations for a variable."); declareOption(ol, "mean_median_mode_file_name", &TestImputations::mean_median_mode_file_name, OptionBase::buildoption, "The Path of the file with those statistics for all the variables."); declareOption(ol, "tree_conditional_mean_directory", &TestImputations::tree_conditional_mean_directory, OptionBase::buildoption, "The Path of the dircetory containing the tree conditional means computed for each variable."); declareOption(ol, "covariance_preservation_file_name", &TestImputations::covariance_preservation_file_name, OptionBase::buildoption, "The Path of the file with the train_set empirically observed covariances and means."); declareOption(ol, "reference_set_with_covpres", &TestImputations::reference_set_with_covpres, OptionBase::buildoption, "The reference set corresponding to the index computed with the ball_tree, with the initial imputations."); declareOption(ol, "reference_set_with_missing", &TestImputations::reference_set_with_missing, OptionBase::buildoption, "The reference set corresponding to the index computed with the ball_tree, with missing values."); declareOption(ol, "missing_indicators", &TestImputations::missing_indicators, OptionBase::buildoption, "The vector of missing indicator field names to be excluded in the distance computation."); inherited::declareOptions(ol); }
static const PPath& PLearn::TestImputations::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::PLearner.
Definition at line 113 of file TestImputations.h.
:
//##### Protected Member Functions ######################################
TestImputations * PLearn::TestImputations::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::PLearner.
Definition at line 51 of file TestImputations.cc.
void PLearn::TestImputations::endtestimputation | ( | const char * | msg, |
... | |||
) | [private] |
Definition at line 284 of file TestImputations.cc.
References PLERROR, and PLWARNING.
{ va_list args; va_start(args,msg); getHeaderRecord(); for (int train_col = 0; train_col < train_width; train_col++) { if (header_record[train_col] == 1.0) PLWARNING("No all variable done!!!"); else if (header_record[train_col] == 2.0){ getOutputRecord(train_col); if(output_record[100]==0.0) PLWARNING("Element %d,%d is at zero in the output file. Meaby this variable was not treated",train_col,100); } } PLERROR(msg,args); }
void PLearn::TestImputations::getHeaderRecord | ( | ) | [private] |
Definition at line 567 of file TestImputations.cc.
{ header_file = new FileVMatrix(header_file_name, true); header_file->getRow(0, header_record); }
OptionList & PLearn::TestImputations::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 51 of file TestImputations.cc.
OptionMap & PLearn::TestImputations::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 51 of file TestImputations.cc.
void PLearn::TestImputations::getOutputRecord | ( | int | var_col | ) | [private] |
Definition at line 630 of file TestImputations.cc.
{ output_file = new FileVMatrix(output_file_name, true); output_record.resize(output_file->width()); output_file->getRow(var_col, output_record); }
RemoteMethodMap & PLearn::TestImputations::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 51 of file TestImputations.cc.
TVec< string > PLearn::TestImputations::getTestCostNames | ( | ) | const [virtual] |
*** SUBCLASS WRITING: ***
This should return the names of the costs computed by computeCostsFromOutputs.
Implements PLearn::PLearner.
Definition at line 646 of file TestImputations.cc.
References PLearn::TVec< T >::append().
{ TVec<string> result; result.append( "MSE" ); return result; }
TVec< string > PLearn::TestImputations::getTrainCostNames | ( | ) | const [virtual] |
*** SUBCLASS WRITING: ***
This should return the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats.
Implements PLearn::PLearner.
Definition at line 652 of file TestImputations.cc.
References PLearn::TVec< T >::append().
{ TVec<string> result; result.append( "MSE" ); return result; }
void PLearn::TestImputations::initialize | ( | ) | [private] |
Definition at line 300 of file TestImputations.cc.
References PLearn::endl(), PLearn::is_missing(), PLearn::isfile(), PLearn::manual_seed(), PLERROR, PLearn::shuffleElements(), and PLearn::ProgressBar::update().
{ // initialize the header file cout << "initialize the header file" << endl; train_set->lockMetaDataDir(); header_record.resize(train_width); header_file_name = train_metadata + "/TestImputation2/header.pmat"; cout << "header_file_name: " << header_file_name << endl; if (!isfile(header_file_name)) createHeaderFile(); else getHeaderRecord(); // choose a variable to test imputations for cout << "choose a variable to test imputations for" << endl; to_deal_with_total = 0; to_deal_with_next = -1; for (int train_col = 0; train_col < train_width; train_col++) { if (header_record[train_col] != 1.0) continue; to_deal_with_total += 1; if (to_deal_with_next < 0) to_deal_with_next = train_col; } cout << "total number of variable left to deal with: " << to_deal_with_total << endl; if (to_deal_with_next < 0) { train_set->unlockMetaDataDir(); // reviewGlobalStats(); endtestimputation("In TestImputations::initialize() we are done here"); } cout << "next variable to deal with: " << train_names[to_deal_with_next] << "("<<to_deal_with_next<<")"<<endl; to_deal_with_name = train_names[to_deal_with_next]; updateHeaderRecord(to_deal_with_next); train_set->unlockMetaDataDir(); // find the available samples with non-missing values for this variable train_stats = train_set->getStats(to_deal_with_next); train_total = (int)train_stats.n(); train_missing = (int)train_stats.nmissing(); train_present = train_total - train_missing; indices.resize((int) train_present); int ind_next = 0; ProgressBar* pb = new ProgressBar( "Building the indices for " + to_deal_with_name, train_length); for (int train_row = 0; train_row < train_length; train_row++) { to_deal_with_value = train_set->get(train_row, to_deal_with_next); if (is_missing(to_deal_with_value)) continue; if (ind_next >= indices.length()) PLERROR("In TestImputations::initialize() There seems to be more present values than indicated by the stats file"); indices[ind_next] = train_row; ind_next += 1; pb->update( train_row ); } delete pb; // shuffle the indices. manual_seed(123456); shuffleElements(indices); // load the test samples for this variable if (indices.length() > max_number_of_samples) test_length = max_number_of_samples; else if (indices.length() < min_number_of_samples) PLERROR("TestImputations::initialize() Their is less examples(%d) for the variable %s then the min_number_of semples(%d)", indices.length(),to_deal_with_name.c_str(),min_number_of_samples); else test_length = indices.length(); test_width = train_width; test_samples_set = new MemoryVMatrix(test_length, test_width); pb = new ProgressBar( "Loading the test samples for " + to_deal_with_name, test_length); for (int test_row = 0; test_row < test_length; test_row++) { train_set->getRow(indices[test_row], train_input); test_samples_set->putRow(test_row, train_input); pb->update( test_row ); } delete pb; }
void PLearn::TestImputations::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::PLearner.
Definition at line 97 of file TestImputations.cc.
References PLearn::deepCopyField().
{ deepCopyField(min_number_of_samples, copies); deepCopyField(max_number_of_samples, copies); deepCopyField(mean_median_mode_file_name, copies); deepCopyField(tree_conditional_mean_directory, copies); deepCopyField(covariance_preservation_file_name, copies); deepCopyField(reference_set_with_covpres, copies); deepCopyField(reference_set_with_missing, copies); deepCopyField(missing_indicators, copies); inherited::makeDeepCopyFromShallowCopy(copies); }
int PLearn::TestImputations::outputsize | ( | ) | const [virtual] |
SUBCLASS WRITING: override this so that it returns the size of this learner's output, as a function of its inputsize(), targetsize() and set options.
Implements PLearn::PLearner.
Definition at line 643 of file TestImputations.cc.
{return 0;}
void PLearn::TestImputations::train | ( | ) | [virtual] |
*** SUBCLASS WRITING: ***
The role of the train method is to bring the learner up to stage==nstages, updating the stats with training costs measured on-line in the process.
TYPICAL CODE:
static Vec input; // static so we don't reallocate/deallocate memory each time... static Vec target; // (but be careful that static means shared!) input.resize(inputsize()); // the train_set's inputsize() target.resize(targetsize()); // the train_set's targetsize() real weight; if(!train_stats) // make a default stats collector, in case there's none train_stats = new VecStatsCollector(); if(nstages<stage) // asking to revert to a previous stage! forget(); // reset the learner to stage=0 while(stage<nstages) { // clear statistics of previous epoch train_stats->forget(); //... train for 1 stage, and update train_stats, // using train_set->getSample(input, target, weight); // and train_stats->update(train_costs) ++stage; train_stats->finalize(); // finalize statistics for this epoch }
Implements PLearn::PLearner.
Definition at line 579 of file TestImputations.cc.
References PLearn::endl(), and PLearn::isfile().
{ // initialize the output file cout << "initialize the output file: " << output_file_name << endl; train_set->lockMetaDataDir(); output_record.resize(6+knnf_mean_cov_err.size()+knnf_mean_miss_err.size()+knnf_nmiss_value_count.size()); if (!isfile(output_file_name)) createOutputFile(); else getOutputRecord(to_deal_with_next); output_record.resize(6); output_record[0] = test_length; output_record[1] = mmmf_mean_err; output_record[2] = mmmf_median_err; output_record[3] = mmmf_mode_err; output_record[4] = tcmf_mean_err; output_record[5] = cvpf_mean_err; output_record.append(knnf_mean_cov_err); output_record.append(knnf_mean_miss_err); output_record.append(knnf_nmiss_value_count); updateOutputRecord(to_deal_with_next); train_set->unlockMetaDataDir(); }
void PLearn::TestImputations::updateHeaderRecord | ( | int | var_col | ) | [private] |
Definition at line 573 of file TestImputations.cc.
{ header_file->put(0, var_col, 2.0); header_file->flush(); }
void PLearn::TestImputations::updateOutputRecord | ( | int | var_col | ) | [private] |
Definition at line 637 of file TestImputations.cc.
{ output_file->putRow(var_col, output_record); output_file->flush(); }
Reimplemented from PLearn::PLearner.
Definition at line 113 of file TestImputations.h.
Definition at line 151 of file TestImputations.h.
The Path of the file with the train_set empirically observed covariances and means.
Definition at line 85 of file TestImputations.h.
Referenced by declareOptions().
Mat PLearn::TestImputations::cvpf_cov [private] |
Definition at line 180 of file TestImputations.h.
real PLearn::TestImputations::cvpf_mean_err [private] |
Definition at line 182 of file TestImputations.h.
Vec PLearn::TestImputations::cvpf_mu [private] |
Definition at line 181 of file TestImputations.h.
VMat PLearn::TestImputations::header_file [private] |
Definition at line 164 of file TestImputations.h.
Definition at line 163 of file TestImputations.h.
Vec PLearn::TestImputations::header_record [private] |
Definition at line 165 of file TestImputations.h.
TVec<int> PLearn::TestImputations::indices [private] |
Definition at line 171 of file TestImputations.h.
Vec PLearn::TestImputations::knnf_input [private] |
Definition at line 183 of file TestImputations.h.
Definition at line 185 of file TestImputations.h.
Definition at line 186 of file TestImputations.h.
Vec PLearn::TestImputations::knnf_neighbors [private] |
Definition at line 184 of file TestImputations.h.
Definition at line 187 of file TestImputations.h.
The maximum number of samples used to test imputations for a variable.
Definition at line 79 of file TestImputations.h.
Referenced by declareOptions().
The Path of the file with those statistics for all the variables.
Definition at line 81 of file TestImputations.h.
Referenced by declareOptions().
### declare public option fields (such as build options) here Start your comments with Doxygen-compatible comments such as //!
The minimum number of samples required to test imputations for a variable.
Definition at line 77 of file TestImputations.h.
Referenced by declareOptions().
The vector of missing indicator field names to be excluded in the distance computation.
Definition at line 91 of file TestImputations.h.
Referenced by declareOptions().
real PLearn::TestImputations::mmmf_mean_err [private] |
Definition at line 174 of file TestImputations.h.
real PLearn::TestImputations::mmmf_median_err [private] |
Definition at line 175 of file TestImputations.h.
real PLearn::TestImputations::mmmf_mode_err [private] |
Definition at line 176 of file TestImputations.h.
VMat PLearn::TestImputations::output_file [private] |
Definition at line 191 of file TestImputations.h.
Definition at line 190 of file TestImputations.h.
TVec<string> PLearn::TestImputations::output_names [private] |
Definition at line 193 of file TestImputations.h.
Vec PLearn::TestImputations::output_record [private] |
Definition at line 192 of file TestImputations.h.
Mat PLearn::TestImputations::ref_cov [private] |
Definition at line 152 of file TestImputations.h.
Mat PLearn::TestImputations::ref_mis [private] |
Definition at line 153 of file TestImputations.h.
The reference set corresponding to the index computed with the ball_tree, with the initial imputations.
Definition at line 87 of file TestImputations.h.
Referenced by declareOptions().
The reference set corresponding to the index computed with the ball_tree, with missing values.
Definition at line 89 of file TestImputations.h.
Referenced by declareOptions().
VMat PLearn::TestImputations::tcmf_file [private] |
Definition at line 178 of file TestImputations.h.
PPath PLearn::TestImputations::tcmf_file_name [private] |
Definition at line 177 of file TestImputations.h.
real PLearn::TestImputations::tcmf_mean_err [private] |
Definition at line 179 of file TestImputations.h.
int PLearn::TestImputations::test_length [private] |
Definition at line 172 of file TestImputations.h.
Definition at line 170 of file TestImputations.h.
int PLearn::TestImputations::test_width [private] |
Definition at line 173 of file TestImputations.h.
string PLearn::TestImputations::to_deal_with_name [private] |
Definition at line 168 of file TestImputations.h.
Definition at line 167 of file TestImputations.h.
Definition at line 166 of file TestImputations.h.
Definition at line 169 of file TestImputations.h.
Vec PLearn::TestImputations::train_input [private] |
Definition at line 156 of file TestImputations.h.
int PLearn::TestImputations::train_length [private] |
Definition at line 154 of file TestImputations.h.
PPath PLearn::TestImputations::train_metadata [private] |
Definition at line 158 of file TestImputations.h.
int PLearn::TestImputations::train_missing [private] |
Definition at line 161 of file TestImputations.h.
TVec<string> PLearn::TestImputations::train_names [private] |
Definition at line 157 of file TestImputations.h.
int PLearn::TestImputations::train_present [private] |
Definition at line 162 of file TestImputations.h.
The stats_collector responsible for collecting train cost statistics during training.
This is typically set by some external training harness that wants to collect some stats.
Reimplemented from PLearn::PLearner.
Definition at line 159 of file TestImputations.h.
int PLearn::TestImputations::train_total [private] |
Definition at line 160 of file TestImputations.h.
int PLearn::TestImputations::train_width [private] |
Definition at line 155 of file TestImputations.h.
The Path of the dircetory containing the tree conditional means computed for each variable.
Definition at line 83 of file TestImputations.h.
Referenced by declareOptions().
Definition at line 189 of file TestImputations.h.
Vec PLearn::TestImputations::weights [private] |
Definition at line 188 of file TestImputations.h.